I think my main concern with upgrading is corrupting files and thus the site either goes down or behaves irregularly. If the upgrade process is painless I am happy to carry out updates for sure.
They key is preparation and a step-by-step process.
Usually, I download the latest version, then prepare the new /system folder with plugins, extensions, modules, whatever is needed. In most cases there will not be changes to /images, /themes, etc. I upload the /system folder, change permissions of config (copied from the old /system folder) and cache, then simply rename the older /system folder (usually something like this—/system_20080426) then rename the new /system folder, and run the update.php file and follow directions. It’s been a few years since I’ve had any corrupt files but you usually find out rather quickly, and reverting backwards is as simple as just renaming folders again.