We’ve got the whole shabang under version control and I dread updating the core and addons to be honest. I make a copy of the site, then delete thru svn and add the files back in - my templates etc are kept out of the system/expressionengine folder like Gareth’s so I’m only losing the repo history for first-party and third-party code (with the exception of config.php. That said, you could do most of the leg work in here with http://ee-garage.com/nsm-config-bootstrap and locate that outside the system/expressionengine folder.
I’m starting to take a serious look at keeping the EE core separate and using a shedload of symlinks (cue bash script) to replicate the ee structure within the repository. Which means the index.php, admin.php, images, templates, css & js are under version control and first-party and third-party files are kept elsewhere.