The Linguist: Right, VPS and dedicated root servers are really a bargain these days. You’ll have total control over the system and much more flexibility to fit the environment for your needs. But, and this is a large but: you’ll have total control over the system ... and have to do all system updates, security etc. yourself. If you aren’t an experienced sysadmin, you should start with a local server first. Please don’t try “learning by doing” for an internet-connected server in the first place.
Remember: Having only one security hole in the system can result in a break-in and hackers storing illegal software for download or even phishing sites on your server - without your knowledge. If they use a rootkit and delete traces of their activities, this can result in major damage ... not only for your reputation: You may pay for xxx GB of traffic resulting from misuse, some providers even have “penality fees” if the owner doesn’t do everything to ensure security and legal problems can rise as well.
I don’t want to discourage you - just mention what can happen. You will have to read security mailing lists and update the system frequently. You should install an intrusion detection system and check for traces of rootkits often. Analyzing server (not only apache) logs is a must. And you should know about side effects when installing upgraded packages (e.g. what might break when going from MySQL 4.0.x to 4.1.x). All this is done by fulltime service personnel for managed hosting - you can get managed exclusive servers as well if you need the additional power.
Having a VPS or root server, you are your own system, database and web administrator. And you’ll have only shell access for web frontends for administrative works. Additionally, internet servers are targeted by literally hundreds to tausends of attacks each day - at least for my servers, the logs for hacking attacks have a size of 800++ kb each and every day ... and monitoring sends me stats via mail quite frequently. If time is money for you, it may count big for you and outrule the monthly fee fast. And if you find out later that a VPS isn’t the best solution for you, you’ll be frustrated if bound 12 month to the offer.
Therefore, I’d encourage you to first train server administration via shell with a server not (permanently) connected to the internet. Perhaps you can set up a dyn_ip access and play around with a local server connected through your internet access line. You can pull the plug at every time, then. In addition, familiarize yourself with the security requirements first by reading appropriated forums for root server owners. There are great ressources out there that cover different hosting situations and linux distros in detail.
And regarding points to look at regarding offers: traffic limits, cost for additional traffic, options and costs for resetting the server and recovery (are these tasks possible via web interface or requires manual actions?), backup space (if it isn’t included, where will you store your backups?), cost for additional domains/ip addresses, possible penality fees and else, ... Its a good idea as well to look for satisfied users for the package you’d like to choose first, because some providers don’t give you the performance (either cpu or bandwidth) in peak times that you’d expect from the marketing bubble. Especially true of “cheap” VPS offers because you really are in a shared environment (“virtual machine”) - but I can’t speak for the provider you mentioned.
Okay, thats what I learned - partly going the hard way 
I own a couple of dedicated root servers, but never needed one for EE really - they all where needed for special tasks that can’t be realized with a standard setup easily ... otherwise I’d personally go for a managed server/webspace.
Just my $.02 (okay, perhaps more $.04 according to the length of the posting
-Markus
