Categories are ways of, well, categorizing your content.
A music site might have categories of rock-and-roll, R&B;, country, classical, etc.
A personal weblog might have categories of home, work, politics, movies, music, books, etc.
The “sports” weblog in a newspaper site might have categories like scores, breaking news, standings, etc. Or maybe instead it might have categories like baseball, football, basketball, etc. Or (more likely) it might have both kinds of categories, with an individual article tagged with both “baseball” and “scores”.
A category is really whatever you want it to be. The main value of a category is that you can have a link that shows the most recent posts in a category. You can also set up separate RSS feeds for a category, for those people only interested in certain postings.
I still don’t know the difference between groups and categories.
I’m not sure which “groups” you mean. There is no relationship between categories and template groups. Template groups are related to presentation, while categories are related to content.
Category groups are mainly of interest to people running multiple weblogs. Category groups allow more than one weblog to share the same categories. If you only have one weblog, you’ll only have one category group. More info here.
The real question that people run into is when to use categories and when to create a separate weblog. More info here.