This issue has plagued my update sites for years and I finally found a workaround.
The problem: you create an Eclipse update site, you add a category, then you add a feature under the category. You build the update site (using either Build or Build All). Sometimes, when you try to install the features from the update site with "Install New Software..." you don't see your category and you need to uncheck "Group items by category". Even if you try to rebuild everything, delete the artifacts.jar and content.jar, you still cannot see your category. Annoying isn't it?
The solution: remove the category and the features from your update site. Save. Add them again. Build All. You need to do this every time you make a change (e.g., update the version number of a feature). This is a silly bug in Eclipse, but it is the only way I could reliably work around the issue.
Root cause: it seems that there is some caching of the site.xml involved and that restarting Eclipse may help, but it never worked for me.