I just saw a post on the CM Pros mailing list describing a project to compare CMS using a markup language called CMSML which is like Description of a Project for CMS with special emphasis on the features that CMS should have. For those of you that are enrolled in the CM Professionals Summit, you can see a presentation by Bob Doyle and Erik Hartman including a demo of a site that includes ratings for 93 CMS systems (how about that for a teaser!).
The tool uses a star-based scoring system for the categories Editing, Content Management, Document Management, Information Retrieval, and Records Management. Each of these categories has from 10 to 30 criteria that sum up to an overall score. The categories sound a little like "uses" similar to the approach that I advocated in an earlier post. For example, looking for a Document Management System vs. a Content Management System. I could see this information being used to get to a short list of products that excel in a few key areas.