Roberto Galoppini has an interesting case study on selecting an open source project management tool. In it, he describes his SOS Open Source methodology for filtering open source projects by looking at a number of factors organized into three categories: sustainability, industrial strength, and project strategy. The case study doesn't go into much detail but Roberto has built a tool that aggregates quantitative and qualitative project information from a number of disparate sources and builds scores. I saw a demo around 6 months ago and was impressed by the graphs he was able to create. While this technique cannot be expected to make a technology decision for you (you need to know your requirements and to have hands-on experience for that), it can be used to filter down the market and help you decide where to invest your evaluation energy.
Despite its ubiquity, open source software is still unchartered territory for most technology buyers. That is not to say that most companies don't use open source software, nearly all companies leverage at least open source utilities, libraries, and infrastructure (operating systems, databases, web servers, etc.). Many companies use open source business applications too. It is just that many companies adopt open source technologies in haphazard and spontaneous ways — at least not with the same level of conscientiousness put into an expensive commercial software purchase. While I don't think buyers should put much stock in Gartner's or Forrester's opinion of technology, it barely exists for open source technologies. That point was hammered home in a recent a Olliance webinar when one of the panelists said that Gartner and Forrester offer no value on open source. All the CIOs on the panel leveraged their peers and internal experts rather than their analyst subscriptions.
Ideally, technology procurement should be able to sense if there is something wrong going on with the project. The information is out there and you can get it in real time (as opposed to commercial software companies that only report quarterly). You just need to know where to look. Tools like SOS Open Source provide a useful high level picture to quickly highlight potential issues that should be investigated. It is unlikely that mainstream analysts will be able to develop this level of awareness for open source projects so I think there is great opportunity for these data aggregation tools.