Evaluating a Community
If you are considering open source software, the community could be the "x-factor" that determines your success with it. The community will influence your experience of the software and shape the application's future. If you are used to commercial software selection, the concept of "community" is probably alien to you. You may be used to reading analyst reports about market share and corporate financials. "Community" feels squishy and qualitative by comparison. Even though the information that can be used to evaluate a community is visible, it takes some work to gather and interpret it. Here are some strategies that I use.
General Activity
Activity can be measured in email and IRC volume, bug activity, release frequency, and source code commits. SourceForge rates project popularity on several statistics. However, this rating system is not particularly reliable because many projects just use SourceForge as an inexpensive place to distribute the software. When a project does not use SourceForge's mailing lists, forums, or bug tracking tools, SourceForge's quantitative metrics are less than useful. Also, the activity of certain consumer projects, such as the Gaim instant messenger client, eclipses enterprise software projects. Ohloh gives a nice snapshot of code metrics such as top contributors, the amounts of comments in their code, the overall amount of code. Ohloh even has a neat little feature of how much it would cost to build the project from scratch. Unfortunately, not all projects are tracked by Ohloh.
The larger open source projects have regular events such as conferences, user groups, sprints, and even recreational outings. Social activity on a project shows personal investment within the community. Stronger relationships between developers usually translates into better communication and more potential for collaboration and negotiation. This is a good sign but not all good projects have a happening social scene.
For me, the most valuable resource is the bug list. While in the commercial software world you hope to be blissfully ignorant of bugs, in the open source world the bug tracking system is where people discuss possible improvements on the system as well as defects. Active bug lists mean that people are using the software and care about it. An issue that just sits could mean either the project does not have the developer bandwidth to enhance the software or that the leadership disagrees with the feature.
Vision and Priorities
Looking at volumes of bugs and identification/closing rates is informative but it is even more helpful to read some of the issues and ensuing discussion about them. That will give you an indication of the project direction. Depending on how you plan to use the software, the direction may be even more important than the current state. Many projects also publish a project roadmap that shows what features will be added and when. Enhancement requests that are considered important to the community wind up on the road map. Enhancements that conflict with the community's vision get rejected. These decisions indicate the communities priorities better than any other information about the project. While, as in all software development, priorities get shifted and there may be delays, it is useful to look at the project's track record of checking off items on the roadmap and meeting milestones. You want to know how reliable these road maps are and understand why features are not on the roadmap.
Leadership
The most successful projects have charismatic and energizing leaders. However, hinging the future of a project on the participation of one person is dangerous. If the project leader burns out, loses interest, or is hired away from the project, will the project survive? The true health of a community is best demonstrated by its ability to withstand a change in leadership. However, many projects have not been tested in this way. I look for projects with a solid secondary tier of contributors and a track record for promoting and recognizing people that show leadership qualities. It is from this pool that the next leader will emerge. Many projects designate managers for every release so contributors can test their mettle in leadership situations. Some projects create sub-teams with their own organizational structures. Others have a rotating leadership team.
Execution
Reliability and quality are not achieved through dedication and good intentions alone. In order for a project to produce good code, it has to have good processes that people actually follow. Usage of the bug list and roadmap show that the project is able to prioritize the work. Having development standards and adhering to them will ensure that the software will meet the vision. The better projects will have clearly defined coding standards and rules for submitting code. For example, many projects require developers to run unit tests and include the results for every code submission. Project leadership needs to be strict in enforcing these rules without discouraging potential contributors. Many healthy projects also have a strong social order where violators are "called out" on the mailing lists for their transgressions. The level of hostility depends on the culture of the project. Some projects are combative on the surface but there is an underlying layer of respect (a little like some families). Other projects are just downright caustic and push people away.
Participation
The better projects encourage and get participation from a functionally diverse community. Participation is not limited to submitting code. Non-technical members can write documentation, test the software, organize events, and answer questions on the mailing lists. Blogging about the software is also a useful form of participation. Projects should be inclusive and encouraging of people that want to participate. From talking with experts that study this stuff, I have learned that turnover within the community is actually a positive thing. New members bring new ideas and perspectives to the project. They test the software in ways that existing users did not anticipate. They also test assumptions. Newbie (n00b) questions may seem tedious to project veterans but they are vital to the health of the project. Projects that do not recognize this value and respond with an "RTFM" (Read The F*** Manual) response do so at their own peril. "RTFM" might be the right answer but there are more polite and helpful ways to say it. Look for the way the community responds kindly to n00b questions.
Economic Ecosystem
Are people making money off this project? If there is no money to be made, people will gravitate to other interests that pay the mortgage. Some projects are commercially sponsored, so developers can expect a paycheck. Some contributors are paid by their employer (that uses the software) to improve the software. Other ways to make money in open source are consulting, support, training, hosting, and writing books. Look for people offering these services for the project. If you can't find any, the community may not be sustainable.
Unfortunately there is no single indicator of community health and strength. I am sure that one day there will be some statistical algorithm that looks at all of these data points (and more) along with project success rates to come up with a single score. However, this will not tell you how well the community will serve you. The best way to evaluate a community is to engage in it with these factors in mind. Build relationships with community participants. Also listen for the gossip. Virtually every big event that has happened within an open source project has been preceded with rumblings either on the mailing list or in casual conversations. Once you join a community, stay as engaged as you can. This wil
l keep your finger on the pulse and give you more leverage and access when you need something.