I have finally gotten around to reading Clay Shirky’s excellent book Here Comes Everybody. I love Clay’s writing style and the way his perspectives make me think. One of the points that really resonated with me was about open source. But before I get into it, I should say that when Clay talks about open source, he is really talking about community developed open source (not commercial or institutional open source). Readers of this blog know that the designation “open source” simply means that it is licensed under an OSI certified license. “Open source” says nothing about how the software was developed. Given that disclaimer, Clay nails it when he describes the advantage of community developed software:

The bulk of open source projects fail, and most of the remaining successes are quite modest. But does that mean the threat from open systems generally is overrated and the commercial software industry can breathe easy? Here the answer is no. Open source is a profound threat, not because the open source ecosystem is outsuccessing commercial efforts but because it is outfailing them. Because the open source ecosystem, and by extension open social ecosystems generally, rely on peer production, the work on those systems can be considerably more experimental at a considerable less cost, than any firm can afford. (from page 245 of the hard cover version)

This quote comes from a chapter called “Failure for Free” that discusses the importance of failure for innovation. The key point is that traditional companies can’t afford to really innovate because they must limit their bets to initiatives that they feel have a high likelihood of success. Innovation winds up being constrained to small tweaks to things that are well known. Real breakthroughs are rare. Community development doesn’t make better ideas but it reduces the cost (and risk) of trying ideas that at first seem unlikely to succeed. Out of all this experimentation comes unexpected successes. Even the failures provide useful data that can be turned into future successes.

When open source critics see the volume of incomplete or dead open source projects as evidence against the sustainability of open source, they are missing the point. A low success rate is critical to real innovation; but that is hard to understand by someone for whom failure is discouraged or not tolerated at all. It is widely accepted that the best design for an experiment is one with a 50% chance of failure: if you know what the outcome will be, the experiment is not worth doing. But if the cost (in time and resources) of the experiment is zero or close to it, the ideal failure rate is much higher.

While we have seen many examples of community developed software out-innovating its commercial peers, this phenomenon has very little to do with open source. An open source license can create an environment that invites contribution by reducing the risk that one will exploit and unduly profit from another’s contribution. But what is more important is how the software is managed. Most successful open source projects have adopted an architecture consisting of a core that is tightly managed for stability and quality and is extensible by adding modules. The bar for module code is considerably lower than core code. This allows an open and dynamic community to experiment with new ideas. The community not only bears the cost of failure but it also brings in new perspectives that the core maintainers don’t have. Some of these extensions will be absorbed into the core or will become part of a commonly used bundle. Others will wither and die.

This pattern of stable core and innovative fringe does not have to be unique to open source. It doesn’t really matter how the core is licensed if entry into the fringe is open. The one area that open source licensing helps with the core is in achieving a level of ubiquity that attracts a community. High licensing fees or the bottleneck of a traditional software sales process can limit the size of the user base and discourage contribution. The community developer needs the promise of a large user base to justify the time investment of contributing his work. But open source is not the only way to achieve a large user base and there are plenty of examples of commercial software and SaaS user communities that have a successful code sharing ecosystem. One company that is particularly successful is Salesforce.com. Salesforce has created a very powerful API and module packaging framework. More importantly, they have priced the product to penetrate small-medium sized companies and non-profits. In particular, the Salesforce Foundation makes the product very inexpensive for non-profits. These strategies have infused the customer population with lots companies that are highly motivated to share. These customers are active both within the AppExchange community and also integrating 3rd party open source projects like Plone (see Plone/Salesforce Integration) and Drupal (see Salesforce module).

On the flip side, there are plenty of commercial open source software companies that not been able to leverage community development. There are two primary ways an open source software product can prevent community development from flourishing. First, there can be factors that limit adoption of the core like in the case of open core products that discourage the use of the open source version. Second, the architecture can be monolithic so the only way for an outsider to add a feature is to fork the codebase. All a software producer has to do is solve those two problems. The supplier doesn’t even really need to provide tools for sharing code. If the install base is large (which usually means the software is useful) and the architecture is modular, developers will find a way to communicate and share code. They can use services like Google Code, GitHub, or SourceForge to collaborate. Too often companies put code sharing tools in place without solving those two problems and then complain when they don’t see any activity. In many cases, the user audience is too small to support a contributing ecosystem. In other cases, the incentives are not lined up appropriately. Shirky calls the three ingredients of a collaborative community: promise, tools, and bargain. The promise is what the contributor can expect to get out of participating; the bargain is the terms and rules for participating. Open source licensing helps with the promise and the bargain. The open source promise is that others may help improve your code. The open source bargain is that people can’t take exclusive ownership and profit from your work. Some communities, like Salesforce and Joomla! have thriving commercial extension marketplaces. In those cases the promise and bargain are different but they are very motivating.

Any widely used software application, if it is appropriately designed, can benefit from community development and the benefits are not limited to the successful contributions. Perhaps the biggest value of community development is increased innovation through reducing the cost of failure. In order to harvest this value, a company needs to actively monitor, analyze and participate in its community. There is a goldmine of information sitting in the chaff of failed contributions as well as the modules that do gain traction. Companies that ignore this value do so at their own peril.

My career spans across the domains of a software developer and a knowledge worker. I participate in software development projects and I also do the things that your typical knowledge worker does: communicate information through reports, presentations, and other knowledge assets. One thing I have noticed from this perspective is that software development has been improving from the adoption of open source tools, behaviors, and practices while most knowledge workers live in the comparative stone ages of siloed working habits. As someone who loves living out in the country and working with people from around the world (and is also trying to reduce his carbon footprint), I look forward to the day when knowledge workers start to harness the techniques that distributed open source development teams have used to effectively produce quality software. Here are some places to start.

• Using Mailing Lists and IRC.

  Mailing lists (and sometimes IRC) are the core channel of communication within open source development teams. Point to point emails (and, shudder, phone calls) are discouraged because they lead to duplicate communication and there is no centralized record of what was said. IRC is preferable to point to point instant messaging for the same reason. Frequently once there is a critical mass of substance on a topic, a wiki page (mentioned later) will be used to capture and organize the idea as it develops.

  Especially in the world of IRC, it is important to include some social value (i.e. fun) in the dialog. Otherwise, the people who have all the answers have little benefit in participating. I have worked on projects where everyone worked within arms reach of each other, listened to music on their headphones but constantly communicated over chat. I liked this way of working because if you had a complex thought in your head that would unravel if you were interrupted, you could ignore the conversation and catch up when you got to a good stopping place. I realized the secondary benefit much later: when the team was broken up and staffed on other projects, the conversation kept on going even though people were spread all over the world.

• Using Wikis.

  Most open source projects use wikis for documentation. Wikis have a number of advantages over Office/Sharepoint: they stress information over formatting; they have excellent history and version comparison; they encourage referencing information (through links) over duplication (copy/paste); there is no local copy to synchronize; and they don’t require special client-side tools to contribute. Every time I open a word processor, I challenge myself whether this needs to be a “document.” My general rule of thumb is that I only do it for proposals, deliverables, and contracts where the audience needs their own local copy.

• Managing Vision and Tasks.

  Even though I have worked on several software projects that wound up going nowhere, I have to say that an even higher percentage of knowledge work is wasted. You could argue that the act of knowledge work leads to learning which creates knowledge. But running around in circles without any results is demoralizing. I like developing software systems because you see tangible results. I wish knowledge work was more results oriented. The better managed software projects maintain a road map which communicates an overall vision of what is going be achieved broken down into milestones. Gantt charts just don’t capture the “why” like a road map does. I think knowledge work could benefit from the same understanding. People should be responsible for mapping their individual activities to a shared set of goals and priorities.

  Open source projects rely heavily on issue tracking systems to manage work. Bugs and enhancement ideas all go into the issue tracking system where they are prioritized and assigned. If you have an idea for a new feature, you create an issue, assign it to yourself and start working. Maybe someone else shares your interest and starts to collaborate with you. In distributed teams, where you can’t see people working, you need other ways to keep track of what people are doing – not just if you are given the role of “project manager.” An issue tracking system would be a good way to achieve this.

• Holding Sprints.

  Open source projects have long understood that effective collaboration depends on communication and trust, which are most efficiently achieved through face to face interaction. For this reason, many open source projects hold sprints (also known as hackathons) where people work closely together (as in 2 to a computer) and mix hard work with an equal balance of socializing and play. In a typical office setting, people need to rush home at quitting time to attend to their home life. If they normally work from home and travel to a sprint, they are more likely to hang out after work and get to know their colleagues.

  I think the sprint technique could have huge potential in knowledge work. It would get people together to share information and it would create networks for people to use in solving future problems. One big thing about software sprints is that they are centered around solving a specific problem or accomplishing a goal. I think that knowledge workers would really respond well to this sense of purpose.

• Componentizing.

  Good developers know how to break down their software into re-usable components and reuse them. Most knowledge workers live by copy/paste (or Save As). Copy/paste not only creates redundancy that is hard to manage, it also frequently inserts unwanted content into your document. If you want to re-use something as it is, try to reference it. Granted, software has the advantage of build processes that assemble the appropriate parts to produce a cohesive product. Technical documentation (in XML) is often built like software. Most other stuff (particularly anything written in an Office (Microsoft or Open) is not.

  A while back I produced a coding standards document for my company. I based it on the Sun Microsystems’ Java Coding Standards. Rather than copy the whole thing and then edit the parts I wanted to change, I referenced the original and created a small addendum document that discussed where we were deviating from Sun’s standards. While this does not create a polished deliverable, it conveyed the necessary information which is what I was trying to do in the first place.

• Designating a Committer.

  In large open source projects, lots of developers submit their code to a small team of committers who review everything before committing it to the source code repository. Knowledge workers are constantly creating assets and dumping them into places where they are often never read. Sometimes these assets are never reviewed but are randomly discovered. Then it is up to the reader to determine if the information is valid. Because of this, most companies are swimming in information of wide ranging quality and accuracy. Search technology vendors claim the answer is in sophisticated retrieval systems. I would argue that more attention to the review and categorization of information at the intake end is needed. The committer for knowledge work would be a subject matter expert with librarian skills. This committer would know about what has already been done, be able to judge quality, and see opportunities to create shared resources. In open source projects, the committer is a position of status earned by climbing up through a meritocracy. The analogous knowledge worker position in most companies is really low on the totem pole. People who move up the latter in the knowledge world tend to drift away from the content and their distance leads to neglect.

I don’t think these practices should be that difficult to adopt but they are rare outside of software development. The obstacles are probably mostly based on culture. Old habits are difficult to break. I think another issue is that knowledge assets are commonly thought of as a by-product of the work rather than the purpose of the work (like software is). Therefore, there is less urgency in addressing the quality of the assets. But if institutional knowledge is a priority of the organization, the assets themselves need greater attention and investment.

Further reading:

• Managing projects the open way
• Managing projects with Trac