Information Week recently published an article about an ongoing study of software bugginess. In 2006, a Homeland Security sponsored Coverity study of the major open source projects (Linux Kernel, Apache HTTP, etc.) found these projects to have a similar defect rate to analoguous commercial products (although commercial software defect rates are not made public). I am not surprised by this finding since these large institutional projects have professional developers (employed by major software companies like IBM and Oracle) working on them.
What is interesting about this article is that a new code analysis tool by Stanford University, called Prevent SQS, runs on a periodic basis and has discovered that bugs are being fixed at an extremely rapid rate. To quote the article:
A total of 7,826 defects have been identified and fixed through the Homeland Security review, or one every two hours since it was launched in 2006. Bugs and vulnerabilities have been found in most open source projects, which isn’t surprising. What is surprising is the speed with which some projects resolve the issues as Coverity airs them on its Web site, versus other projects that lag behind.
By putting vulnerabilities out in the open, a project with a large community is able to mobilize and respond to risks that users of the software are not willing to tolerate. However, commercial software products tend to be less open about their bugs and prioritize the addition of new features where they hope to achieve competitive advantage in acquiring new customers. That is not to say that these bugs do not get fixed. Anyone who has run a Microsoft IIS environment knows about all the security updates and packages you need to keep up with. However, the commercial patch probably comes after the bug has been exploited on a customer installation and has become a customer relationship liability.
Related posts:
- US pays to make open source safer The U.S. Department of Homeland Security has recently pledged...
- Evaluating PHP Applications While not quite as trendy and chic as Ruby...
- A Tale of Two Releases For the past month or so, there has been...
- A new metaphor for living with open source This morning I was meeting with a client and...
- Community Development I have finally gotten around to reading Clay Shirky’s...
Their assertions are not very interesting without access to the 50% of the data that is missing. For instance, it would be very telling to see the distribution of defect rates for commercial software. Does it all hover around 1/1000?
They also do not mention what they compared: means or medians? Looking at the drastically lower defect rates for open source packages that people actually use (as opposed to yet another IRC client someone wrote once for fun), I have my doubts about their conclusions.
Software as a Service I think changes this. Not only are bugs identified quickly users of google and salesforce.com can get bugs fixes, enhancements and updates distributed realtime. Things are always in bets.
Additionally these companies, who don’t think about distribution of patches, can focus on testing as priority number one. I know google has a new test best practice sheet over every urinal instead of the sport scores every morning.