After collecting requirements, the second most difficult component of a CMS selection is taking all the information that was gathered during the evaluation phase and using it to make a decision. This is where people get crazy with spreadsheets and scoring in the hopes that math will somehow heroically make a complicated and confusing (and, lets face it, subjective) decision obvious and irrefutable. The process looks something like this… There are a bunch of selection criteria. People rate the products on each criterion. People weight the criteria. You do some multiplication and addition and out comes some very quantitative looking numbers. Nothing looks more convincing than a score where one option has more points than another. But, users don’t necessarily want to use a system just because it has the highest cumulative, weighted score. They want to use a system that helps them efficiently get their jobs done while introducing the fewest number of annoyances. if the measurement of accuracy is the overall satisfaction with the solution, this method is extremely faulty.
There are several reasons why the matrix scoring method fails to accurately select the right solution. First, the rating and weighting wind up being very subjective and arbitrary. Veterans of this approach know this to be true when the they remember the feeling of not knowing what to put down or wanting to change their score when they see another product or have more coffee. Second, the final score hides information that is important to the users. A typical example is where a user finds a very important (to him) feature totally unusable but that is overshadowed by excellent ratings in a bunch of less important features. Usually you can’t correct this with weightings – especially if there are lots of selection criteria. You can’t discuss trade-offs and compromises if you are just working with total scores. Lastly, criteria tend to be of unequal granularity. How can a broad criteria like “usability” be compared with something as specific as “SSL on the login page?”
I take a different approach to the decision making process. Instead of forcing the selection committee into making numerical ratings, I ask them to list their doubts with each solution. Examples of doubts are:
- a concern that the feature would not support a specific task
- unnecessary complexity or awkward behavior in doing a specific task
- an unsatisfactory explanation by the supplier about how a feature worked
- doubt about the vendor’s stability or ability to support the customer
- a potential technical incompatibility with the legacy infrastructure
Each of these doubts are investigated as whether they are valid (that is, if it was a misunderstanding or oversight), if there is a suitable work-around, or if there is a reasonable compromise. Through some facilitated sessions, we work through comparing the relative weaknesses of the competing solutions and determining what is tolerable. Follow up demos and calls with the vendors are scheduled and executed. Ultimately, the solution with the fewest legitimate and significant concerns wins. Facilitating these sessions is not as easy as simply reporting matrix scores but I think that it is good that people put some real intellectual energy into making such an important and complex choice.
At first glance, this system seems designed for selecting the lesser of evils and to some extent that is true – there is no such thing as a perfect solution and there will always be compromises (I should note here that there is an option of selecting nothing if no solution is good enough) – but it is really no worse than a numerical system that decides a 5 out of 1000 score is better than 3 out of a 1000. The benefit of the doubt technique is that it keeps the focus on things that have real impact on users and forces users to think through the implications of specific aspects of the solution. This is better than having a user register their concern as low numerical score and then just moving on. A secondary benefit selection committee members learn about their needs and software features as they watch demos and their selection criteria becomes more sophisticated. This approach allows potentially important information to enter the decision making process at any time. Also, after the product is selected, the selection committee can all clearly verbalize the reason behind the decision. If there is a complaint about the implemented solution, a selection committee member can say that they identified it as a concern and then explain the plan to lessen the impact.
I will be discussing this technique as well as all the other components of my CMS selection methodology in my “How to Select a Web Content Management System” workshop at the upcoming Gilbane Conference in San Francisco (June 2-5, 2009). Register before April 28th and save $200 off. Sign up for the full conference package and get an iPhone Touch.
Related posts:
- When it is not all about the software When I help companies through a CMS selection, I...
- Leading requirements You have just mentioned that maybe the pain you...
- Deane Barker’s tips on requests for proposals While it may seem counter-intuitive to listen to a...
- To Tell or Not to Tell? One of the main services that I provide my...
- CMS Selection Workshop Last week I was a panelist in a jboye08...
You make an excellent point. And as usual, you’re brave enough to suggest an alternative.
In a perfect world, I would hope people would be equally brave enough to base their choice on positive criteria — accepting the fact that there’s no such thing as 100% certainty. But realistically, I think you’re probably right, and it’s important to a) know nothing will be perfect; b) know exactly what the problems are going to be; and then c) dealing with those head on.
But it still feels, well, not completely right. The problem is, as you say, “this system seems designed for selecting the lesser of evils”. I’d like it if you could diffuse that a more strongly. (And then, for encore, a solution for global warming
).
Thanks for the comment Adriaan. One other thing that I forgot to note in the post is that this approach can be used to think about which product’s implementation of a feature is best (that is, most appropriate). For example, if both versioning systems are good but one is better because it automatically versions with every save (assuming that is what you want it to do), the doubt for the other product would be “does not version on every save.” Then the conversation turns to confirming that this feature is desirable and, if so, how the system could be customized to support that feature. At the end of the day, the selection process needs to determine how frustrated the user will be from feature’s absence rather than the joy from its presence. We tend to ignore things that work properly.
Well, IMHO, you are wrong… you are thinking of CMS selection as a feature/function evaluation process (which is a terribly terribly common IT person mistake)… and not thinking about what the business is trying to achieve on the final sites (brand loyalty, sign-ups, e-commerce sales, self-service, churn prevention). The only way to properly assess is to work top down from business web site requirements – then looking for the CMS and other infrastructure to support that.
Hi Carl,
You are right that in any web initiative you need to think of business goals. Some of those will drive business process; some will drive organizational change; some will drive the execution of the initiative; and some will indicate needed features. In your example, there are some underlying system capabilities that would need to support something like e-commerce or self service. Even IT people understand that
But this article isn’t about *looking* for a CMS. It is about making a decision between 2 (or 3) *found* products that seem to be viable solutions. Back to your example, let us say that your creative staff love the presentation capabilities of product A but the e-commerce functionality is very weak. My process would explore ways to mitigate the e-commerce issues. This could mean customization or, potentially, integrating another e-commerce platform. Depending on the feasibility of these alternative strategies, the creative team may need to see what could be done to the presentation capabilities of product B or what compromises are acceptable.
[...] via Seth Gottlieb, Doubt « Content Here. [...]