When I help companies through a CMS selection, I focus on the whole solution rather than just the functionality of the software. Factors such as vendor compatibility and expertise availability (internal and external) also affect the sustainability of the solution &mdash sometimes even more than the feature set. Several of my recent consulting projects have de-emphasized the software component of the solution even more. These selections were part of larger initiatives that required significant help from an outside partner. In one case there was a comprehensive site redesign that included digital strategy, re-branding, and information re-architecture as well as implementing new functionality. In another case, the client was shifting to an outsourced model where a partner was to maintain the full infrastructure and assume all development responsibilities. In situations like these, while the software is important, the biggest risk is choosing the wrong partner to work with.

A dual selection like this poses a real problem. If you focus on the partner, you have the software choice made for you. I know that there are systems integrators that claim technology agnosticism but I seriously doubt them. The truth is that it takes a couple implementations on any platform to get it right. In some cases, it takes many (like 5) projects to run out of ways to mess things up. That is the downside of flexibility. So, when someone says “technology agnostic,” I hear either “we don’t have skills on any platform” or, like the the waitress at Bob’s Country Bunker said: “Oh, we got both kinds [of music]. We got Country, and Western.” It could be even worse: the integrator who claims neutrality can be paid by the software vendor to recommend a solution.

The other option is to go with a pure design agency and select a product (and integrator) afterwards to implement that design. This can be inefficient because the designers can arbitrarily and unknowingly make decisions that make implementation harder. You will probably need to rescope and refactor the design based on the native capabilities of the platform you choose. A lot of time can be saved if you have someone to suggest more efficient options during the design process. I know I am going to get slammed here by developers and software companies saying their product can do anything. If they could, they would have 100% market share. Pure strategy and design companies also tend to underestimate the cost of implementation so you might running out of money when it is time to implement the design.

It’s a chicken and egg problem. You can’t choose a product until you have had help with your requirements and you can’t get help with your requirements until you choose a development agency (that implicitly comes with a product). I have found this variation of my standard process to be effective.

1. Gather the requirements that are the most meaningful to a product selection. I call these leading requirements.
2. Use these functional and non-functional requirements to filter down the software marketplace to a very short list of product options (probably no more than two).
3. Find a couple of the best web agencies who specialize in working on either of these platforms — ideally, two for each of the two products.
4. Invite these agencies to present a solution based on their preferred platform. Like in my normal selection process, the presentation consists of the demonstration of scenarios that you defined when you gathered your leading requirements.
5. Evaluate the presentations across two dimensions: the product and the agency.

I have found this process to be extremely effective. The benefit that I didn’t anticipate was that the preparation of the prototype tested three very important aspects of the integrators: their consultative process for turning customer articulated requirements into a solution; their mastery over the platform; and their relationship with the software vendor. But the process is not perfect. Here are some issues:

• A professional services company has lower margins than a software company. They don’t have the prospect of an all-profit software license deal to justify a big bet on a sales initiative. Professional services companies will put up with a lot less run around than a software company will — especially the good ones. So the process has to be efficient and they need to have a good shot at winning. This means that you need to: have a short list of no more than four integrators; have budgeting worked out before you contact with them; and work together with them to get what you need. This collaboration is also useful in getting a feel for what it will be like to work with the firm.
• Some integrators have dedicated sales teams that can prevent you from getting to know their consulting capabilities. You should do everything you can to work directly with real consultants. Not only is it important to learn about their style and capabilities, delivery staff are less likely to tell you what they think you want to hear.
• Those filtering steps (1-3) are very hard if you are just learning about web development and don’t know your way around the industry. You need to talk to lots of people and learn from their experiences or work with someone who has.

Despite all of these challenges, doing a dual selection is not impossible. In fact, it can be quite fruitful. You just need to be focused and disciplined in your approach and execution. If you are interested in learning more, I am teaching a workshop on Selecting a CMS at the Gilbane Conference in Boston on December 1st. I hope to see you there.

While it may seem counter-intuitive to listen to a supplier telling you how to buy, you should definitely read Deane Barker’s article “Five Tips to Getting a Good Response to a Content Management RFP.” Deane is a co-founder of Blend Interactive, a web design and development firm. That may put him on the other side of the negotiation table but, as a potential partner, he wants you to be successful in your initiative as much as you do. That is actually not so out of the ordinary. As a consultant, you want to spend your time with clients who you have a great working relationship with. The better consultants can be more selective in the opportunities they pursue and nothing sends off more bad vibes than a dysfunctional selection process.

The article agrees with all the advice that I give on my blog (it even quotes me!). The one tip that buyers are going to question is openly stating budget. I tend to go back and forth on that myself. The benefit is that budget is the best way to communicate what you think the size of the project is. Getting that piece of information out in the open early will help the vendor present a solution that is in line with what you had envisioned. It also reduces the risk of harboring unrealistic expectations of what you can do. The risk of communicating budget is that the integrator will inflate the price to maximize margin. My current thinking is that you shouldn’t be working with a partner who you fear will take advantage of you. You should structure your selection process to verify the integrity as much as the skills and experience of the vendor.

If you are in the market for a development partner, read these tips. If you can get to Boston next month, you should join me in my CMS Selection Workshop at the Gilbane conference. In fact, Deane is going to be there too.

After collecting requirements, the second most difficult component of a CMS selection is taking all the information that was gathered during the evaluation phase and using it to make a decision. This is where people get crazy with spreadsheets and scoring in the hopes that math will somehow heroically make a complicated and confusing (and, lets face it, subjective) decision obvious and irrefutable. The process looks something like this… There are a bunch of selection criteria. People rate the products on each criterion. People weight the criteria. You do some multiplication and addition and out comes some very quantitative looking numbers. Nothing looks more convincing than a score where one option has more points than another. But, users don’t necessarily want to use a system just because it has the highest cumulative, weighted score. They want to use a system that helps them efficiently get their jobs done while introducing the fewest number of annoyances. if the measurement of accuracy is the overall satisfaction with the solution, this method is extremely faulty.

There are several reasons why the matrix scoring method fails to accurately select the right solution. First, the rating and weighting wind up being very subjective and arbitrary. Veterans of this approach know this to be true when the they remember the feeling of not knowing what to put down or wanting to change their score when they see another product or have more coffee. Second, the final score hides information that is important to the users. A typical example is where a user finds a very important (to him) feature totally unusable but that is overshadowed by excellent ratings in a bunch of less important features. Usually you can’t correct this with weightings – especially if there are lots of selection criteria. You can’t discuss trade-offs and compromises if you are just working with total scores. Lastly, criteria tend to be of unequal granularity. How can a broad criteria like “usability” be compared with something as specific as “SSL on the login page?”

I take a different approach to the decision making process. Instead of forcing the selection committee into making numerical ratings, I ask them to list their doubts with each solution. Examples of doubts are:

• a concern that the feature would not support a specific task
• unnecessary complexity or awkward behavior in doing a specific task
• an unsatisfactory explanation by the supplier about how a feature worked
• doubt about the vendor’s stability or ability to support the customer
• a potential technical incompatibility with the legacy infrastructure

Each of these doubts are investigated as whether they are valid (that is, if it was a misunderstanding or oversight), if there is a suitable work-around, or if there is a reasonable compromise. Through some facilitated sessions, we work through comparing the relative weaknesses of the competing solutions and determining what is tolerable. Follow up demos and calls with the vendors are scheduled and executed. Ultimately, the solution with the fewest legitimate and significant concerns wins. Facilitating these sessions is not as easy as simply reporting matrix scores but I think that it is good that people put some real intellectual energy into making such an important and complex choice.

At first glance, this system seems designed for selecting the lesser of evils and to some extent that is true – there is no such thing as a perfect solution and there will always be compromises (I should note here that there is an option of selecting nothing if no solution is good enough) – but it is really no worse than a numerical system that decides a 5 out of 1000 score is better than 3 out of a 1000. The benefit of the doubt technique is that it keeps the focus on things that have real impact on users and forces users to think through the implications of specific aspects of the solution. This is better than having a user register their concern as low numerical score and then just moving on. A secondary benefit selection committee members learn about their needs and software features as they watch demos and their selection criteria becomes more sophisticated. This approach allows potentially important information to enter the decision making process at any time. Also, after the product is selected, the selection committee can all clearly verbalize the reason behind the decision. If there is a complaint about the implemented solution, a selection committee member can say that they identified it as a concern and then explain the plan to lessen the impact.

I will be discussing this technique as well as all the other components of my CMS selection methodology in my “How to Select a Web Content Management System” workshop at the upcoming Gilbane Conference in San Francisco (June 2-5, 2009). Register before April 28th and save $200 off. Sign up for the full conference package and get an iPhone Touch.