After collecting requirements, the second most difficult component of a CMS selection is taking all the information that was gathered during the evaluation phase and using it to make a decision. This is where people get crazy with spreadsheets and scoring in the hopes that math will somehow heroically make a complicated and confusing (and, lets face it, subjective) decision obvious and irrefutable. The process looks something like this… There are a bunch of selection criteria. People rate the products on each criterion. People weight the criteria. You do some multiplication and addition and out comes some very quantitative looking numbers. Nothing looks more convincing than a score where one option has more points than another. But, users don’t necessarily want to use a system just because it has the highest cumulative, weighted score. They want to use a system that helps them efficiently get their jobs done while introducing the fewest number of annoyances. if the measurement of accuracy is the overall satisfaction with the solution, this method is extremely faulty.
There are several reasons why the matrix scoring method fails to accurately select the right solution. First, the rating and weighting wind up being very subjective and arbitrary. Veterans of this approach know this to be true when the they remember the feeling of not knowing what to put down or wanting to change their score when they see another product or have more coffee. Second, the final score hides information that is important to the users. A typical example is where a user finds a very important (to him) feature totally unusable but that is overshadowed by excellent ratings in a bunch of less important features. Usually you can’t correct this with weightings – especially if there are lots of selection criteria. You can’t discuss trade-offs and compromises if you are just working with total scores. Lastly, criteria tend to be of unequal granularity. How can a broad criteria like “usability” be compared with something as specific as “SSL on the login page?”
I take a different approach to the decision making process. Instead of forcing the selection committee into making numerical ratings, I ask them to list their doubts with each solution. Examples of doubts are:
- a concern that the feature would not support a specific task
- unnecessary complexity or awkward behavior in doing a specific task
- an unsatisfactory explanation by the supplier about how a feature worked
- doubt about the vendor’s stability or ability to support the customer
- a potential technical incompatibility with the legacy infrastructure
Each of these doubts are investigated as whether they are valid (that is, if it was a misunderstanding or oversight), if there is a suitable work-around, or if there is a reasonable compromise. Through some facilitated sessions, we work through comparing the relative weaknesses of the competing solutions and determining what is tolerable. Follow up demos and calls with the vendors are scheduled and executed. Ultimately, the solution with the fewest legitimate and significant concerns wins. Facilitating these sessions is not as easy as simply reporting matrix scores but I think that it is good that people put some real intellectual energy into making such an important and complex choice.
At first glance, this system seems designed for selecting the lesser of evils and to some extent that is true – there is no such thing as a perfect solution and there will always be compromises (I should note here that there is an option of selecting nothing if no solution is good enough) – but it is really no worse than a numerical system that decides a 5 out of 1000 score is better than 3 out of a 1000. The benefit of the doubt technique is that it keeps the focus on things that have real impact on users and forces users to think through the implications of specific aspects of the solution. This is better than having a user register their concern as low numerical score and then just moving on. A secondary benefit selection committee members learn about their needs and software features as they watch demos and their selection criteria becomes more sophisticated. This approach allows potentially important information to enter the decision making process at any time. Also, after the product is selected, the selection committee can all clearly verbalize the reason behind the decision. If there is a complaint about the implemented solution, a selection committee member can say that they identified it as a concern and then explain the plan to lessen the impact.
I will be discussing this technique as well as all the other components of my CMS selection methodology in my “How to Select a Web Content Management System” workshop at the upcoming Gilbane Conference in San Francisco (June 2-5, 2009). Register before April 28th and save $200 off. Sign up for the full conference package and get an iPhone Touch.
Communicating Across the Techno Gap
Monday, March 9th, 2009You have probably heard people talking about “dotted lines” in organizational charts. That is when someone has partial accountability to someone other than his direct manager. An org chart notation that I would really like to see is a “flashing red” line. I would use it to show when someone reports to a manager who has no clue what he does.
Any reader of Dilbert knows that this happens all over organizations but the case that I am the most keenly aware of is when a technologist reports to a non-technologist (or, perhaps worse, a former technologist). As a consultant, this is the area where I am regularly deployed. I am often hired by the non-technical manager to help him validate and/or understand what his technical staff are doing. I am also often hired by the technical staff to validate and explain their strategy. Consultants get put into this position because this is an area where companies struggle on their own. They need an outsider to facilitate and verify because communication and trust is so compromised. While I do love this work and would be happy to perform this role at your organization, here is some free advice to get you started so you can make some progress on your own.
Communicating to a non-technologist
Your manager doesn’t really understand what you do despite your best attempts to explain it. You suspect that he doesn’t care. Maybe he doesn’t want to understand. Maybe he can’t help that his brain just locks up when you say words like “refactor.” You don’t feel appreciated. You are about to stop trying.
When talking to your non-technical manager, don’t try to dumb down what you say but for every detail mention, make a very clear connection to something that matters to the business. That usually means cost (short term and long term), quality, and impact. By impact, I mean things that will benefit the business: time savings, competitive advantage, etc. Provide inputs for any financial analysis he wants to do.
Don’t get bogged down in numbers. Instead, draw comparisons because the units are probably meaningless. At the end of the day, the non-technical manager wants to be reassured that progress is being made and nobody is making “unconventional” technology decisions. An example of the former, is saying that you are “doubling storage” rather than “adding a 500GB RAID 5 storage array.” For the latter, say “companies like EBay use MySQL” rather than “MySQL is a fully ACID compliant database.”
More important than anything you say is how you listen. Listen to his concerns. Figure out what kind of proof will alleviate these concerns and provide it. Prototyping is a great way to show a technology in terms that a non-technical person can relate to – much better than UML.
Communicating to a technologist
Admit it. You are a little afraid of your tech guy. He has strange working habits and speaks in a language that you only barely understand. Even though you manage him, you have very little visibility over what he does. Unlike with other people who report to you, your advice to him is pretty much worthless because you both know that you have no idea what you are talking about. Whenever you ask a question, the answer just confuses you more. You just stop asking. The scariest thing you can think of is that your tech person leaves and, after drifting helplessly for weeks, you learn that your entire infrastructure is in chaos and about to collapse. What is going on in this technical empire he has built under your nose?
If you don’t know anything about technology, you should start learning. Unless you are retiring in the next couple of months, you can only expect technology to become a more important part of the business that you are supposed to run. Have your technology guy help educate you. Ask him about the technical details underneath what you see as a user. But don’t just listen to your internal technology people, listen to what is happening outside of your organization. Talk to your peers at other companies and learn about what they are doing. Read stuff online. Hire a good consultant. Share what you are learning with your technical staff in a non-confrontational way. Don’t go in with managerial bravado. You are the student – be humble. It is a warning sign if they get defensive when you show a genuine interest.
The best approach for making technology decisions is to describe what you need in very clear terms (your technical person will call that requirements gathering). I like usage scenarios as a way to communicate these things. Ask your technical staff to put together their best solution and schedule time to review its viability and sustainability. Have an open dialog about the implications. Meet regularly with the team – not just at the end to “sign-off.”
More important than anything you say is how you listen. Treat the interaction not as manager but as a partner and teammate. You both have the same goals. When the technical discussion gets out of your depth, don’t shut down! Instead, connect it to something that you do understand and that matters to your responsibilities. Share the insight that you have about the company’s needs and what your boss will ask for.
Posted in commentary, management | No Comments »