I just published my CMS Selection Workshop handout on Scribd. The handout contains:
Examples of how to specify content types
Example usage scenarios
An example RFP table of contents
An example vendor demo agenda
Enjoy!
A customer struts into a car dealership, slams a 200 page requirements document down onto a salesman's desk, and triumphantly declares "I know exactly what kind of car I want to buy." The startled salesman opens the document to a random section and starts to flip through a few pages that describe a lug nut in excruciating detail. He looks at another random section and sees requirements about how the steering wheel should be joined to the steering column. After regaining his composure, the salesman looks up and says "from this document, I can definitely see that you are looking for a car. What do you want to use it for?" The business analyst suddenly looks confused and says "I don't know. I don't drive."
This is not just a lame joke. It describes a scenario happens all the time in CMS selections. There are two main problems here. First is the obvious problem that the customer believes himself an expert in cars because he has done a ton of research but he doesn't have the critical experience of having driven one. He can name all the features of a car and knows what they do but he hasn't had to use them. The second issue is more subtle. His 200 page requirements document is more like a design specification for a product that has already been built. It goes into details that are unnecessary like how the steering wheel must be connected to the steering column. What kind of penalty does he give if the steering wheel is connected in a different (and perhaps better) way? More importantly, there is no way his requirements document can be exhaustive. It would really have to be 20,000+ pages to cover every detail with same depth. So entire aspects of the car are probably omitted. Maybe it was something important like which side of the car the steering wheel is on. Rather than try to design your own car in a vacuum and then go around and see which one matches it, it would be better to draw up some coarse filters (price, intended usage, etc.) and then look at cars that passed the filter in their totality and see which one feels right.
This sounds obvious for car buying but you would be surprised how many CMS buyers collect requirements like that car customer; Or do the abridged version where they just name countless features (which in car shopping would produce a list like "6 cup holders, 1 gas pedal, 1 brake pedal, 1 clutch (optional), 5 gears, 3 windshield wipers, 6 windows, 4 wheels, 4 tires, 1 spare wheel/tire ... "). In many cases the requirements are gathered by people who have never used, nor intend to use, the CMS. They can't paint the bigger picture of the user, the task, and the content.
By focusing on how people work, rather than the features themselves, CMS evaluation criteria can identify features that are important and with enough context to understand which implementations of that feature will make it useful. In the car dealership story, if the customer walked into the dealership and said "I drive like a maniac and the wheels of my last three cars fell off," the salesman would not only know that the customer needed lug nuts but really beefy lug nuts, a good suspension, and perhaps a driving lesson.
Scenarios are the best way to capture this context for a software selection. A good scenario will describe the intent behind the task (what is the user trying to accomplish?), the context (what time, resources, and information does the user have?) and the flow (how does the person work? Who else does he need to collaborate with?). In the process of documenting a scenario, a number of features will be identified — features you might not even think of in a requirement brainstorming session. After writing a scenario, I typically list features at the bottom of the scenario to call out what functionality was used. Scenarios don't have to be long or comprehensive. Usually 1/2 to 1 page will capture enough of the story to understand what needs to happen.
To beat on the car buying analogy once again, you could think of a scenario like the route for a test drive. If you live in a city and rarely use a highway, the best test drive would be to drive in traffic and try to parallel park and park in your neighborhood parking garage. That would be more informative than driving on the interstate. If you test drove all the cars on the same route you would notice some big differences; like that you can park the Honda Fit in a compact car space but you can't even get the Ridgeline into the parking garage because the turning radius is too big. Your average car dealer probably will not give you much flexibility on your driving route, but your CMS vendor will (or should). Use that access to your advantage and create the most realistic driving conditions possible.
The car buying analogy breaks down in one key area. When you buy a car, you sign the paper work and then you drive it off the lot. Content management systems are not like that. Before you can use a CMS, you need to implement the software to support your content, processes, and web design. You need to configure, customize, and extend the platform. Scenarios will help this process because, once you buy the software, they turn into the user stories that will drive your implementation planning and long term road map. Some user stories will be achievable by configuring out of the box functionality; others will take more effort.
So when you find yourself slogging through a spreadsheet with hundreds of rows of requirements, think of that car buyer and ask yourself "are these requirements really going to help me find a CMS that I will be able to use to manage my website(s)?" If you are honest with yourself, the answer will probably be "no." If it is "no," put away the spreadsheet and start writing scenarios.
A dirty little secret in the CMS industry is that, while in-context editing is often what sells a CMS, the "power user" interface is usually what winds up getting used after implementation. This phenomenon obviously creates problems in the selection process because, when the sales demo focus on an interface that users will quickly grow out of, any usability impressions are irrelevant. This is also part of a bigger problem: the importance of in-context editing for sales has caused many CMS vendors to neglect their power user interface.
It is easy to understand why the sales demo gravitates to the in-context user interface: the audience finds it more intuitive. What is less obvious is why. In a typical CMS sales demonstration, the audience has the perspective of a site visitor. After all, this is not their site. They have no responsibility for it. As a site visitor, we think of editing the content that we see: "I see a typo;" "that sentence is hard to read;" "I would prefer to see another picture here." The user just wants to go in and fix it — like a Wikipedia article. Until it's fixed, that content issue is going to bug the user so directness and immediacy are critical. Like with a wiki, the in-context is ideal for solving these kinds of problems.
The content manager, however, has an entirely different perspective. The content manager is thinking more about the whole web site than any one page. The content manager has to solve problems like re-organizing the website and making global content changes. Needing to manually change every single page of a website is not acceptable so content reuse should be top of mind. From this perspective, the appearance of a page is less important than the actual content, which also includes information you can't see on the page but drives the behavior of the site. You can even go so far as to say that the visible page (what the visitor sees) actively hides information that the content manager needs to see. The visitor shouldn't know where else a piece of content is featured on the web site or what caused the personalization logic to show this item in this particular case — but the content manager does. Incidentally, this is why you should make product demos focus on scenarios. Scenarios force you to think about what the content manager does - not just dream of being able to edit somebody else's web site.
Yes, you can make the argument that the occasional content contributor (who 80% of the time experiences the site as a visitor) needs a simplified user interface to fix the issues that they notice or keep a few bits of information up to date. But, as an organization gets more sophisticated with managing content, those cases of simplistically managed pages (with no reuse and no presentation logic) get less frequent. At that point, you are just talking about the "about us" page and some simple press releases. Are you surprised that this is what your basic generic CMS demo shows? Furthermore, I am beginning to believe that the occasional user is a myth (another blog post).
In-context editing interfaces are steadily getting more powerful by exposing functionality like targeting and A/B testing but there inevitably comes a point when the content manager wants the full power of the application at his fingertips. As the in-context editors get better, that point gets pushed further out. But adding complexity and power to the in-context editing interface will no doubt steepen the learning curve for the occasional user and minimize the wow factor of the demo. And no CMS vendor will do anything to reduce the wow factor of their product demo.
You have just mentioned that maybe the pain you are feeling in managing your web content may be eased by implementing a web content management system (WCMS) and, all of the sudden, I.T. paratroopers are sliding down ropes with their software selection methodology and other "artifacts." You get suspicious as you recognize the same faces that "helped" you the last time around but are reassured that they have "learned aLOT" from their recent time and expense system procurement.
Spreadsheets are opened and fingers are poised over keys. They cue you with "R00001. The system shall be....?"
"easy to use?" is your diffident response.
"R00001. The system shall be easy to use. OK. R00002. The system shall...?"
Three months later, you have a spreadsheet with a thousand rows of "shalls" that any WCMS vendor (plus most ERP vendors) will say "yes" to. But, worst of all, they mean nothing to you. You are now back to precisely where you started.
What happened here?
Generic requirements gathering processes are self absorbed. They are optimized to comprehensively find business requirements, not understand them within the context of business goals. And the more the requirements are abstracted from the goal of managing content, the less they mean. Quantity and completeness are measurements of success rather than usefulness. What is more, most generic requirements analysis techniques are designed for building custom software rather than selecting software. While custom software development goes from requirements to design, when implementing an existing WCMS, much of the design is already in place. The trick to finding a WCMS is to match your needs with a pre-existing design. Generic requirements are an indirect path to that result.
In a WCMS selection, requirements gathering should stop when you have enough information to filter down the marketplace for a (3 or 4 product) short list. After that, the selection process takes a more experiential aspect where you look at the usability of the software and organizational compatibility with the suppliers that will help assemble the solution (software vendor and systems integrator). To be sure, this part of the process seems overly subjective to some - but, honestly, how objective is an aggregated "user friendliness" score of 3.2?
I like to focus on what I call "Leading Requirements." A leading requirement has at least one of two characteristics: it is important to the organization and/or it is a powerful filter on the marketplace. To be important, a requirement must critically affect daily usage or primary functions of the system. Powerful filter requirements get to the big demarcations of the marketplace - things like baking vs. frying, technology stack, content modeling, content reuse, workflow modeling, licensing strategies, etc. By focusing on leading requirements I can usually get down to 3 or 4 viable solutions that (based on industry gossip) appear to be working well at like companies and are sustainable.
By focusing on leading requirements, I can afford to take the time to document what each one really means. For functional requirements, I write "usage scenarios" that describe users using the solution to complete a business task (like publishing an article). At the bottom of each scenario, I list out the discrete requirements that were identified. These 2-4 paragraph narratives are then useful in the demo process because they become the script for the customized product demonstration.
After the product is selected, requirements are refined to determine what features will be enabled or implemented (and how) in the first release (and subsequent releases) of the system. At this point, you know the platform you are building on so you can explain requirements within the context of the native capabilities of the software. You can also adjust scope to leverage out of the box functionality and even prototype to clarify what you are talking about.
Focusing on leading requirements is not easy. It requires intimate knowledge of the business processes and also the content management industry. Still, there is no faster way to get to a short list of viable products and deeply evaluate them. If you want to learn more about leading requirements, I am teaching a workshop at the Gilbane conference in Boston. Will I see you there?
After you watch a couple of vendor demos, it doesn't take long to realize that the performance of the demo (how well the presenters know the product and how well they understand and connect with the audience) plays as much a part of the product impression as the quality and the capabilities of the product itself. Given that the sales team probably is not going to be around during your implementation or when your users first start using the system, this should scare you if you are basing your selection on the product demo. While it is important that a software vendor cares enough about your business to put some thought and effort into showing you the product, you also want to build your system on the the most suitable product. Here are some tips to manage vendor demonstrations that will isolate the important aspects of the vendor and the product and filter out the extraneous information that may confuse or distract you. For those software vendors out there, I hope that you read this and also Tony Byrne's advice. For people selling and evaluating open source, there are some slight nuances that I will mention at the end but probably cover more thoroughly in a different post.
A successful demo is all about preparation. You need to prepare the vendor (or systems integrator or in house staff if you are evaluating non-commercial software) with the information they need so they can do their best. You also need to prepare the audience on what they should be looking for.
Only do demos with a short list of vendors. Work with someone who knows the market to give you a short list of products to look at. That doesn't mean asking someone "what are the best CMS." If they know anything, they will tell you that it depends on your requirements. If they have an opinion. Well, it is just going to be that: an opinion. You need to focus on a short list for two reasons. First, if the vendor knows that he is in a field of 10 candidates, he is not going to invest as much in the demo. He will have a junior sales team give a generic demo. Second, when subjected sit through 10 demos, your staff will not invest as much of their attention in evaluating each product and they will start to muddle the products together.
Clearly define what you want the demo to show. Because content management systems (especially web content management systems) are so flexible, a demo should be a prototype that you define according to your requirements. Just like a prototype, you need to clearly specify what it needs to do. The approach that I find the most effective is using scenarios that describe tasks that need to be accomplished using the system. The demo should show how the user would accomplish that task using the product. The demo should also recognize the constraints introduced by your architecture. The vendor should not know you features that would not work in your architecture. Neither should they show features that you don't need. The demo should show your content. Ask the vendor to configure a content types that matches the the most complex content type in your content model.
Validate that the vendor understands your requirements. Have the vendor prepare a written response describing how their product can support your scenarios. Review it and give them feedback with ample time to adjust their demo in case they misunderstood what you need.
Prepare the audience. Prepare your audience for the demo by telling them what they should be looking for. A scorecard that lists the scenarios is useful for keeping people's attention on their needs, not gimicky features. If the audience does not understand basic content management theory (separation of content and layout, re-usability, content lifecycle, etc.) address that before the first demo. Vendors are actually pretty good at explaining that stuff but there are more effective uses of their time. Also, vendors tend to up their game when the realize they are dealing with a sophisticated audience.
The demo should use everyone's time as effectively as possible and should be organized to ensure that vital information is communicated to the right people.
Limit company background information. The vendor should be able to introduce their company and make the case that it is a stable company, it gets content management, and knows your industry. However, you need to contain the amount of time that they take to do it. They should be able to build a level of credibility and comfort with the audience but not infringe on the time they have to talk about their product within your context. Hopefully your short-list exercise already pre-qualified the vendors along these lines.
Mind your manners. Even if your corporate culture thinks it is OK for staff to attend meetings in-body only, keep distractions to a minimum. Ask your audience to put aside their email, blackberries, and cell phones and pay attention. Give the vendors every opportunity to engage with the audience. If the vendor is missing the mark, don't tune out. Instead, help steer them back on course. If you can't do that, politely end the meeting as quickly as possible and be happy that you were able to eliminate an option in a very hard decision.
Mark your scorecards. Without making it feel like a Bingo hall, have the audience take notes in their scorecards so that they remember what they saw and their impressions. By the time they have gotten back to their desks and answered their first of fifty waiting emails, they will have forgotten half of what they saw.
Break up the meeting. A thorough demo is enough to tax anyone's endurance. Not everyone needs to hear everything and people tend to lose focus after sitting for long periods of time. I usually break up demos into three main sessions. The first is the company background and functional session that all the stakeholders should attend. This is when the vendor walks through the scenarios and helps business users visualize using the product to get their jobs done. The next session is the technical session that shows what is going on behind the scenes and how the system can be customized and integrated. All the business users that are still awake can leave for the technical part. If they are asleep, leave them alone and let them dream about life with better content management tools. They can use the rest. The third session is the project management and licensing part where the vendor talks about the licensing needs, cost, and professional services. Your project management people and tech leads need to be part of the discussion. Everyone else can go back and extinguish the fires that have probably ignited during their absence.
"Yes" is not a good enough answer. When you ask if the system can do something, don't let the vendor get away with a simple "yes." Have them show it. And if they are not prepared to show it, have them describe how it would work and how much effort it would take to get it to work like that. You could also ask to speak to other customers that are using the product in that way.
Don't let wait long to get feedback from the audience. It doesn't take long for people to forget. Follow up and plan the next steps as soon as possible.
The post mortem. As soon as possible, get everyone in a room and have them express their observations and impressions. Ask them what they didn't see. Hopefully, they have notes on their scorecards. There were probably some scenarios that were not adequately explained. Get this information so you can follow up with the vendor.
Schedule follow ups. Talk through what additional information is needed with each of the vendors who earne
d further consideration. For the vendors that didn't make the cut, explain why. If the demo was a disaster but you think the product still has potential, you could give them another chance or you could take it as a sign that they are not prepared to support you. Remember, after the contract is signed, things are only going to get worse.
Prototype. If there is a question about something, build a prototype and allow users to bang on it. Different vendors will have different policies around this. Some create hosted sand boxes and allow business users to experiment. Others provide trial versions of the software so that a customer can attend training and try to build the prototype themselves.
Demos can either clarify or confuse, inform or misinform. If run properly, they can be the most important part of the selection process. At the end of the day, both you and the vendors are after the same goals. They want customers that are successful with their software. You want to be a customer that is successful with their software. However, that doesn't mean that a sales team can't get swept up trying to win a deal. It also doesn't mean that business users will not lose sight their goals when distracted by flashy features and a compelling demo performance. Be up front with this and try to work together to achieve this goal.
What about open source software? For the commercial open source products out there, this advice still holds. You just want to be even more sensitive about using the vendors time efficiently because they have less to gain in terms of licensing revenues. Assign some of your technical staff to dig around the product (and the community) for themselves. If a commercial open source vendor is able to invest in large sales teams, you can be pretty sure they have a pricing model (around support and maintenance) where they can collect revenues that are equivalent to commercially licensed software. Either that or they haven't had to think about building a sustainable software business yet. For community-based projects, you are not going to get a sales team. You should form an internal team (or pay a systems integrator) to build the prototype and play the role of the sales engineer. You probably need to do more homework to decide what platform(s) you should evaluate and be even more diligent in documenting your requirements. Otherwise, your developers will get drawn to nifty architectures and technology buzzwords and neglect what your business users need.
A while back, I wrote a post on selecting a CMS. I have since gotten requests for more step by step instructions. While, the process is not so formulaic that it can be written into a recipe, this is the Content Here approach. This methodology is optimized for the complex content management projects. Some of the steps require a deep background in the content management marketplace or help from an expert. But here they are.
Document the content types. Before doing anything, you need to know about the content that is to be managed: its structure, frequency of creation and updates, and who is responsible for it. Usually, this is a good time to visualize better ways of structuring, organizing, and managing content assets.
Document the scenarios. Write short, narrative-based scenarios describing what the system will be used for and how. Stay away from implementation specific details as much as possible. For example, rather than say "the user clicks a 'check spelling' button and the system lists misspelled words," say "the system notifies the user of misspelled words." This leaves it open to the product to determine the best way to meet the requirement - is it with color underlining as the user types?
Document the legacy systems. Most Content Here clients require integration with legacy systems. It is important to understand them, how they are deployed, and their interfaces. A diagram showing the existing enterprise architecture and how the content management platform fits in is very helpful.
Filter by technology. While content management is all about usability, start by developing a short list of products that are technically capable of meeting the requirements. The biggest reason for doing this is that evaluating for usability is such a subjective and intensive process and it is impractical for both the customer (and the vendor) to demo more than a couple of products (you will see this later on). You might have some non-functional constraints like the system has to be customized in Java or run on Windows. To do this step effectively, you really need to be immersed in the marketplace and know the different products and who is using them for what. Buy a CMS Watch report (WCM or ECM) or hire a consultant (like Content Here) or, better yet, do both. Based on the questions that I get, I think that sites like CMS Matrix are more likely to confuse you than help you.
Filter by price. If a product is way beyond your price range, you should either filter it out or see if there alternative licensing models that meet your budget.
Engage with the vendors. If you have been following the steps 1 through 5, you should understand your content management requirements and have a short list of 3 or so products that would all probably work in your environment. The next step is to find the product that your users will feel the most comfortable using. Hopefully, the software vendor understands your needs and will partner with you to meet them. If two products are very close functionally, I tend to go with the vendor that will be more helpful in the initial implementation and beyond. This is where I think the traditional RFP process is broken. Rather than create a formal procurement process that tries to compare "apples to apples," you want an informal, interactive process that will highlight the differences between the vendors and their approaches. You want to share as much information with them as you can. If you don't trust a vendor, they should not be on your short list.
Prototype and learn. Because each vendor on your short list has a good chance of winning the deal, and you are not giving him a 100 page RFP to answer, he should be willing to invest the time to build a prototype that will demonstrate how his product will support your content types and usage scenarios. If an open source product is on your short list, consider building your own prototype or paying a systems integrator to do it (this will give you an idea of what it is like to work with the systems integrator). You should evaluate the prototype as actively as you can. Ask what can be changed and how. Ask the vendor to give you access to the prototype so that users can play with it at their own desks. Have your users think outside of their normal day to day and be innovative. Teach them about best practices in content management.Hopefully, the prototype demonstration was like a training session so the business users are comfortable finding their way around the system. If you intend to have your technical staff help with the implementation and/or maintenance of the system, now is a good time to request the developer documentation and access to the customer support site. Schedule a technical session where the prototype builder walks through the code and configuration that he did to build the prototype.
Project what an implementation would look like. To get an idea of the total cost of the solution, do a project planning exercise of what it would take to implement the solution and customizations and migrate your content. You will be better off if you plan for an iterative deployment where the first release supports the bare essentials and subsequent releases layer in more functionality. This is especially the case when your users are new to content management or have been using a limited toolset. They will learn so much about their needs from the first release and have wonderful ideas for improving and extending the system.
There you have it. Eight easy steps to select a CMS.
I just ran across this article by James Robertson of StepTwo Designs. The general point is that the best way to convey the requirements for a product is to create a narrative that describes the business need for the system. I like this strategy. It is a refreshing departure from the typical "column fodder" technique where customers create feature matrices to compare products. In addition to keeping focus on the business need, which is always a good thing, I think this technique breaks what I think are dysfunctional buying and selling habits created by checklist based selection.
On the buying side, putting a feature on a checklist makes an assumption that the feature (and the way it has been implemented) will solve a particular business problem. This assumption can be incorrect because there may be better ways to solve the problem. For example, auditing may be a more effective way of keeping a history of changes than versioning. Or, the feature may be implemented in such a way that it is unusable to the users.
I think that matrix buying leads to bad products as well. To compete, product companies need to be able to "check-off" certain features. This can either lead to marketing overstating functionality of the product or, worse, products growing un-useful features for checklist coverage. Having the most features does not make a product the best. Not only is there a tendency toward diminishing returns, there comes a point where additional features actually reduce the utility of the product by making it overly complicated.
So what do you do with narratives? State what your users need to do, and challenge your vendor to show how that goal would be satisfied with their product. If there is a disconnect (and there will be most of the time) have a dialog about how the product can be modified. Better yet, talk about how other customers have used the product to meet a similar need.
The narrative technique seems less quantitative than the matrix method. However, I think that these matrices are always less scientific than they seem. Focusing on narratives (both of the problem and the solution) is more work for both the buyer and the seller because they have to communicate a lot more. More importantly, the business users are more involved. They can visualize using the solution in their context. Just remember to control the natural desire for bright shiny objects.