How to make the most out of a vendor demo
After you watch a couple of vendor demos, it doesn't take long to realize that the performance of the demo (how well the presenters know the product and how well they understand and connect with the audience) plays as much a part of the product impression as the quality and the capabilities of the product itself. Given that the sales team probably is not going to be around during your implementation or when your users first start using the system, this should scare you if you are basing your selection on the product demo. While it is important that a software vendor cares enough about your business to put some thought and effort into showing you the product, you also want to build your system on the the most suitable product. Here are some tips to manage vendor demonstrations that will isolate the important aspects of the vendor and the product and filter out the extraneous information that may confuse or distract you. For those software vendors out there, I hope that you read this and also Tony Byrne's advice. For people selling and evaluating open source, there are some slight nuances that I will mention at the end but probably cover more thoroughly in a different post.
Preparation
A successful demo is all about preparation. You need to prepare the vendor (or systems integrator or in house staff if you are evaluating non-commercial software) with the information they need so they can do their best. You also need to prepare the audience on what they should be looking for.
-
Only do demos with a short list of vendors. Work with someone who knows the market to give you a short list of products to look at. That doesn't mean asking someone "what are the best CMS." If they know anything, they will tell you that it depends on your requirements. If they have an opinion. Well, it is just going to be that: an opinion. You need to focus on a short list for two reasons. First, if the vendor knows that he is in a field of 10 candidates, he is not going to invest as much in the demo. He will have a junior sales team give a generic demo. Second, when subjected sit through 10 demos, your staff will not invest as much of their attention in evaluating each product and they will start to muddle the products together.
-
Clearly define what you want the demo to show. Because content management systems (especially web content management systems) are so flexible, a demo should be a prototype that you define according to your requirements. Just like a prototype, you need to clearly specify what it needs to do. The approach that I find the most effective is using scenarios that describe tasks that need to be accomplished using the system. The demo should show how the user would accomplish that task using the product. The demo should also recognize the constraints introduced by your architecture. The vendor should not know you features that would not work in your architecture. Neither should they show features that you don't need. The demo should show your content. Ask the vendor to configure a content types that matches the the most complex content type in your content model.
-
Validate that the vendor understands your requirements. Have the vendor prepare a written response describing how their product can support your scenarios. Review it and give them feedback with ample time to adjust their demo in case they misunderstood what you need.
-
Prepare the audience. Prepare your audience for the demo by telling them what they should be looking for. A scorecard that lists the scenarios is useful for keeping people's attention on their needs, not gimicky features. If the audience does not understand basic content management theory (separation of content and layout, re-usability, content lifecycle, etc.) address that before the first demo. Vendors are actually pretty good at explaining that stuff but there are more effective uses of their time. Also, vendors tend to up their game when the realize they are dealing with a sophisticated audience.
During the demo
The demo should use everyone's time as effectively as possible and should be organized to ensure that vital information is communicated to the right people.
-
Limit company background information. The vendor should be able to introduce their company and make the case that it is a stable company, it gets content management, and knows your industry. However, you need to contain the amount of time that they take to do it. They should be able to build a level of credibility and comfort with the audience but not infringe on the time they have to talk about their product within your context. Hopefully your short-list exercise already pre-qualified the vendors along these lines.
-
Mind your manners. Even if your corporate culture thinks it is OK for staff to attend meetings in-body only, keep distractions to a minimum. Ask your audience to put aside their email, blackberries, and cell phones and pay attention. Give the vendors every opportunity to engage with the audience. If the vendor is missing the mark, don't tune out. Instead, help steer them back on course. If you can't do that, politely end the meeting as quickly as possible and be happy that you were able to eliminate an option in a very hard decision.
-
Mark your scorecards. Without making it feel like a Bingo hall, have the audience take notes in their scorecards so that they remember what they saw and their impressions. By the time they have gotten back to their desks and answered their first of fifty waiting emails, they will have forgotten half of what they saw.
-
Break up the meeting. A thorough demo is enough to tax anyone's endurance. Not everyone needs to hear everything and people tend to lose focus after sitting for long periods of time. I usually break up demos into three main sessions. The first is the company background and functional session that all the stakeholders should attend. This is when the vendor walks through the scenarios and helps business users visualize using the product to get their jobs done. The next session is the technical session that shows what is going on behind the scenes and how the system can be customized and integrated. All the business users that are still awake can leave for the technical part. If they are asleep, leave them alone and let them dream about life with better content management tools. They can use the rest. The third session is the project management and licensing part where the vendor talks about the licensing needs, cost, and professional services. Your project management people and tech leads need to be part of the discussion. Everyone else can go back and extinguish the fires that have probably ignited during their absence.
-
"Yes" is not a good enough answer. When you ask if the system can do something, don't let the vendor get away with a simple "yes." Have them show it. And if they are not prepared to show it, have them describe how it would work and how much effort it would take to get it to work like that. You could also ask to speak to other customers that are using the product in that way.
Follow up
Don't let wait long to get feedback from the audience. It doesn't take long for people to forget. Follow up and plan the next steps as soon as possible.
-
The post mortem. As soon as possible, get everyone in a room and have them express their observations and impressions. Ask them what they didn't see. Hopefully, they have notes on their scorecards. There were probably some scenarios that were not adequately explained. Get this information so you can follow up with the vendor.
-
Schedule follow ups. Talk through what additional information is needed with each of the vendors who earne
d further consideration. For the vendors that didn't make the cut, explain why. If the demo was a disaster but you think the product still has potential, you could give them another chance or you could take it as a sign that they are not prepared to support you. Remember, after the contract is signed, things are only going to get worse. -
Prototype. If there is a question about something, build a prototype and allow users to bang on it. Different vendors will have different policies around this. Some create hosted sand boxes and allow business users to experiment. Others provide trial versions of the software so that a customer can attend training and try to build the prototype themselves.
Conclusion
Demos can either clarify or confuse, inform or misinform. If run properly, they can be the most important part of the selection process. At the end of the day, both you and the vendors are after the same goals. They want customers that are successful with their software. You want to be a customer that is successful with their software. However, that doesn't mean that a sales team can't get swept up trying to win a deal. It also doesn't mean that business users will not lose sight their goals when distracted by flashy features and a compelling demo performance. Be up front with this and try to work together to achieve this goal.
What about open source software? For the commercial open source products out there, this advice still holds. You just want to be even more sensitive about using the vendors time efficiently because they have less to gain in terms of licensing revenues. Assign some of your technical staff to dig around the product (and the community) for themselves. If a commercial open source vendor is able to invest in large sales teams, you can be pretty sure they have a pricing model (around support and maintenance) where they can collect revenues that are equivalent to commercially licensed software. Either that or they haven't had to think about building a sustainable software business yet. For community-based projects, you are not going to get a sales team. You should form an internal team (or pay a systems integrator) to build the prototype and play the role of the sales engineer. You probably need to do more homework to decide what platform(s) you should evaluate and be even more diligent in documenting your requirements. Otherwise, your developers will get drawn to nifty architectures and technology buzzwords and neglect what your business users need.