Reference Checks
Without a doubt the tool that software selections most neglect is the reference call. More often than not, reference calls are treated as a mere formality — a box to check after the decision has been made just in case someone asks. When checking references after the decision has been made, the reference checker is just going through the motions hoping against the unlikely chance that something negative turns up. Any hints at issues or other useful information are summarily ignored. When done this way, reference calls are probably not worth doing at all.
But reference calls don't have to be this unproductive. When done properly they can provide critical information of what it is like to partner with a software supplier and use its software — after all, isn't that what a software selection is all about? Doubters of the efficacy of reference checks point to the fact that the software supplier is in control of the process. Vendors only connect you to customers that they have been successful with. This can be a valid point. But I would counter with two facts: one, it is desirable to know what types of clients and engagements a supplier has been successful with; and two, every software integration/implementation project has its challenges and it is useful to know what they were and how the supplier overcame them.
To make reference calls as effective a possible, I recommend the following guidelines:
-
When to call? Call references as early in the process as you can. Usually this is once you have established your short list of products and you have qualified your organization as a legitimate buyer. Insufficient references is a great way to rule out products that are not worth considering. The vendors that you work with may balk at connecting you with references this early. This is understandable since these people are their customers and they don't want to annoy them by repeatedly asking them to talk to unqualified prospects. You need to work out some kind of compromise here. Limit your request to one really relevant customer rather than a long list from which you will call two or three. You might be able to connect to one of these customers through your own network. Also, if you have benefited from a reference call, "pay it forward" by offering to be a reference for products your company owns.
-
Which Customers? Be very selective as to which references you speak with. You only want to talk to companies that are using the software to solve similar problems. When the software salesman flashes the customer logo slide (you know the one I am talking about), don't be dazzled by the household brands that you recognize. Really dig into what the customers are doing with the software. If you work for a university and you see another university's logo on the slide. Stop the conversation and ask what that university is doing with the software. You might learn that only a tiny department uses the product. In that case, don't put much stock in that "customer success story." If, like your employer wishes to do, the customer has standardized all departmental websites onto the platform, put that customer on your list of references to request. In addition to learning about their use of the product, you might also learn about some organizational challenges that would be relevant to your initiative. It is a red flag if the vendor can't point you to any customers who have used its software to do what you want to do.
-
Which Contacts? Only talk to reference contacts who are intimately involved with the implementation and day to day usage. Senior level managers tend to be sheltered from the detail that you want to know. They usually only hear about usability issues if the users are picketing down the halls. If they did hear about a usability issue, they many not know if it was related to the product or the implementation. Ideal reference contacts are project managers who were involved with the planning, execution, and rollout of the implementation. They know what "out of the box" features took weeks to "configure."
-
What to ask? It is important to have a list of questions to guide the conversation but don't be afraid to go off script. Look for clues of problems and dig in to unravel hidden issues. As a baseline, you want to know what they use the software for, when and to whom they first rolled it out, what version they are on, workflow and user segmentation, how actively they maintain the solution. Ask open ended questions because every one of those topics can be a source of issues.
If you follow these guidelines, reference calls can become a central part of your selection process rather than a useless formality. You will get first-hand knowledge from someone who has been down the path you are considering and can avoid spending too much time looking at products that have not demonstrated an ability in your problem domain.