Tuesday, March 6, 2007

POC, Prototype, or Pilot? When and Why

I frequently hear the terms "Proof of Concept" (POC), "Prototype", and "Pilot" used interchangeably. This is unfortunate because they mean different things and serve very different functions. In fact, I think that if you want to successfully implement technology, you should be doing all three. The good news is that, at least to some level, you probably are. But the extent to which you are benefiting from POCs, prototypes, and pilots depends on the level of consciousness and deliberateness you practice. By "deliberateness," I mean having an understanding of purpose. I use "consciousness" to say that you pay attention to the process and are aware of the results. More on that later.

Proof of Concept

A Proof of Concept (POC) is a small exercise to test a discrete design idea or assumption. Software developers tend to do POCs instinctively when they experiment with technology. An example of a POC is testing whether one technology talks to another. To be deliberate, a POC should clearly state what it is to be proven and to what degree. For example, it may not be enough that a message can be transfered from system A to system B if the complete solution calls a specific level of performance and reliability. You may need to test under certain conditions. POC's often take the outward form of a simple "Hello World" but that may not be enough if the system needs to support an extended character set.

The results of a POC need to be measurable so that they can be fed into the decision making process. Your decision should be based on thresholds like a performance level that is acceptable. Thats the consciousness part. It may not be enough to for someone to just say "yeah, we tested that and it works." You want the results of the test to see how or how well it works. A POC may involve secondary research if you trust the source. Maybe another group tested the same concept. If their process was as rigorous as yours, you may be able to use their results. There are also user interface POCs that explore the usability of a specific UI control or visual metaphor. These need to be designed and tested with the same degree of thoroughness.


A Prototype is a more fleshed out system that tries to simulate the full system or at least a material part of it. Prototypes are used to test the viability or usefulness of a system or subsystem. You plunk an intended user of the projected system in front of the prototype and he/she/it (the user might be another system) should be able to visualize the experience of using the complete system. The completeness of the prototype depends on what you are testing. For example, you might want to build a three dimensional physical model to test an innovative new shape for a telephone handset. If there is question as to whether the person can hear a speaker which has been designed in an unconventional place, the prototype may need a speaker. Or possibly, you could do a POC of just holding a speaker in that position and testing audibility in different conditions. Pretty much all manufactured goods go through a prototyping process because that is usually the best way to figure out the feasibility and cost of manufacturing the product. For software, prototypes can have varying degrees of fidelity from visual mockups to working software.

You need to be deliberate in defining what you hope to gain through building the prototype and the minimal level of completeness that it would take to get that information. There is some disagreement as to whether a prototype should be constructed to be thrown away or whether it should be the initial building blocks of the complete system. I think it depends on a number of things. You should invest as little as possible in the prototype otherwise the sunk costs might make you stick with a bad idea longer than you should. If that means building it out of a technology which is agile to use but unsuitable for full scale deployment, then use the easy technology with the expectation of throwing it away. However, if you are dead sure on the core idea and you just want to prototype different variations, it may be more practical to build the core technology framework and then build prototypes off of that.

If you go through the hassle and expense of building a prototype, use it. Get it in front of your intended audience and listen to what they say. Don't just listen. Hear them. If you are not doing to do that, don't call it a prototype. Call it an iteration.


A Pilot uses the full production system and tests it against a subset of the general intended audience. The reason for doing a pilot is to get a better understanding of how the product will be used in the field and to refine the product. Maybe there are some open ended questions about scalability that only a live audience using the product can answer. If you are doing a Pilot (or Beta) for a reason other than to build hype, make sure that you target the right group of people and you are prepared to collect the right metrics. You may consider prepping your pilot group to evaluate the full breadth of the application and give them channels to provide feedback. If there are particular areas of the application that you really want to test, call attention to them. Determine what kind of quantitative metrics you will be collecting and have the tools and time to analyze them. If you are going to do a Pilot you should be prepared to act on the information that you collect and fold refinements into a general release. You should probably have a process to prioritize enhancements ideas and a threshold for determine what enhancements to implement.

POCs, Prototypes, and Pilots are all important tactics in implementing technology. However, just using the words is not enough. You need to engage in these activities deliberately and methodically for them to be of any value. And you will never realize that value unless you consciously record the results and turn this data into usable, actionable information.