Are we experimenting or are we just playing around?
I frequently hear the term "experiment" used to describe a low effort, low stakes attempt at something. Like when you don't have the time or authority to commit sufficient attention and resources to an idea but you are curious to learn a little bit more. While that certainly describes one aspect of experimentation, it ignores the broader scientific method that makes experiments powerful learning tools.
The full scientific method is much more intentional and rigorous than just playing around with something. It requires precise process and detailed interpretation of observations to answer specific questions. As a result, the findings of a scientific experiment are much more useful and compelling.
As an example, let's consider a company that wants to experiment with AI assisted coding. The "playing around" version would be to hand out some seats to a few developers and then ask them what they think. The implicit research question might be "will developers like to use an AI coding tool after using it for a couple of weeks?" If that is really what you want to know, great! Assuming that you have chosen the right tool and tasks to use it on, this casual experiment might give you useful information about potential adoption. But it won't tell you how much money, time, and attention to invest in AI coding. It also won't tell you about the greater implications and opportunities of AI.
To think strategically about AI coding, you need to ask deeper, more substantive questions. For example:
- Will developers be more productive using an AI coding tool?
- Will AI increase or decrease code quality?
- Will using AI allow developers to spend more time doing high impact things?
- What elements of the software workflow have the most potential for optimization with AI?
- Will product value grow faster with AI in the software development lifecycle?
Answering these questions will give more confidence in deciding how and how much to invest in AI, but they are much harder because they require quantitative measurements. To do experiments that matter, you need to formulate your hypotheses as a change in measurable metrics. For example, "productive" needs to be expressed in quantifiable units like story points, features, release frequency, cycle time, etc. One hypothesis could be "developers using AI assisted coding will complete more story points per sprint than developers who do not ." Then you can design an experiment that controls for developer experience and task complexity. No metric is perfect and tracking them can be expensive overhead. For this reason, most engineering organizations don't even have a baseline to be able to measure a productivity gain.
You need to have a balance of open ended exploring and structured experimentation. Exploring can help you get familiar with the context and lead to observations that call for deeper, more structured scientific inquiry. For example, when playing around with AI coding, the team may remark that they "feel more productive" and that would naturally continue to more specific questions like are they objectively more productive and, if so, by how much? Are there costs to this productivity.
The greatest value is being able to test core assumptions that bigger decisions are based on. Assumptions exist all over the place whether you recognize them or not. You might have implicit assumptions about the root cause of the customer problem that your product or feature is trying to solve. You may assume that a person would use a capability if they had it. You may assume that using a capability would lead to optimization of the greater process. All of these assumptions are probably testable with intentional experiments. As the experimentation process progresses, it turns assumptions into knowledge that leads to confident investments.
So the next time you use the word "experiment," ask yourself the following:
- Do I have a specific question that I am trying to definitively answer?
- Can I formulate a hypotheses that can be tested?
- Do I have quantifiable data that would change if the hypothesis was correct?
If you don't have those things, you are not experimenting, you are just exploring.
The next time someone in a meeting says "let's experiment with this," you'll know what question to ask: "What exactly are we trying to learn?" If they can articulate a specific hypothesis they want to test with measurable outcomes, you're about to do real science. If they just want to "see what happens," that's exploring – which is fine, but call it what it is. Both have their place, but only one will give you the confidence to make bigger bets.