Conducting Lean Experiments
The Biggest Risk in Product Development is that we build a product/feature no one wants
Hey PMs, Today’s newsletter is brought to you by our partner Dayone.
Struggling to bring your new business idea to life? Day One creates a community-powered program for early-stage founders to learn how to launch businesses. Get a hands-on curriculum, feedback, and a tight-knit peer community. Learn more and apply today!
What is Lean Experiment
Lean Experiments are based on the Lean Startup approach to creating new products and services under conditions of extreme uncertainty. Lean Experiments are designed to quickly and cheaply gather evidence to validate or invalidate risky assumptions about your product.
Hypothesis-Driven Product Management
It’s no longer good enough for a Product Manager to say, “I think users want this feature.” Instead, you need to ask, “What outcome do we predict this feature will have?” and validate your answer with empirical data.
Hypothesis-driven product management is the practice of treating the development of new products as running a series of experiments. Instead of formulating requirements, we formulate hypotheses along with some validation criteria that state how strong of a signal we need to consider the hypothesis true. We use what we learn from each experiment to iterate on our ideas until we get where we want to be, or, until we determine that the product isn’t viable and cancel the effort.
Experiments Test Our Assumptions
A lean experiment is the smallest experiment we can run to quickly test our assumption. We start small and fast and then increase the scale and scope of our experiments over time.
An experiment consists of three parts: a hypothesis, a test, and validation criteria.
The hypothesis is a falsifiable version of our assumption. Remember to make sure you are only testing one variable in each hypothesis, otherwise, you won’t get reliable data.
The test is how we intend to test our hypothesis and provide it true or false.
The validation criteria are the signal we need to see to consider the hypothesis true.
Types of experiments:
A/B Test A comparison of two versions of a product or feature to see which one performs best. Works best with large sets of users for small incremental optimizations of an experience and business model.
Concierge Test A technique to replace a complex automated technical solution with humans who directly interact with the customer. Helps us validate whether anyone wants our product.
Wizard of Oz Test A technique to replace the product backend with humans. The customer believes they are interacting with an automated solution. Helps us validate whether anyone wants our product.
Smoke Test is Commonly a website that describes the product’s value proposition and asks customers to sign up for the product before it’s available. Helps us validate whether anyone wants our product.
The Scientific Method
Make observations
Formulate a hypothesis
Design an experiment to test the hypothesis
State the indicators to evaluate the result of the experiment
Conduct the experiment
Evaluate the results of the experiment
Accept or reject the hypothesis
If necessary, make and test a new hypothesis
Validating your experiment
The questions you should ask before you start validating your experiment are the following
1. Validate the problem. Is this a problem worth solving?
2. Validate the market. Some users might agree that this is a problem worth solving. But are there enough of them to make up a market for your product?
3. Validate the product/solution. The problem might exist, but does your product actually solve it?
4. Validate willingness to pay. There might be market demand and a great product. But will people actually be willing to reach into their wallets and pay for it?
Experiments
Every good experiment has four things :
Observation – Something we can see or notice
Hypothesis – a restatement of the assumption, to start with the words “we believe that…”
Test – the thing you’re going to do or build to validate the hypothesis through empirical evidence
Evidence – a metric that clearly shows whether your hypothesis is correct or incorrect