15/36 : Lean Experiment & Case Study
"Test fast, fail fast, adjust fast." - Tom Peters
What is the Lean Experiment
Lean Experiments are based on the Lean Startup approach to creating new products and services under conditions of extreme uncertainty. Lean Experiments are designed to quickly and cheaply gather evidence to validate or invalidate risky assumptions about your product.
The Art of Lean
The lean process enables organizations to speed up and focus experimentation in order to reduce wasted effort. Many organizations spend a great deal of time and resources on building solutions that don’t end up achieving their intended impact. Lean accelerates the process of weeding out ineffective ideas and helps quickly validate ideas that show real promise.
The Origins of Lean
To understand the lean method and its applicability it helps to understand its origins. Part of a broad revolution in the business world, lean belongs to a set of innovation and process improvement methods that also includes Six Sigma, which managers at Motorola developed to enable error reduction; Agile, a flexible and iterative approach to software development; and Human-Centered Design, a solution-building process created by leaders at the design firm IDEO.
Lean has two distinct strains: “lean production” (also known as “lean manufacturing”), a structured method first developed by Toyota more than 25 years ago that applies to complex processes like manufacturing, logistics, and health services; and “lean startup,” a set of principles and practices developed in Silicon Valley over the past decade that help entrepreneurs and intrapreneurs launch new products and services.1 Think of lean production as a way to maximize the efficiency and impact of a good idea, and think of lean startup as a way to figure out whether an idea is worth pursuing in the first place. Although the two strains developed separately and have distinct processes, they share a commitment to identifying clear hypotheses, conducting rapid experiments, and developing new product or service models in response to experimental data.
Over the past decade, several developments—increasing global competition, accelerated technological change, the emergence of big data—have forced nearly every major company to adopt data-driven, rapid experimentation methods in most aspects of their operations.
Today, when you buy a pair of stretch pants at H&M or download a new iPhone app or make a purchase from Amazon or click a link on Facebook, you are generating data for a series of experiments that will inform how companies make their next strategic decision. Companies that have incorporated rapid experimentation into their operations range from large corporations like General Electric, Target, 3M, and Xerox to high-growth start-ups like Dropbox, Etsy, and Upworthy.
The Elements of Lean
The lean process, as it applies to the business world, has several core components. We have adapted those components to form a model that suits the way that organizations operate. Here we will list the components in the order that they might occur in a typical lean experimentation project. But keep in mind that lean is more circular than it is linear, and the sequencing of components in any given experiment will vary. (See “The Lean Experimentation Process” below.)
Ideation and analysis | With your target constituents in mind (or, better yet, with your constituents in the same room), generate ideas for programs and solutions that you think might solve their problems or help them achieve their aspirations. These ideas are what we call “value hypotheses.” As you develop such ideas, analyze similar programs and solutions that already exist, and figure out how your approach might improve on those offerings. (In the business world, that process is called “competitive differentiation”).
Constituent discovery | Get out of your office and listen to the people you hope to serve. Through surveys and one-on-one conversations, find out what your constituents truly need and want. Put your value hypotheses in front of constituents, and observe how they respond to those ideas. (In the business world, this process is called “customer discovery.”) Done well, constituent discovery will bring to light ideas that you hadn’t considered, and those ideas in turn should lead you back to the ideation phase. Ideation and constituent discovery should complement each other in a rapid feedback loop.
Building | Determine the one or two “riskiest hypotheses” that apply to your idea. A risky hypothesis, in this context, is an assumption that is critical to the success of your idea—an assumption that may, however, prove to be invalid. In the lean process, you should focus your attention on the riskiest hypotheses. To test those hypotheses, develop an MVP (that is, a basic prototype of your idea). Also create a rough financial model for your idea that covers cost estimates and potential revenue sources. In many cases, your MVP will be a small-scale version of your program or service. (One common lean tactic is to customize and test pre-built products. This approach is widespread in the technology world, where there has been a proliferation of ready-to-use tools for developing apps, social platforms, and the like.2) Another option is to build a “paper MVP”—a lean tool that dramatically reduces the cost of testing demand for a program. A paper MVP can take the form of a simple flyer about a not-yet-built program, for example, or a basic online sign-up page for a prospective service.
Testing | Design a plan to validate (or invalidate) your riskiest hypotheses. Then roll out your MVP to a group of constituents and collect data on how they react to it. Be sure to test the MVP in a way that will provide data on metrics that pertain to those hypotheses. Avoid focusing on vanity metrics that might give you feel-good results but don’t actually help you validate or invalidate an idea.
Responding to data | Analyze the results of your test. Did your MVP appeal to fewer people than you had hoped it would? Did it encounter unforeseen logistical challenges? Did you charge a price for it that ended up being too high?
If your data show that you have a flop on your hands, hit the reset button and begin the experimentation process again before investing more resources in your idea. In the lean startup field, that’s called a “pivot.”
If your data show promise, use feedback from the test to build a better iteration of your idea. Then test that version of the idea, and continue iterating and testing the idea until you have verified that it will deliver its intended value. We call this process the “build-test-respond” cycle. (It’s a variation on the “build-measure-learn” cycle used in the lean startup model.)
Scaling up | Once you have an idea that works, use the data that you have gathered during the constituent discovery and testing phases to get buy-in—from your board, your staff, and your funders—for implementing the idea more widely. As you scale up, continue to run experiments on ways to increase efficiency and to create additional value for your constituents.
Hypothesis-Driven Product Management
It’s no longer good enough for a Product Manager to say, “I think users want this feature.” Instead, you need to ask, “What outcome do we predict this feature will have?” and validate your answer with empirical data.
Hypothesis-driven product management is the practice of treating the development of new products as running a series of experiments. Instead of formulating requirements, we formulate hypotheses along with some validation criteria that state how strong of a signal we need to consider the hypothesis true. We use what we learn from each experiment to iterate on our ideas until we get where we want to be, or, until we determine that the product isn’t viable and cancel the effort.
Experiments Test Our Assumptions
A lean experiment is the smallest experiment we can run to quickly test our assumption. We start small and fast and then increase the scale and scope of our experiments over time.
An experiment consists of three parts: a hypothesis, a test, and validation criteria.
The hypothesis is a falsifiable version of our assumption. Remember to make sure you are only testing one variable in each hypothesis, otherwise, you won’t get reliable data.
The test is how we intend to test our hypothesis and provide it true or false.
The validation criteria are the signal we need to see to consider the hypothesis true.
Types of experiments:
A/B Test A comparison of two versions of a product or feature to see which one performs best. Works best with large sets of users for small incremental optimizations of an experience and business model.
Concierge Test A technique to replace a complex automated technical solution with humans who directly interact with the customer. Helps us validate whether anyone wants our product.
Wizard of Oz Test A technique to replace the product backend with humans. The customer believes they are interacting with an automated solution. Helps us validate whether anyone wants our product.
Smoke Test is Commonly a website that describes the product’s value proposition and asks customers to sign up for the product before it’s available. Helps us validate whether anyone wants our product.
The Scientific Method
Make observations
Formulate a hypothesis
Design an experiment to test the hypothesis
State the indicators to evaluate the result of the experiment
Conduct the experiment
Evaluate the results of the experiment
Accept or reject the hypothesis
If necessary, make and test a new hypothesis
Validating your experiment
The questions you should ask before you start validating your experiment are the following
1. Validate the problem. Is this a problem worth solving?
2. Validate the market. Some users might agree that this is a problem worth solving. But are there enough of them to make up a market for your product?
3. Validate the product/solution. The problem might exist, but does your product actually solve it?
4. Validate willingness to pay. There might be market demand and a great product. But will people actually be willing to reach into their wallets and pay for it?
Experiments
Every good experiment has four things :
Observation – Something we can see or notice
Hypothesis – a restatement of the assumption, to start with the words “we believe that…”
Test – the thing you’re going to do or build to validate the hypothesis through empirical evidence
Evidence – a metric that clearly shows whether your hypothesis is correct or incorrect
Case: Boosting E-commerce Conversion Rates : A Lean Experiment Case Study
Title: Increasing Website Conversion Rates through Personalized Product Recommendations
Abstract: This lean experiment aims to increase the conversion rates of an e-commerce website by testing the effectiveness of personalized product recommendations. The hypothesis is that by offering personalized recommendations based on user behavior and preferences, we can increase the likelihood of users making a purchase.
Observation: The current website has a high bounce rate and low conversion rate. Users are spending a lot of time browsing the website, but they are not making purchases.
Hypothesis: We believe that by offering personalized product recommendations based on user behavior and preferences, we can increase the likelihood of users making a purchase. This will be done by implementing a recommendation engine that analyzes user data and offers personalized product recommendations on the website.
Test: We will build and implement a recommendation engine that analyzes user data and offers personalized product recommendations on the website. The engine will be trained using historical user data and real-time user behavior data. We will then monitor the website conversion rate and compare it to the previous conversion rate before implementing the recommendation engine.
Evidence: The metric we will use to measure the success of the recommendation engine is the conversion rate. We will compare the conversion rate before and after implementing the recommendation engine. If the conversion rate increases, then our hypothesis is correct. If the conversion rate remains the same or decreases, then we will need to re-evaluate our hypothesis and make changes to the recommendation engine.
Observation – The current website has a high bounce rate and low conversion rate. Users are spending a lot of time browsing the website, but they are not making purchases.
Hypothesis – We believe that by offering personalized product recommendations based on user behavior and preferences, we can increase the likelihood of users making a purchase. This will be done by implementing a recommendation engine that analyzes user data and offers personalized product recommendations on the website.
Test – We will build and implement a recommendation engine that analyzes user data and offers personalized product recommendations on the website. The engine will be trained using historical user data and real-time user behavior data.
Evidence – The metric we will use to measure the success of the recommendation engine is the conversion rate. We will compare the conversion rate before and after implementing the recommendation engine. If the conversion rate increases, then our hypothesis is correct. If the conversion rate remains the same or decreases, then we will need to re-evaluate our hypothesis and make changes to the recommendation engine.
Case: Amazon Lean Experiment: Wizard of Oz
Retail giant Amazon is another big-name platform that grew thanks to the power of an MVP.
Founder Jeff Bezos originally came up with the name Amazon because it’s the largest river in the world.
It was the perfect name for his vision for his company the biggest bookstore in the world.
Source: Dittofi
Bezos decided to sell books first because they were readily available, were easy to ship at a low cost, and had a worldwide demand.
Amazon’s MVP adopted a Wizard of Oz approach. This refers to an app with a manual backend that the user doesn’t know about.
🔥 Top three quotes from our Instagram page
If everything is important, then nothing is. - Patrick Lencioni
Efficiency is doing things right; effectiveness is doing the right things - Peter Drucker
Design is not just what it looks like and feels like. Design is how it works.
- Steve Jobs