A recent study found that startups using A/B testing scale faster, launch more products, and attract more venture capital.
However, not all startups benefit from these benefits. Early-stage startups using an experimental strategy typically exit more quickly, reflecting management’s lack of experience with how to apply the methodology. Another possibility? Their idea wasn’t very good.
“One of the coolest findings in our paper is that experimentation doesn’t necessarily drive startup growth,” said Rembrand Koning, assistant professor of strategy at Harvard Business School. “Instead, we find that companies that adopt A/B testing scale faster and fail. Sometimes founders just have a bad idea and in these cases experimentation can move you to the next opportunity faster.”
In fact, companies of all types have adopted popular experimental methods, including design thinking and lean startup approaches, to create some of the world’s largest tech companies. But can a Silicon Valley mantra that encourages early testing of new ideas and products really help companies succeed?
To confirm this, Koning, along with Sharique Hasan and Aaron Chatterji of Duke University’s Fuqua School of Business, looked at 35,262 high-tech startups established worldwide between 2008 and 2013.
Researchers analyzed startup characteristics, web metrics, and technology adoption trajectories over a four-year period from April 2015 to determine whether, but how, A/B testing impacted company performance. What they found was that the simple and low cost of using A/B testing to make incremental changes to a website, such as improving design and presentation elements, increased their focus on big-picture strategies and innovation. will be
“By making all these little improvements a lot easier, A/B testing also frees up time for companies to tinker with the little things, which we now delegate to our A/B testing tools.” he says “Founders can now think about more radical changes like innovations and new products. [A/B testing adoption] doesn’t sacrifice big ideas as it drives more changes overall.”
On average, companies that used A/B testing to select the best features for their website had higher page views, with visitors who browsed more pages and stayed longer. However, these companies have reached natural endpoints more quickly, such as expansion or failure.
“Companies that have adopted A/B testing are more likely to reach zero pageviews, which they consider as evidence of going out of business. But they ae also more likely to have extreme pageviews, resulting in well over 50,000 views per week..” ‘ says Corning.
Separating The Effects Of A/B Testing
Researchers used data publicly available via the website BuiltWith to identify companies up to eight years old with A/B testing software installed. These tools allow businesses to randomly assign different versions of their website to different visitors to compare their performance.
Because BuiltWith data shows when each company installed the software, researchers were able to use data on company SimilarWeb’s weekly website page views to study the company’s performance before and after the start of testing.
Researchers mitigated the sample’s potential for selection bias by comparing each company themselves before and after the test software was implemented. For example, a company that chooses A/B testing is likely inherently better. To determine whether A/B testing led to more product launches or better VC funding, the researchers looked at news articles and CrunchBase data.
From 2015 to 2018, 4,645 startups adopted A/B testing technology on their websites. On average, these companies had about 10% more weekly pageviews, were 5% more likely to raise VC funding, and launched 9-18% more products.