5 Keys to Developing a Culture of Experimentation

Some industry watchers estimate that product-led brands grow 10X faster. With a lot riding on product decisions, product managers must validate assumptions, test new ideas and continually improve products. Rapid experimentation is their essential tool.

And yet, only four out of 10 marketers view testing and experimentation as a critical success factor. According to the Harvard Business Review, “As companies try to scale up their online experimentation capacity, they often find that the obstacles are not tools and technology but shared behaviors, beliefs, and values.” As HBR explains, “For every experiment that succeeds, nearly 10 don’t — and in the eyes of many organizations that emphasize efficiency, predictability, and ‘winning,’ those failures are wasteful.” 

What’s needed is a different set of shared behaviors, beliefs and values — a culture of experimentation.

Experimentation has driven some of the greatest market breakthroughs of their day, like Subway’s $5 footlong and McDonald’s all-day breakfast, as well as the creation of juggernauts like Amazon Web Services, which generated $80 billion in 2022. Companies with a true culture of experimentation — such as Amazon, Google, Facebook, Netflix and booking.com — run thousands of tests each year, not just a few times like many other companies. Booking.com, for example, has quadrillions (millions of billions) of landing-page permutations live at any given time. 

Hear more about Booking.com’s emphasis on experimentation by tuning in to a recent Masters of MAX podcast. 

How to build a culture of experimentation
Developing an experimentation culture differs from developing an experimentation strategy.

An experimentation culture requires support from the executive level and all teams to experiment constantly and reflexively. There must be agreement that experimentation will lead to better results, and everyone must be fully behind it. An experimentation strategy is also important, but its focus is on defining the goals, KPIs, hypotheses and success criteria for experiments.

Here are 5 keys to developing a culture of experimentation, given our experience working with hundreds of brands across dozens of industries for more than a decade.:

1.  Get executive buy-in
To take root, a culture of experimentation needs a champion at the top. The most important buy-in is around data-driven decision-making and a willingness to embrace failure, and that buy-in needs to occur at the leadership level. 

After buy-in, everything about a culture of experimentation flows from leadership initiatives: communicating the vision, aligning with business goals, and funding and resources. Ultimately, leadership will need to sign off on roles, skill sets, best practices, technology tools and KPIs. 

2. Remove barriers to testing 
Initiating and refining tests — possibly hundreds or thousands of them at any given time — is  impossible if you have to rely on developers alone. 

All of your employees must have access to tools they can use themselves — such as Airship’s A/B testing capabilities across messages, journeys and no-code app experiences, which don’t require ongoing developer support. 

3. Empower everyone to test
In addition to making experimentation tools and technology readily available, the organization should also authorize and then encourage all employees to use the tools. A culture of experimentation empowers everyone to test hypotheses and test fast. Think of experimentation as a daily practice for marketers, where everyone can and should be involved. When anyone has a hypothesis that needs testing, they should be able to formulate an experiment and carry it out quickly. It should be a safe environment for employees to use their imaginations, go out on a limb, and celebrate experiments and findings throughout the company. 

As a corollary, organizations need to create a mechanism for testers to align pre-experimentation so as not to break or interfere with other experiments. 

4. Accept that some experiment hypotheses will fail 
In a culture of experimentation, people see failure in experiments not as wasted tests, but as important opportunities for learning. If A is the control and B the modification, they can’t both win. A corollary of this is that the more tests you run, the greater the chances of finding a winner. Further, the earlier in development you start testing, the sooner you’ll be able to focus resources on the best options. 

Netflix blogs about its institutional acceptance of failure: 

The low win rate in our experimentation program is both humbling and empowering. It’s hard to maintain a big ego when anyone at the company can look at the data and see all the big ideas and investments that have ultimately not panned out. But nothing proves the value of decision making through experimentation like seeing ideas that all the experts were bullish on voted down by member actions in A/B tests — and seeing a minor tweak to a sign up flow turn out to be a massive revenue generator.

— Netflix Technology Blog, 2022

5. The document learnings
Set up a centralized, visual reporting environment that provides real-time insights from all active experiments, as well as customer, channel and campaign data, all in one place. Include a dashboard to track mobile app KPIs across the app lifecycle so you can quickly and easily gain insights into your app performance. 

Everyone should have  a common understanding of the end-to-end optimization strategy, and know what factors lift performance across various experiments. While these winning factors may be unique in every situation, they establish a shared history to continue to engrain your organization’s culture of experimentation.

Register to join two leading brand experts as they explore how continuous experimentation and optimization drives better results for businesses in a December 5 webinar.