How Experimentation & Segmentation Help Drive Better Results

Brand leaders have thousands of experiments running at any given moment. Why? Because their businesses depend on optimizing every customer interaction. A works better than B. A₁ works better than A₂. And so on. Every day, day after day. It’s the behind-the-scenes rhythm of testing that helps everyone make better decisions.

While experimentation is widely understood as a way to help craft experiences that appeal to audiences, marketers can also test the audiences themselves. What’s the best audience segment to receive certain content or offers?

Why Segmentation Is Important

Every marketing organization is constantly on the hunt for the best and biggest audience for their products or services. They want to find people who fit the ideal profile as easily and affordably as possible.

For owned channels — where you know the individual users or addresses you’re speaking to — you need to sub-segment your user base into discrete groups based on channel engagement, purchase history, propensity models from your data science teams, and other criteria, depending on your goals. The identification of and marketing to these groups is what we mean by segmentation. 

The ultimate goal is to send to these specific audiences marketing communications that encourage further engagement with your brand — the right message, at the right time, to the right audience on the right channel. This is the holy grail of owned channel marketing. 

A segment should be dynamic, where users can flow into and out of that segment in real time based on changes in their behavior (or lack thereof) with your brand. Good segmentation drives higher engagement, which drives better business results.

Why Segment Experimentation Is Necessary

One of the keys to segmentation is experimenting with what makes an effective segment. You’ll likely have a hypothesis about what makes a logical collection of users based on the information you have about them and what kind of messaging you think they’ll find interesting. You’ll put all the pieces together to create an audience, craft a message and then send it out hoping the audience finds the message engaging. Often this works exceptionally well based on leading indicators like opens, reads and clicks, but you won’t always know if the message is driving improvement in your core metrics.

You need to be able to test your hypothesis. We’ve already talked about a few variables in a message that you can test, such as audience, content, time and the channel. To measure results and know what’s working we need to be able to test these and other variables individually. Otherwise, if we get a great result (or a bad one) we won’t know which variable had the biggest impact. 

Testing each attribute individually to deliver better results can seem like a daunting task. It doesn’t have to be if you have the right tools — and the organizational motivation to do so.

It Helps to Have a Culture of Experimentation in Your Organization

I’ve lost track of the number of times I’ve heard someone in a marketing role say, “We didn’t have time to test it … we just needed to get it live.” 

Establishing a culture of experimentation in an organization requires that everyone slow down, think about the pieces that make up a marketing campaign, messaging series or product update, and ask the obvious questions: Why are we sending this out at noon on a Tuesday? Why do we have two buttons instead of just one? Why are/aren’t we including emoji? Why are we sending an SMS first instead of an email or an app push? Ideally, these conversations include analytics teams, marketers and even product teams to make sure they account for all the variables that could affect who gets your message and how likely they are to engage with it.

Getting the best results requires that you generate data and make data-driven decisions. To generate that data, you’ll need to create experiments that allow you to test your assumptions. Creating a control group of a small percentage of your audience will allow you to test the effectiveness of a particular channel. Creating an A/B test between different versions of the message will allow you to see if your users respond better to one type of message versus another. Creating two segments with slightly different audience criteria (for example people who spent up to $50 versus those who spent over $200) will allow you to see if your audience selection is too narrow or not narrow enough. Again, it’s important to test each hypothesis specifically to ensure you can differentiate the results. 

Once you’ve run your experiment, you should have a wealth of data you can use to understand business results and inform the next test, and so on. But how do you build an experiment and how would you analyze the data?

It Helps to Have the Right Tools 

It’s possible to build experiments manually. You can “cut a list” from your customer database of people who meet certain criteria, split that list to create A/B segments or control groups, and then import that list into your marketing tools. However, there are a few drawbacks to this approach:

• You can only target the customers who are in your database at the time you cut the list. You’ll be excluding anyone who’s signed up for your email program or SMS messaging, or installed your app since the list was cut. 
• Measuring results can be laborious depending on what tools you have at your disposal. Extracting data is time-consuming. Transforming it and loading it into your database/lake/warehouse for analysis can be very manual.
• Timely follow-up messaging based on customer engagement is not usually not an option when cutting lists. For example, if I wanted to send a better offer to the people who didn’t engage with my email, I’d have to first go back to my database and pull the users who meet those criteria. 

At Airship, we’re always working to make sure that creating audience segments, holdouts and A/B experiments is easy for our platform users. We believe that using the right tools is critical to helping you make decisions that benefit the customer and the brand. Last week, we announced a group of experimentation capabilities that enable brands to enhance the customer experience with more personal, more timely and overall, more delightful interactions. 

Airship’s segmentation engine makes it easy to create detailed cohorts of customers based on all the data you’ve collected in Airship, such as attributes, events and device/channel information. With our mission of delivering everything you need to capture customer value inside and outside the app™, we’ve expanded the technology for testing to bring you next-level experimentation. 

Airship’s Experimentation Hub allows you to create, launch and evaluate experiments to give you a common understanding of how best to optimize your campaigns, end-to-end. You can easily build an A/B experiment to test segments, as well as variants of message copy or content.

You can also create global holdout groups to measure the overall effectiveness of your marketing campaigns in Airship against your defined goals and metrics. Turn customer actions into goals for each experiment and watch which version leads to more conversions. 

When it comes to getting useful data out of Airship, this task is made easy by our Real-Time Data Stream and Performance Analytics tools. You can add Campaign Categories to your messaging, which makes it much easier to match up the Airship data with the rest of your marketing stack by including your own campaign or experiment identifiers in the results. 

With our Fall Launch updates to Airship’s Performance Analytics tool, you can leverage visually dynamic reporting for Goals you create, and you can track success across each phase of the customer lifecycle with App Health Reporting, making it easier to optimize your campaigns. 

Want to learn more about using Airship’s experimentation capabilities? Register for our December 5th webinar where we’ll hear from brand leaders on how to use experimentation to systematically fine-tune everything and anything you share with customers. Register Now 

Navigator is Airship’s customer newsletter covering the latest mobile industry trends, product updates, use cases and best practices, and other learning resources. It’s yet another resource to help you deliver better mobile experiences and create greater value more quickly. If you’d like to receive our monthly Navigator newsletter please sign up here.