A/B Testing 2.0: Continuous Experimentation

Out with A/B Testing, In with Continuous Experimentation

In the iconic movie Glengarry Glen Ross, four real estate salesmen go above and beyond to sell their Florida developments, and their rule of thumb was this: ABC, or, “always be closing.” For marketers, we hold just as tightly to our own version of the phrase: ABT, “always be testing.”

This mantra is so familiar to marketers because testing different iterations of the customer experience is a proven route to improving metrics and business results. Whether it’s personalizing a website, mobile app, email campaign, or even direct mail, creating different variations and analyzing which performs best is at the heart of data-driven marketing.

But while A/B testing—and multivariate testing—is a powerful tool for marketers to inform their decision making, it is not always the best solution for making improvements within email marketing programs. Brands also need a way to test the best experience for each person and then improve over time.

The Challenge With A/B Testing

As technology has evolved, A/B testing has become much easier to build and deploy, and measuring results between control versions and new variations has become far more efficient. But while this process has greatly improved, traditional A/B testing tends to be geared towards specific business use cases. It tests creative variations determined by a marketer—background images or CTA button text and colors to name a few—within a single campaign that’s split across different audiences.

While this approach will help marketers determine which of the variations tested delivers the most clicks, the method is at odds with what we know about personalization. More than any other recent marketing strategy, 1:1 personalization has proven time and again its unparalleled effectiveness for not only improving short-term engagement, but long-term loyalty. However, the results of A/B testing can only yield the best message variation for the majority of people.

Traditional A/B testing approaches simply aren't designed to test marketing messages that are personalized to individuals, and improving those experiences beyond a single interaction—such as building and maintaining customer relationships over a series of messages—remains out of reach. Additionally, in order to test for multiple short- and long-term objectives simultaneously at an individual level, marketers need to update their toolkit to include continuous experimentation.

What Does Continuous Experimentation Mean?

Continuous experimentation isn’t just another superficial marketing buzzword. It means embracing AI to constantly learn from each customer experience and implement those changes for the next message, and the following messages after that. Rather than A/B testing one experience and analyzing and implementing those results, AI can be used to create personalized email variations—tens of thousands of them—that automatically adjust based on each recipient's unique response or lack thereof.

Just think of how this new approach to testing can enhance marketers’ most common email program: the batch send. Within one campaign, active customers receive individualized experiences designed to improve product discovery across a brand’s catalog while dormant customers receive entirely different personalized messages designed to drive re-engagement. The results of each experiment are then automatically factored into the next campaign, improving performance over time, for both the brand and each recipient.

In addition to experimenting with email variations, solutions like Movable Ink Da Vinci will also optimize for multiple objectives at once (clicks, conversions, discovery, engagement), rather than a single element chosen by the marketer, as is common with A/B testing. This unlocks far greater potential for nurturing your entire email list with each send.

L.L.Bean’s Approach

A personalized journey through every email sounds too good to be true. And if you’re skeptical of yet another claim to fame from AI, who can blame you? AI is everywhere in martech, and some of the promises can seem inflated when a new product is simply re-labelled GPT.

But creating personalized content and cadences while generating greater insights from testing is possible. Just take a look at how L.L.Bean is achieving it today through Da Vinci. 

L.L.Bean is a retail icon with countless customers, and today they send hundreds of thousands of campaign iterations daily all through the power of multi-zone personalization and AI automation. A few short years ago, adding this level of nuance to marketing messages at scale would’ve been impossible, but today, retailers like L.L.Bean can test and optimize for each customer’s unique preferences and habits. Over time, these insights and optimization foster lifetime value, increase loyalty, and reduce reliance on promotions.

Ultimately, L.LBean’s email production workflow is not only efficient and quick-to-market, it uses AI to guide customers through individualized journeys to discover new products all while providing the marketing team with insights into which creative assets perform best.

Take Testing to the Next Level

In a world where personalization is key, using advanced solutions like Movable Ink Da Vinci to drive continuous experimentation is how marketers will keep their edge in a busy marketplace. By building upon existing testing structures and creating truly personalized experiences, marketers will drive lasting engagement and results.

Curious to learn more about the AI solution that’s built for marketers’ most pressing challenges? See Movable Ink Da Vinci in action, or check out the related resources below.