Market research firms use A-B testing, also called split testing, to evaluate competing ideas with reliable data. You can use A-B tests to answer questions like which message resonates better, which product concept appeals more, and which call to action gets more responses. Market researchers use A-B testing to answer questions like these with evidence rather than opinions.

Defining A-B testing

At its simplest, the A-B test asks audiences to react to two options, then analyzes the responses to find  the stronger one.  Split testing is a controlled way of comparing:

  • Advertising messages
  • Products or product concepts
  • Brand positioning statements
  • Website designs
  • Landing page design, content, and flow
  • Pricing structures or offers
  • Package designs
  • Email subject lines and body copy
  • Social media posts
  • Calls to action
  • Customer experiences and journeys
  • Sales propositions

Testing can compare the two options in terms of understanding, relevance, appeal, purchase intent, preference, perceived differences, and likelihood to recommend, among other things. What makes A-B testing so useful is that it can be incorporated into quantitative surveys, usability studies, digital experiments, product testing and other types of primary research.

Benefits of A-B testing

Brands and market research firms use A-B testing to gain key benefits:

  • Evidence-based decision-making: Instead of making a guess, you have insight into how people respond and which option performs best
  • Reduces risk: Testing before investing identifies problems early and reduces the cost of making the wrong choice
  • Optimizes marketing effectiveness: Testing multiple options can identify those with the greatest potential for success.
  • Speeds decision-making: Well-structured A-B tests help you quickly winnow out weaker options and find the strongest ones.
  • Supports continuous improvement: Every test generates results that your team can use on the next iteration and the next
  • Uncovers unexpected insights: Good A-B test techniques don’t just uncover what performs better, they explore why, challenge assumptions, and uncover opportunities

A-B testing in action—examples of successful tests

The value of A-B testing is best illustrated by looking at how companies that have successfully used it. Here are examples from multiple industries where A-B testing generated useful insights and informed decisions:

Procter & Gamble: Improving the online experience for Crest

You don’t have to own the whole customer journey to use A-B testing. While retailers own the transaction, P&G still has a stake in how Crest is discovered, evaluated and selected on the retailers’ digital shelves. P&G used A-B test to compare options for product descriptions, benefits and claims, images and video, and more. Crest brand managers could use the insights to optimize key elements. The testing reportedly found that adding social-proof messaging to a Crest product page increased conversion by 25% and revenue by 31%.

Netflix tests content presentation

Netflix is well-known for its relentless research and learning culture. It uses A-B testing for improving the user experience and driving up watch time. It found that small changes in artwork, images, titles and how content is presented to different audiences influence viewing behavior. Given Netflix’s scale, modest improvements based on A-B test results can have significant impact on the business.

NYU Langone Health: Improving clinical decision support

NYU Langone uses electronic health record alerts to remind clinicians to recommend vaccinations to patients. The problem was, doctors were getting so many alerts that “alert fatigue” set in, with busy doctors too often ignoring them. Researchers used A-B testing to compare different versions of a vaccination alert and measure how clinicians responded. The first change produced little improvement, but the results gave researchers information they could use to refine the approach. Further testing ultimately reduced the number of alerts from 23.1 to 7.3 per patient per day without reducing vaccination orders. The key finding that more alerts didn’t produce better results and just added to alert overload to no benefit.

Will A-B testing be replaced by AI?

Nearly every conversation about AI discusses the possibility that it will replace human effort and thinking. That is almost never the case. Artificial intelligence will change A-B testing and all market research methodologies. It won’t eliminate it.

The better question is: How will A-B testing be changed by AI? Here are some ways AI is already being used:

  • Generating more options faster: That means being able to test more headlines, messages, concepts and creative treatments for more insights and doing it quickly.
  • Creating realistic stimuli: It is possible to mock up ads, packages, web pages, and other stimuli very quickly and at low cost for more meaningful audience reactions.
  • Reading and analyzing test results: It can spot patterns, group audiences, and even suggest more ideas to further investigate.
  • Speeding up testing itself. AI-powered testing can make it possible to run more experiments than traditional manually fielded tests. AI makes it possible to test more variations in less time.

Recommendations for getting more value from A-B testing

Thinking about A-B testing as a strategic research methodology, you can get the most value by keeping several principles in mind:

  • Define the decision the research needs to inform before starting, and define what success looks like from the outset.
  • Change one meaningful variable at a time to get a clear reading without muddying the test with multiple options.
  • Recruit a sample audience that clearly reflects customers and decision-makers.
  • Don’t limit testing to preference. Tie it to behavior.
  • Combine A-B testing with qualitative and quantitative research for a fuller picture and to lower risk.
  • Use AI as a tool to cost-effectively accelerate and expand your testing, recognizing it is not a substitute for human thinking

A-B testing is a simple concept that is highly strategic. It provides invaluable learning to inform both major decisions and incremental tweaks

Frequently asked questions

What is A-B testing?

A-B testing compares two versions of a product, message, concept, experience or other stimulus with comparable audiences to determine which performs better. The results provide evidence that can inform marketing, product and strategic decisions.

How is A-B testing used in market research?

Market research firms can use A-B tests to compare concepts, positioning, messaging, advertising, packaging, pricing and other alternatives. Testing can reveal which option generates the strongest response among a target audience.

What are the benefits of A-B testing?

A-B testing can replace assumptions with evidence, reduce the risk of making the wrong decision and identify what resonates with specific audiences. It can also uncover insights that internal teams may not anticipate.

Can A-B testing be used for strategic decisions?

Yes. A-B tests can help companies evaluate strategic alternatives such as value propositions, market positioning, pricing approaches and go-to-market messages before making significant investments. It is also valuable for fine-tuning, making small improvements that add up with scale.

Will AI replace A-B testing?

AI can make A-B testing faster by generating test alternatives, identifying patterns and analyzing results, but it cannot eliminate the need to validate ideas with real audiences. In many cases, AI is likely to increase the value of A-B tests by making it possible to test more hypotheses more quickly.