Home / Front Page / A/B Test Results

A/B Test Results

Headline and image A/B tests within Front Page allow you to optimize the performance of an article on your homepage, and over time they reveal patterns about the most engaging ways to present your articles to attract and retain loyal readers.

This article explains how to interpret the A/B test results after the test has completed and provides tips for creating tests that are set up for success.

Understanding A/B Test Results

Results for completed tests for the past 30 days can be found in the A/B History dashboard, linked in the left-hand nav bar under Front Page.

Here’s a great example of a highly successful headline test. This test shows how even a great starting headline can potentially be further improved through workshopping.

Let’s break down the results.

First of all, the bar at the top of the test confirms that a winning variant was “chosen based on clear performance increase.” This means the measured results met the statistical significance gold standard. The “Time on Position” column tells us it took just over 18 minutes to collect enough data to declare a winner.

The CTR column has two rows for each variant. The top row shows the actual CTR for that variant during the test (clicks / homepage impressions) and the lower row shows the margin of error.

Pro Tip

Hovering your mouse over the CTR for each variant will reveal the exact number of clicks the variant accumulated during the test.

Analyzing Results

For the example shown above, the baseline A variant had a CTR of 1.54%, and the B variant showed significant improvement with a 2.1% CTR, but it was the C variant that won the day with a 2.96% CTR.

In other words, the C variant was getting about twice as many clicks as the A variant!

These metrics reveal the evolution of a good headline to a great headline.

✅ “Police want to honor heroes for rescuing swimmer, 2”: Excellent start but the old fashioned newspaper headline format of putting the age after a comma is slightly stuffy

✅ ✅ “Toddler’s scary rescue in Livonia hotel pool caught on camera”: This flows better and reveals that there is video footage of the incident

✅ ✅ ✅ “Toddler was at the bottom of a Livonia hotel pool. Cameras caught what happened next.”: The dramatic two-sentence format is perfect for this equally dramatic scenario, and the video footage promises to satisfy the curiosity gap

Best Practices

A/B testing relies on capturing as much data as possible in a short period of time to achieve statistical significance. As a general rule of thumb, homepages that have more than 3 million monthly pageviews should be able to regularly run successful tests on various homepage positions.

For homepages with smaller audiences, there are a few tricks to running tests that achieve statistically significant results:

  1. Hero slot: Focus tests on the most visible homepage position. This is where most of the eyeballs go, so it gets an outsized percent of the impressions and clicks.
  2. Primetime: Test during the time(s) of day when your homepage audience peaks. This is different for each site, but you can see homepage PVs over time in the Front Page dashboard.
  3. Variants: Limit tests to just 2 total variants. The more variants you create, the more the impressions will have to be divided up, necessitating more time to identify statistically relevant engagement differences between them.
  4. Distinction: Make sure there’s true variety in your. Subtle differences in headlines will not lead to dramatically different click behavior from readers. A/B tests are a great vehicle to experiment and flex your creativity!

Reasons Why Tests Fail

Not all A/B tests will end in the successful “winner chosen” conclusion. Some will wind up being inconclusive or get stopped for various reasons.

An “inconclusive” result means that no statistically significant winning variant could be identified within the 40-minute max test window.

Pro Tip

Even an inconclusive test can reveal valuable insights. Hover your mouse over the CTR for each variant to see how many clicks each accumulated.

Tests may also be “Stopped” for several reasons:

  1. A Newsroom user chose to stop the test within Front Page.
  2. The headline or image of the article was changed in the site’s CMS during the test.
  3. The article being tested was moved to a different region of the page while the test was in progress.
  4. Other technical issues can also interrupt tests. Our system is designed to be extremely safe, so if it gets confused at all about whether or not the correct article is in the correct slot, it will immediately stop the test.

To get started with A/B testing, contact your Newsroom Engagement Manager or newsroom-help@taboola.com for instructions on how to implement Front Page.