December 01, 2018
Guide to a/b testing for beginners

Which version of these introductory sentences do you prefer?

(A): “Online store owners must conduct tests of their marketing elements with scientific precision.”

(B): “Online store owners are really just mad alchemists—standing in the lab of their websites, trying to find the magical elixir for converting website visitors into customers.”

Version A is a little dry, but it delivers the point quickly. Version B is more engaging, but it also takes longer to read. Maybe you like your sentences to get to the point, or maybe you like them more poetic. (Remember your answer. You’ll need it later.) Either way, you just participated in a round of A/B testing. We hope you feel smarter.

As both of these sentences suggest, A/B testing is a scientific approach that every marketer and online store owner needs to master to improve their conversion rate optimization (CRO). Why? Because it’s a proven strategy for converting more customers and selling more products and services.

Now that we’ve covered a real-world example of A/B testing, let’s look at an academic definition.

What Is A/B Testing?

A/B testing is the process of identifying the more preferred version of two or more examples of marketing content. During the A/B testing process, marketers run versions of a web page, observe which gets more interaction (e.g., likes, clicks, views), then use that version to optimize conversion rates. A/B testing is also known as split testing, variant testing, or bucket testing.

Although web pages are a common subject of A/B testing, you can split test virtually any marketing element. At the micro-level, a store owner could try varying the color of a “Buy Now” button to see which one got more clicks. At the macro-level, a firm could run two versions of their checkout process to see which one resulted in fewer carts abandoned.

Why A/B Test at All?

A/B testing is not just a thing you do. It’s an attitude towards your business that keeps you dedicated to constant improvement. Plus, split testing is simple to execute, affordable to maintain, and applicable to every part of your business. These qualities make it a low-risk endeavor. A/B testing is the way you can fail fast and break small things. Use it to improve these areas of your business:

  • Engagement
  • Traffic
  • Marketing ROI
  • Bounce rates
  • Conversion rates
  • Sales
  • Content quality

The A/B Testing Process

A/B testing is basically an experiment following the scientific method. Here are the seven basic steps of A/B testing.

1. Identify What You Want to Test

Gather what conversion rate optimization (CRO) data you currently have. What specific conversions are you trying to raise? More subscriptions? Higher email opens? Increase your newsletter sign-ups? Now, identify the marketing asset responsible for these conversions—like form field design, email copy, and landing page layout.

2. Decide What “Success” Looks Like

You’re doing testing because you’re dedicated to improving your site. So, you know what underperformance looks like. But what does success look like? You have to decide on that before you continue with testing. Otherwise, you can’t correctly interpret the results. You won’t have any data points to compare. Is it a percentage increase in page opens you’re after? Maybe it’s a rise in revenue over the quarter. It could simply be a true vanity metric, like more positive customer feedback on your website redesign. Whatever “success” looks like, decide now.

3. Find a New Strategy

You may know what your goals are, but do you know how to get there? Start by gathering best practices. For example, if you want to split test a landing page, include best practices for a highly converting landing page. Maybe you want to try out a new blog post format. Make some hypotheses about your outcomes that match up with your success goals.

4. Create a “B” Version of the Original

Create a variation of the asset you want to test. Find online tools like a CTA Button Generator or Headline Analyzer to test out new approaches. If you’re using WooCommerce, try an A/B testing plugin to manage your tests. Or use complete A/B testing platforms like Unbounce or Optimizely, which have easy-to-use page builders that don’t require developer resources or site integration. Of course, you can have a version C, D, E,…, which can give you more data points, but that’s more creating, tracking, and analyzing you have to do.

5. Deploy Your New Strategy

Run the test and take measurements. Randomly assign visitors to interact with your control version (A) and experimental (B) version. To keep your test valid, control all variables external to any changes you’re making.

For example, you’ll usually want to test both variations at the same time and for the same duration, unless time is what you’re testing. If possible, target customers using the same browser and devices (e.g., Chrome and Desktop). Different browsers can vary in page load speeds depending on bugs, fixes, and software versions.

6. Analyze the Data

Now that you’ve run your experiment, it’s time to look at the results. But just because landing page A got more newsletter sign-ups than B doesn’t mean you’ve got a winner. The difference in the two may not be statistically significant. You can’t control for every variable, and randomness could be the explanation for different results. Upload your data to a free A/B test calculator to see if your results are actually meaningful and not the result of random chance.

7. Repeat Ad Infinitum

If you want real results out of your CRO strategy, make A/B testing a mainstay of your overall marketing efforts. A/B testing is never finished, only abandoned. It’s a simple idea: The more you optimize, the better your conversion rate will get.

A/B Testing Example

Here’s a real-world example of the A/B testing process at work. The analytics company ComScore wanted to increase leads for their sales team through social proof and customer testimonials. So, the company decided to A/B test around the effectiveness of these campaigns.

ComScore’s Director of Web Marketing, Ferry Gijzel, began by gathering existing company data. Gijzel ordered an audit of demo requests to see how effective the company’s lead forms on their product pages were at generating requests.

The audit revealed lower numbers than were expected—something definitely needed changing. But change how? The answer lay in A/B test results.

Gijzel and team investigated their existing product page design, researched ways to adjust them, and hypothesized probable outcomes.

At the time, ComScore’s pages included a product description, customer testimonial section, and a CTA for a demo request. Gijzel hypothesized that making the customer testimonials more prominent on the page would lead to an increase in demo requests. If more prospective leads saw a recognizable brand singing ComScore’s praises, then demo requests should increase.

The ComScore marketing team created three product page variations—changing the placement of customer logos on the page—and kept their current one as a control. Then they tested the new variations on 2,500 new visitors and tracked the page views of the last step in the funnel—the “Demo Request Confirmation” page.

The A/B test results showed a clear winner—a variation using a vertical layout with a prominently displayed logo on top of the testimonial. The new product page increased leads by 69% compared to the original one.

Google-Safe A/B Testing

When running multiple variations of a landing page, you’re redirecting traffic to multiple URLs. If you’re thinking that A/B testing might affect your SEO ranking, you’re right. Google understands and advocates for split testing. But you have to follow Google’s Webmaster rules for A/B testing if you want to keep from being penalized. If you follow the rules, you’re good to go.

Here’s how:

No Cloaking

Cloaking is the practice of presenting different content or URLs to human visitors versus Google Bots. Basically, Google doesn’t want you dividing your audience up this way because it can’t accurately “read” your pages if you send its bots to different content than your human users see. The mismatch creates inconsistent search results. Cloaking is banned by Google’s Webmaster Guidelines and will get you demoted and possibly removed from the search engine completely.

Use rel=“canonical”

When running an A/B test with multiple URLs, use the rel=“canonical” link attribute on all of your alternate URLs to show that the original URL is the preferred version. Google recommends rel=“canonical” rather than a noindex meta tag because it matches your intentions better.

Avoid 301s, Use 302s

When your A/B test redirects users from your original URL to a variation URL, use a temporary (302) redirect not a permanent (301) redirect. This tells Google you’re only diverting traffic for a short amount of time—that you’re experimenting. The 302 will keep the original URL in Google’s index rather than replacing it with the test page.

Keep Your Tests Short

Google advises site owners to keep testing periods as short as possible. The search engine suggests two weeks as a guideline for most tests.

Once the test is finished, update your content to the preferred versions and remove all of the other variants. If you wait too long, Google will think you’re trying to trick them.

Google gets especially suspicious when you have only one variant that you’re serving to a large percentage of your visitors.

A/B Testing Tools

There are plenty of A/B testing tool options to choose from depending on the size of your site and testing needs. These tools fall into two categories.

A/B Testing Platforms

All-in-one platforms like Optimizely or VWO have plans to meet the needs of small or enterprise-level stores. These paid testing platforms make it easy to manage multiple experiments at once. And most feature a page builder or visual editor for customizing your page variations with your own branding. So, building out your variations won’t require developer resources.

Testing platforms track your incoming A/B testing data, reporting on results and automatically testing for statistical significance. These platforms are easy to use and integrate with your site, but their monthly pricing plans may cost more than you’re willing to spend.

A/B Testing Plugins

Testing plugins are an affordable alternative to all-in-one testing platforms. If you’re using an ecommerce solution like WooCommerce, you have access to WordPress-supported A/B testing plugins like these:

Content Experiments With Google Analytics

If you’re into using Google Analytics, you can use Google’s Content Experiments to track changes to features of your website. The process is a little complicated and requires some experience with Google Analytics. But Content Experiments is free to use, and you can do things like this:

  • Compare how web page variations or app screens perform using a random sample of your users
  • Define what percentage of your users are included in the experiment
  • Choose which objective you’d like to test
  • Get updates by email about how your experiment is doing

To use Content Experiments, you’ll need to get a Google account, create a Google Analytics account, and add tracking codes to your web pages. After you set up Content Experiments, you can manage your experiments more easily by setting up A/B testing with WordPress page builders like Beaver Builder.

A/B Testing Keeps It Real

Testing removes the subjective component from your decision-making. It eliminates common contaminants within the decision process—like relying on the highest-paid person’s opinion. Ideas can actually move from whiteboard sessions to real-world implementation. In short, A/B testing keeps your marketing real.

Store owners who attribute their success to making “gut decisions” won’t be in business long. It’s a romantic notion that is quickly being replaced by experimentation and testing, which brings us back to this question:

Which version of the sentence did you pick? A or B?

(A): “Online store owners must conduct tests of their marketing elements with scientific precision.”

(B): “Online store owners are really just mad alchemists —standing in the lab of their websites, trying to find the magical elixir for converting website visitors into customers.”

On a gut level, version B is the more engaging, less bland introductory sentence, but it may be the worst at converting. Or, it may make no difference at all. The point is, you can’t know unless you test it.

Need an Ecommerce Hosting Solution?

Our Managed WooCommerce Hosting platform reduces query loads by 95% while automatically handling image compression and backups, giving you more time to focus on testing and optimizing your store.

Check out our plans to get started today.


Nexcess, the premium hosting provider for WordPress, WooCommerce, and Magento, is optimized for your hosting needs. Nexcess provides a managed hosting infrastructure, curated tools, and a team of experts that make it easy to build, manage, and grow your business online. Serving SMBs and the designers, developers, and agencies who create for them, Nexcess has provided fully managed, high-performance cloud solutions for more than 22 years.

We use cookies to understand how you interact with our site, to personalize and streamline your experience, and to tailor advertising. By continuing to use our site, you accept our use of cookies and accept our Privacy Policy.