A/B-Test: Tipps für Online-Händler + Beispiel
Friday, 13 October 2023
Latori GmbH

A/B testing: tips for online retailers + example

You are an online retailer and 100% convinced of your brand and your products, but the sales figures are not as expected? Maybe you are using the wrong marketing strategy or customers are leaving your site after only a few clicks. There can be many different reasons for this. A/B testing (or AB testing) can be used to find out the real cause and then eliminate it.

In this article we will show you what these A/B tests are all about, how you can carry them out and what advantages this can have for your company.

Are you using Shopify or Shopify Plus and could use professional support for A/B testing or in other areas in your e-commerce business? Then feel free to contact us and we will be happy to help you with advice and support.

A/B-Testing Definition

An A/B test is a test method for evaluating two variants of a system by testing the original version against a slightly modified version. This method is mostly used in software and web design with the aim of increasing certain user actions or reactions. Over the years, it has become one of the most important testing methods in online marketing. However, A/B testing is also used in other areas, such as comparing prices, designs or elements in online shops.

Reading tip: You can find tips on conversion rate optimisation here.

A/B-Testings: Advantages and disadvantages

With A/B testing, you can test every change to your marketing activity, website, etc. and decide on the more successful version based on solid data. This brings the following advantages for you:

Advantages of A/B testing

  • Higher conversion rates

  • More satisfied customers/website visitors

  • Optimised time and budget management

  • Better insight into the needs of the target group

  • Possibility to implement qualified results immediately

However, besides its opportunities, A/B testing also brings a few challenges:

Disadvantages of A/B testing

  • Several tools necessary

  • Only one hypothesis possible per test

  • Confusion among clients

  • For small sites: time-consuming and statistical significance difficult to achieve

Reading tip: Find out what Customer Lifetime Value means and how to calculate it here.

Procedure for A/B testing

In A/B testing, target groups (such as website visitors or newsletter recipients) are divided into two subgroups: Group A and Group B. This division must be random. Depending on the target group, test objects such as landing pages or advertisements are also divided into two parts: the original variant and the modified variant. The two variants should only differ in one component, because this is the only way to clearly attribute response differences to the changes. Then use the original version for group A and the modified version for group B and compare the reactions. Reactions here mean the desired effect, such as subscribing to a newsletter or ordering a product.

In addition to improving the user experience, A/B testing is also a means of increasing conversion rates. Statistical testing techniques used for A/B testing depend on the characteristics of the data used.

Reading tip: We explain how you can test your page speed with Shopify here.

A/B testing: You can use these statistical approaches

There are two statistical methods used in A/B testing around the world: The Frequentist Approach and the Bayesian Approach. Both methods have their advantages and disadvantages. The following comparison between the two methods will help you understand the differences.

Frequentistischer Ansatz Bayes’scher Ansatz
Die Frequentistische Methode (auch Chi-Quadrat-Methode genannt) ist objektiv. Die Bayes’sche Methode ist deduktiv.
Ergebnisse können nur am Ende des Tests analysiert werden. Der Test muss außerdem eine bestimmte Zeit lang laufen, bis korrekte Daten generiert werden können. Ergebnisse können noch vor dem Ende des Tests analysiert werden, da sich diese Methode auf Wahrscheinlichkeiten bezieht.
Für die Analyse werden Tests durchgeführt und nur aus den Daten des aktuellen Experiments Schlussfolgerungen gezogen. Bei diesem Ansatz wird auch das Wissen aus vorherigen Experimenten mit in den aktuellen Datensatz einbezogen. Hier dienen also auch vorhandene Daten dazu, Schlussfolgerungen zu ziehen.
Dieser Ansatz gibt einen geschätzten Mittelwert der Stichproben an, bei denen die Originalversion (A) die modifizierte Version (B) schlägt. Er gibt jedoch keine Auskunft über die Fälle, in denen sich die modifizierte Version als besser herausstellt. Außerdem kann nicht festgestellt werden, wie weit A und B voneinander entfernt liegen oder wie hoch die Wahrscheinlichkeit ist, dass A B schlägt. Der Bayes’sche Ansatz berücksichtigt ebenfalls die Option, dass A B schlägt. Er gibt darüber hinaus auch eine errechnete Spanne der zu erwartenden Verbesserung an und zeigt exakt, wie weit A und B voneinander entfernt sind.

Reading tip: Learn the Dos & Don'ts for Shopify merchants when it comes to customer recovery.

A/B Testing with Shopify: How to run A/B tests step by step

All websites can benefit from A/B testing, as each has at least one measurable goal. Whether you have an online shop, a news site or a lead generation website, your goal is usually always to increase the conversion rate.

So A/B testing can be done on any page, but also for all elements of the website: From push messages to design, to buttons like "search" or filters, and many more. Here are 10 examples that might inspire you:

  • Titles and headlines: For example, start by changing headings or article content to find out what appeals to users. Changing color or font can also make a difference.

  • Call-to-action: the CTA button is critical. Color, font, placement, and words used ("buy," "add to cart," "order," etc.) can significantly impact conversion rates.

  • Buttons: Other buttons can also be important. They can change size, shape and color to attract more visitors.

  • Images: Images are just as important as text. Try different images and vary the size, aesthetics and placement.

  • Page structure: the structure of your pages, be it the home page or categories, should be well thought out. You can change elements like carousels, banners or popular products.

  • Algorithms: Use different algorithms to turn visitors into buyers or increase shopping cart value.

  • Business model: rethink your business model to generate higher profits, e.g. by adding suitable products or services.

  • Forms: Forms need to be clear and concise. Test different wording, optional fields and placements.

  • Pricing: A/B testing with prices is challenging. Use your imagination to test the impact on conversion rate, e.g. by offering savings or variants.

Reading tip: In this article, you will find out which options are open to you in mobile shopping for your own shop and what you should pay attention to.

What types of A/B tests are there?

There are different types of A/B tests that should be carried out depending on the page or test objective:

  • Classic A/B testing: In classic A/B testing, your visitors see two or more variations of a page at the same URL. This allows you to measure the success of different variations of a particular element.

  • Split test or redirect test: Split tests redirect your traffic to another URL or several different URLs. This can be particularly interesting if you are hosting a new site on a server.

  • Multivariate tests (MVT): Multivariate tests can measure the impact of multiple changing elements on the same page. For example, you can change banners, text colours and even your design. This allows you to determine which variation works best for you.

  • A/A tests: With an A/A test, you can test two identical versions of one or more elements. The hits on your website are divided into two groups, each of which sees the same changes. This way you can see if the conversion rate is similar for each group and confirm that your solution is working properly.

Reading tip: In this post, we'll look at increasing average order value, or AOV and give you 12 tips on how to increase this metric on Shopify Plus.

How to carry out an A/B test step by step

Below we go through the individual steps to help you achieve successful results in your A/B test.

1. Identification of problems and potentials

At the beginning, it is important to determine what is to be determined in the first place. In this step, make sure to only note things that are really verifiable and where there is potential for optimisation. Based on this, you can collect your user data with the help of interviews or heat maps, for example.

2. Selection of the test group

For certain test objectives, it can be worthwhile to include a test group. An example of this is the optimisation of your newsletter. Including existing subscribers there would not be very effective.

3. Formulate the hypothesis

Based on the problems identified in step 1, you now formulate a hypothesis for each. The individual variables must always be verifiable and measurable and the hypotheses should not contradict each other.

4. Selection of the tools

Step 4 is now about choosing the right tools. In the next chapter we present a selection of well-known tools for conducting A/B tests. It is important that you take the time to choose the right tools for your skills, test types and requirements. Shopify also offers built-in A/B testing tools that allow you to create variants and manage the test.

5. Performance of the test

Once you have decided on one or more tools, it is time to conduct the A/B test. Make sure you have a sufficiently large test group and a meaningful runtime. Particularly in e-commerce, certain phases such as Christmas could strongly influence visitor numbers. Here, too, tools help to determine the so-called reliability rate. To exclude coincidences as much as possible, this rate should be at least 95%.

6. Evaluation of the results

After the implementation, the results have to be considered with regard to the hypothesis. Depending on which tool you have chosen, this may already include an evaluation and archiving function. If the results turn out to be very different from your previous assumptions, you should review the initial data and your assumptions again and, if necessary, adjust the test criteria accordingly.

7. Implementation of the results

If your test result meets your expectations, you can implement all optimisations. After that, it is important to keep an eye on the website and all changes.

8. After the test is before the test

Once you have completed a test, you can start a new test with another hypothesis. For example, you can gradually test and optimise different elements of your onlineshop.

Reading tip: How to create legally compliant Shopify Cookie Banners.

Examples of well-known A/B Testing Tools

There are now many different A/B testing tools for websites and online stores. Well-known examples include:

While tools such as Visual Website Optimizer and SiteSpect tend to be more cost-intensive due to their additional functions, there are also programmes such as Kameleoon where a free freemium account is possible for up to 2,500 visitors per month. In addition to the price, user-friendliness or usability is of course also decisive for many. Optimizely, for example, is considered a beginner-friendly option.

However, if you run a Shopify or Shopify Plus shop, there are also a whole range of great plugins available in the Shopify App Store. There is something for you there for every type of A/B test.

Reading tip: How Shopify's Shop Pay payment method and checkout solution works.

A/B Testing Example: Webshop

Let's say you run an online shop where you sell high-quality skin care products. You notice that you have a lot of visitors on your site, but hardly any of them make a purchase. This observation is the first step in the right direction, because with existing data you have at least already been able to identify this problem.

So the next step is to find out what you could A/B test to eliminate the problem. Surveys can be very useful for this, as they also save you from creating new variants that end up completely going nowhere.

Once you have completed your survey, you can build two different variants based on it. Let's assume that the product description has revealed a potential for improvement. In this case, you could create two different variants of this description, such as "Pore refining face cream with vanilla scent" and "Vanilla Whipped Cream" and test which one is better received, i.e. generates more sales.

Reading tip: Here's what you need to keep in mind when redesigning your online shop.


Even small changes can make a big difference. With the help of A/B testing, you can eliminate all weak points in your website, your online shop, your app or even in marketing measures, provide the best optimised version and skyrocket your conversion rate. So take advantage of this opportunity, conduct A/B tests. Simply follow our step-by-step instructions in the article and make sure that you determine the data precisely. We hope this article was helpful for you and wish you good luck with A/B testing.

If you need professional support from Shopify and Shopify Plus experts, we would be happy to help you. Please contact us.

Frequently asked questions about AB testing

What is A/B testing?

A/B testing involves comparing two versions of a website, app, etc. with each other. The goal is to determine which of the two versions is better received by the (potential) clientele in order to ultimately increase the conversion rate. The A stands for the original version and the B for a slightly modified version. The variants are shown to the users randomly, so that one part of the users gets to version A and the other part to version B. The version A is the original version and the version B is the modified version. Depending on how the user behavior turns out, it can be deduced which version is better liked and should therefore be retained.

Why use A/B testing?

A/B testing brings a number of benefits. Mainly, it is about increasing the conversion rate. In addition, testing contributes to the optimization of the store. We reveal all other opportunities and risks of this testing method in the article.

What content should you test via A/B testing?

You can perform A/B tests on every website, in every online store and also in every app. Basically, just about all elements can be tested here as well. Examples include titles, CTAs, buttons, images, forms, and prices.

What are the best A/B testing tools?

There are numerous tools for A/B tests on the market, which is why you should consider in advance what suits your skills and requirements. Well-known web analytics tools include Google Analytics and Adobe Analytics. If you want to work with heatmaps, the programs crazyegg and ClickTale might be interesting for you. For session recording, we recommend mouseflow, among others. We have listed further tools for you in the article.

How does A/B testing work?

How does A/B testing actually work? AB testing compares the current version of a website with a changed version, or several. You can change the entire website here or just individual elements such as call-to-action buttons.

What factors should be optimized before an AB test?

Before conducting an A/B test, certain factors should be optimized to ensure that your tests provide meaningful and reliable results. Some important factors that should be optimized before an A/B test are clear objectives and hypotheses, clear target metrics, sufficient test duration or test and control groups.

Shop Usability AwardShop Usability Award
Wir schätzen alle unsere Kunden, Nutzer und Leser, egal ob weiblich, männlich, divers oder nicht-binär. Der Lesbarkeit halber verzichten wir auf Gendersternchen und nutzen weiterhin das generische Maskulinum. Wir sprechen damit ausdrücklich alle an. Bitte beachten Sie außerdem, dass wir Zitate zum besseren, sprachlichen Verständnis leicht angepasst haben.
Shop Usability Award Winner 2023