Have you ever scrolled through a famous digital platform such as Google, Youtube or Instagram and wondered how these companies optimize the design of their product experience for the user? How do they make these websites instantly catch the user’s attention, or make sure that the browsing experience is as smooth as possible and the information just where you want it to be? When visiting these websites, one often browses through them without noticing the details of their design, as if browsing them is as natural and unconscious as the feeling of breathing. What is the secret optimization formula behind the design processes of these websites? Indeed there is a secret, just that the secret is already commonly known and applied by the world’s largest and most successful tech businesses such as Google, IBM, Microsoft and Spotify. In this article, we will take a glimpse into A/B testing, the secret formula, and how this framework for digital experimentation can help you to radically improve your online business.
Defining A/B Testing
Now that we have caught your attention, we can slowly introduce you to what A/B testing is. A/B testing is the creation of variations of a website, which get tested in a testing environment with the user. The “A” in A/B testing represents the original version, the control group, while the “B”, represents the treatment group, the variations that are put against the original’s performance. The treatment group “B” can be further expanded to “C, D, E, F… n”, and so on, further adding to the number of variations that are put against the control group and each other. The experiment can be carried out in a closed environment with a selected group of users or in the public domain of a live website. In parallel to the execution of the experiment, performance indicators, such as conversion rates, interaction rates, numbers of visitors, etc. are being recorded. These indicators help to select the variation that performs the best against the control. At the end of the A/B test, the best performing variation is then made the default and the testing cycle starts over again. This is how A/B testing works on a basic level.
A Brief History of A/B Testing
But before we jump straight into the details of A/B testing, let’s quickly go through the brief history of A/B testing and how it evolved into the current method that is widely used in tech today. Going back in time the story begins in the year 2000, with Google engineers experimenting with variations in the design of Google’s search result page. The goal was to find out the optimal amount of search results displayed on a single page. After the experimentation, they found out that displaying 10 search results led to the most amount of desired user interaction. Ever since then 10 results per page remained as the answer to their design question. In the following years, other tech companies such as Amazon, MSN and Bing followed by experimenting where and how to display advertisements throughout their platforms. In Bing’s case, the selection of the winning variation in the A/B test increased the revenue drastically by 12%. Today, each of these companies run tens of thousands of A/B tests annually, making it the strongest tool for increasing a website’s performance.
Three Best Practices in A/B Testing
So you might wonder how this can be applied to your business’ website? This is why in the following section we will present you three essential best practices in A/B testing that are easily applicable and will give you real-world cases and benefits of the A/B testing method.
1. Select what you want to test, before you test.
Before you start creating random variations of your website, you should consider what you want to test in the first place. What is the purpose of your A/B test? Is it to test an ad? The size of an ad on your website? The content of the ad or its positioning on the website? Those are the questions that have to be answered and kept in mind when running an A/B test. For what was not measured during the test, can’t be proven after the test. To give you an idea of how you can apply that, we will look into an example where we take the positioning of the ad as the element we want to look at in our A/B test. In this case, we will choose the number of users clicking on the ad’s link as the unit of measurement of desired performance. Now we create a variation, where the ad is displayed inside the content section of the website, while we keep the original, where the ad is at the beginning for controlling purposes. After assigning the testing variations to two user groups, we can see how the different positioning of the Ad affects the outcome in performance. if there is a significant difference in performance, we apply the superior variant for the ad positioning. Now we have improved upon the selected aspect of the website via A/B testing by conducting a specific test.
2. Running many tests over a long period of time.
The easiest way to increase the potential of A/B testing is to increase the numbers of tests that run simultaneously. This allows you to have higher odds of getting a successful test outcome as you diversify your bets. For example, when comparing an A/B test consisting of only one variation from the original to an A/B test that has a hundred variations, it becomes very clear which of these two will lead to the closest point to the maximum performing outcome. In another perspective, the chances of getting even a worse performing result in the test consisting of only one variation are also significantly higher, as all the bets are being put on one trial. Furthermore, the tests can be distributed and conducted throughout a longer period of time, of let’s say a month instead of just a single day, to see how different variations in the A/B test perform over a time period that involves many user sessions. Applying this method will help you to mitigate the effects of “novelty”, in which the involved users will often show an initial heightened interest and level of interaction with a new tested feature or element that then decays over time. Thus, running the test over a long period of time, while also running many of them allows you to increase the performance of your A/B tests while also gaining more truthful data from it.
3. Building an experimentation culture
The last vital decision and endeavour to make when implementing A/B testing as part of any business, is to take a closer look at the current culture. Do employees shy away from risks and uncertainty? Are there any signs of blame games going on whenever a project fails ?
These are just two of the common signs that your organization may not be able to fully embrace a culture where testing and experimentation is the norm. This kind of situation is normal in larger companies, since usually a few cultures end up being predominant over the entire company. For example, a company focused on rather simple and repetitive tasks as its core business, such as a sales-driven organisation, may apply the tolerance of failure established in the core business to other functions of the business, such as R&D. Therefore, building and maintaining a suited company culture – especially in the part of the business which will be mainly tasked with the testing – is key. An environment which facilitates testing and data-driven decisions are the basis for any course of action needs to be established. This can be done by normalising frequent tests and making sure that leaders are role models for this kind of culture. Furthermore, the required infrastructure and decision power need to be put in place. This enables teams to perform and really showcase some of the key strengths of A/B testing. Often this means flying under the radar for some time, making it easier for a certain culture of experimentation to coexist in a company with otherwise completely different norms. In order to truly capitalize on the benefits of A/B testing and experimentation, in the long term it is often easier to embed it at the core of the organization.
Making the First Step
Having said that all, how can you capitalize on this knowledge? It all seems to be a lot and you probably ask yourself: where do I start with all of this? A/B testing is a methodology that requires technical infrastructure that allows you to automatically redirect incoming users towards the desired test variations. But luckily there are many easy-to-implement solutions already existing out there, such as the tools Google Analytics and Google Optimizer. These tools allow you to implement the technical basics for your A/B testing endeavours. The use of Google Analytics allows you to collect quantitative customer data, such as the number of visitors, conversion rate, interaction and use of tools, etc., while Google Optimizer allows you to display A/B test variations on your live website. Thus the combination of the two tools allows you to run A/B tests on your website without the need to develop and implement every technical detail on your own. This saves not only precious time, but also money that would have been spent on additional software developers. But of course, if your business can afford the full financial investment of an A/B testing infrastructure for your website, it would make more sense to choose a custom solution that fits your business’ needs.
Three Pitfalls in A/B Testing
Once you have taken the first steps and established the infrastructure, culture and the expectations for your testing endeavours, there are a few major pitfalls that might prevent you from getting more out of it. In the following part, we have highlighted some of the most common mistakes which you should avoid, as well as how you can deal with them.
1. Averaging out valuable data
While it may be tempting to come up with decisions when looking at the outcome of a test across all user segments, this overview often simplifies the data. Amidst these simplified dashboards which aim to give you a brief overview, it’s easy to forget that behind this consolidated data, there is a diverse set of users making up a population of data points, each acting in their own unique and different way.
Thus, it is useful to predefine certain user segments when running a large scale test. This empowers you to look beyond the average of the data set and enables you to segment users in regards to their geographical location, behaviour, past engagement, or even their previous decisions in other A/B tests. In a second step, targeting a predefined segment of users enables you to test innovations which aim at encouraging a defined set of behaviours for a user segment.
An example for this was when employees at Amazon recognized that a certain type of users would often purchase products at unusual hours during the day. By narrowing down testing to target only those users, they were able to find that those users would respond to an Ad placement on an external website only if it corresponded roughly with their usual purchase time. This not only made timed ads extremely effective for these users, but also meant that including them in testing environments with average users would have likely presented them as outliers, and the data is ignored.
Therefore, segmentation creates a positive feedback loop in which by testing more, you will be able to segment the user base more, which enables you to run highly segmented experiments which then can help you to narrow down the user base even further.
With that in mind, you should still not try to get more out of the data than there really is. Statistical outliers are common among all kinds of tests, and there is no exception when it comes to A/B testing. Don’t get caught up trying to follow individual users who represent extreme outliers, instead find a good balance.
2. Not considering users as part of a community
A second common pitfall of A/B testing is to consider each user as a unique entity and to forget that some actions may have a network effect and influence the behaviour of many other users. Changes that include users being encouraged to interact with other users in different ways, often also affect the entire system in which these users operate. An experiment which encourages certain behaviour such as a user sending a message to another user on their birthday might also influence the users not directly affected by the experiment, in this case, the user receiving congratulations for their birthday. One common solution to this issue is to use A/B network testing. In order to get the full picture of the implications of a change, in an A/B network, isolated user groups represent control and test cases. This helps you to understand the systemic impact of any changes, and see the downstream effects of certain changes. Simply put, it helps you see the butterfly effect following a butterfly’s wing flaps.
3. Lack of patience and strategic consideration
Just as it may take many interactions to create a user behaviour pattern, it may also take a while to change it. That’s why short term testing can seldom reveal the real effects the change may have on the user in the long run. Furthermore, the effects may differ substantially depending on the exposure a user has on a certain treatment. It is therefore advised to keep tests running until user behaviour has stabilised before drawing any conclusions. Another approach to show the long term systemic effects of changes, is to run time series experiments. This means exposing a large segment of the market to a certain change for a longer timeframe, aimed at showing the long term effects of certain treatments against the control.
It is important to note that many experimentation projects are not fully able to take off because too much is expected too soon. While simple tests can be established with relative ease, it may take many iterations and an extended duration to establish a solid testing routine and the necessary infrastructure to support it.
Key Takeaways
Once you have the necessary technical infrastructure, it is an easy step to make towards running the A/B test. Just keep in mind the three best practices and three pitfalls that will guide you towards your end goal of increasing your website’s performance, whether it is about improving the user experience, or gaining more clicks on advertisements. Always be selective of what you want to test, and keep tests running as frequently as possible, and for extended durations. Build a culture that allows for frequent tests, and which embraces experimentation as a key tool for optimization. And while you are building on these best practises, always keep in mind that averaging out data means missing out on valuable insights. Segment your user base, and always keep in mind that there may be an entire network of users affected by just one change. Lastly, always keep in mind that testing is a long term effort and that you should always set the expectations about this straight. Testing is about adopting a mindset of continuous improvement, and doing so will greatly empower you to make the right decisions.
Sources
Stefan Thomke. (2020). Building a Culture of Experimentation
https://hbr.org/2020/03/productive-innovation#building-a-culture-of-experimentation
Iavor Bojinov, Guillaume Saint-Jacques and Martin Tingley. (2020). Avoid the Pitfalls of A/B Testing
https://hbr.org/2020/03/productive-innovation#avoid-the-pitfalls-of-a-b-testing
Daniel McGinn. (2020). The Power of These Techniques Is Only Getting Stronger
https://hbr.org/2020/03/productive-innovation#the-power-of-these-techniques-is-only-getting-stronger
Jessie Chen. (2016). How Netflix does A/B Testing Jessie Chen
https://uxdesign.cc/how-netflix-does-a-b-testing-87df9f9bf57c
Amy Gallo. (2017). A Refresher on A/B Testing
https://hbr.org/2017/06/a-refresher-on-ab-testing
Stephen Courtney. (2019). What is AB Testing?
https://www.convertize.com/what-is-ab-testing/
Wikipedia Article. A/B testing
https://en.wikipedia.org/wiki/A/B_testing
Thank You for sharing.
Thank You for sharing.
Thank You for sharing.
Good afternoon Mr. Gugler and Mr. Lin
Your blog post on A/B testing has been highly enjoyable to read. In fact, your post has illustrated many aspects of the digital world that were unknown to me. As a fellow blogger this has prompted me to write a response to your post because I like to engage in contemporary and relevant topics as you can see on my latest blog post “Essential Marketing Knowledge That 99% of Artists Lack”.
Your introduction is very clear and captivates the reader’s attention instantly. You then follow up with a definition of A/B testing which was very easy to read but at the same time it didn’t feel like any information was lost. This is very pleasant for the reader as oftentimes blogs are full of long introduction stories and other clutter.
The part called “Three Best Practices in A/B Testing” was yet another interesting read however, there I saw a very important point to consider. How would this work for a small business or start-up? Because sure, it will work with hundreds of users, but beginners, such as people reading an introductory blog post on A/B testing, rarely have many users on their websites. And statistics gain merit only through the quantity of tests they execute.
With that being said I do want to say that the three practices are highly compelling. Especially with the easy to use tools that you recommend in the following paragraph.
The next section “Three Pitfalls in A/B Testing” is exactly the in depth and practical information blog readers seek! In my opinion, you could even write a whole blog post on the second point of this chapter because as most paragraphs it is just a slice of a very vast topic.
The key takeaways sum your post up nicely. One thing you could add there is an outlook to your future blog posts and what they will discuss.
Reading your blog post was an overall nice experience. As mentioned, at times it was difficult to imagine how your advice would translate to a small start-up. But I think that the beauty of blogs is that they allow the reader to immerse him or herself into a world that is not yet his or hers. I am looking forward to reading your future posts.
Kind regards
Caspar Danuser