The what and when of A/B testing
“Experimentation is the least arrogant method of gaining knowledge.”
— Isaac Asimov
Let’s lay the foundations of A/B testing with its definition and brief history.
A/B testing, at its simplest, is a way to compare two versions of something, in our digital product case it is a web page or an app, to figure out which performs better.
In A/B tests, users are randomly split between the two groups where one group sees the original version (Control) and the other sees the new version (Variation) as shown in the figure below:
During the tests, we collect the necessary metrics and in the end we prove or disprove our hypothesis by analyzing the data we collected.
How it all started
Scurvy was the leading cause of death among seamen in the 18th century. James Lind, a physician in the British Royal Navy, ran a test to prove his theory that citrus fruits could cure scurvy. For this purpose, he divided the 12 sick sailors into six pairs, and provided each of them with a different supplement in their diet: cider, vitriolic elixir, vinegar, sea water, two oranges and a lemon, or a purgative mixture.
As a result of what some have considered the first clinical trial in history, only the two sailors who took the citrus fruits improved. Of course it was a bit far away from today’s A/B testing standards and statistics since it was based on a small number of observations, but in the end the result was convincing and decisive enough since the impact was so dramatic.
Ronald Fisher designed experiments to determine what effect various external factors had on the growth of plants and crops. He was the first to figure out the basic principles, statistics and mathematics behind A/B testing and make them a science. His work is considered to be the starting point of modern hypothesis testing.
Google engineers ran their first A/B test to determine the optimum number of results to display on its search results page. The first test was unsuccessful due to glitches that were caused by long loading times, but that in itself yielded a critical insight — tenths of a second could make or break user satisfaction in a precisely quantifiable way.
A/B testing is like a standard in today’s world since it is especially useful in combination with agile software development. Microsoft and several other leading companies including Amazon, Booking.com, Facebook, and Google each conduct more than 10.000 A/B tests annually, with many tests engaging millions of users. Today, A/B testing is utilized by businesses of all sizes across various industries.
While the things we test change and evolve everyday, it is quite stunning to see that A/B testing has been helping us for ages to establish causalities and understand more about the world whether online or offline.
We’ll be discussing why we are doing A/B tests next time. Stay tuned!
References:
https://experimentguide.com/history/
