How much change can I test with A/B testing and when is it too much?
Put light against light — you have nothing. Put dark against dark — you have nothing. It’s the contrast of light and dark that each give the other one meaning. — Bob Ross
Sky is the limit! We can either test just a single element or the full-page redesign or everything in between depending on our hypothesis and goal!
For example, if we have a broad hypothesis like a page is confusing, hard to navigate and looks outdated from users’ point of view, and we want to address this by helping our users to better understand the main interaction points and create a higher perception of the product, then it is not helpful just to switch the position of the headline with an image. What we really need to do in this case is to change the structure of the page by changing multiple elements in combination to create the desired effect.
Example of a test with a broader hypothesis where a redesign of Airbnb home page was tested:
However, if we have a more narrowed down hypothesis focused on a single element, then it makes sense to just make changes and test different versions of this particular element.
Example of a test with a more focused hypothesis where only the wording for the button label was tested by Airbnb:
In general, as a rule of thumb, we need to focus on creating enough contrast between the variations in line with our hypothesis while spending the least effort. For creating the desired behavioral change of our users, the changes we make need to be clear and distinguishable.
Here you can read the previous article in this A/B testing essentials mini series which is about when is A/B testing helpful and when it is not: https://aybalacoskun.medium.com/when-is-a-b-testing-helpful-and-when-not-70243898c0bf
We’ll be discussing A/B/n and Multivariate testing next time. So, stay tuned!
References:
https://cxl.com/blog/one-change-per-test/
