“Without data, you’re just another person with an opinion.” — W. Edwards Deming
Creating a hypothesis is an essential part of the A/B testing process since we are creating our variations based on our hypothesis. In the end, what we learn out of the tests is strongly correlated with our ability to come up with the right hypothesis.
A hypothesis is an educated and testable statement that we make prior to running an A/B test that consists of:
A hypothesis helps to answer the question: “What are we hoping to learn from this experiment?”…
“The difference between theory and practice is larger in practice than the difference between theory and practice in theory.” — Jan L.A. van de Snepscheut
We already discussed that A/B testing is a way to compare two versions of something to figure out which performs better. Now, let’s look at A/B/n and Multivariate tests a bit closer:
A/B/n testing is an extension of A/B testing where more than two versions are compared against each other at once. We use it when we have more than one proposed variation to test our hypothesis. A major advantage of this is that, it…
Put light against light — you have nothing. Put dark against dark — you have nothing. It’s the contrast of light and dark that each give the other one meaning. — Bob Ross
Sky is the limit! We can either test just a single element or the full-page redesign or everything in between depending on our hypothesis and goal!
For example, if we have a broad hypothesis like a page is confusing, hard to navigate and looks outdated from users’ point of view, and we want to address this by helping our users to better understand the main interaction points…
There are a variety of things we can test with A/B testing ranging from;
to less user visible backend changes like;
However, A/B testing has also its limitations and there are some cases where it might not be the best or most straightforward tool to use. Here are the two most prominent use cases:
The main reason is that, as data-driven and learning organizations, we want to make data-informed product and design decisions rather than just relying on guesswork and intuition.
We often tend to take our assumptions as facts and believe in our gut feeling and intuition while making decisions. However, we don’t take the consequences of making wrong assumptions into account. Just because of these wrong assumptions a majority of new startups and even some established corporations fail.
So, rather than taking our assumptions as facts, we should create hypotheses based on our assumptions and test these hypotheses to make better product…
I had set out to create the user journey map of a price comparison website where a journey map is defined as the visualization of the process that a person goes through in order to accomplish a goal. It helps us to learn more about our users and how they interact with our product. If we are open to their feedback, at the end it is a great tool for uncovering opportunities for improvement to create a better experience for our users.
While creating the journey map, I struggled a lot to come up with one, since there are lots…
Let’s lay the foundations of A/B testing with its definition and brief history.
A/B testing, at its simplest, is a way to compare two versions of something, in our digital product case it is a web page or an app, to figure out which performs better.
In A/B tests, users are randomly split between the two groups where one group sees the original version (Control) and the other sees the new version (Variation) as shown in the figure below:
During the tests, we collect the necessary metrics and in the end we prove or disprove our hypothesis by analyzing the…
Recently I came across Google’s report about the research they conducted on the buyer decision making process, which is a really interesting topic not only for marketers but also for us product people.
I really enjoyed reading the report and also found it quite insightful. Since I am working at an e-commerce price comparison company, it was even more fascinating to me because our users have lots of different options they can explore even without having to leave the page they are on.
That’s why I decided to write my next piece on this where I summarize the 100-page report…
As product people, we are basically problem solvers. Most of the time the problems we are given by our users or stakeholders are just symptoms, or side effects, rather than the root cause leading to the problem. And sometimes we aren’t given a problem at all. That’s why we always need to start with identifying the right problems which starts with asking the right questions. Because at the end of the day solving a wrong problem might be worse than having no solution at all.
This is where I find design thinking quite valuable within the product management process and…
Product Owner for A/B Testing @idealo