“Without data, you’re just another person with an opinion.” — W. Edwards Deming

Creating a hypothesis is an essential part of the A/B testing process since we are creating our variations based on our hypothesis. In the end, what we learn out of the tests is strongly correlated with our ability to come up with the right hypothesis.

So let’s start with what a hypothesis is:

A hypothesis is an educated and testable statement that we make prior to running an A/B test that consists of:

A hypothesis helps to answer the question: “What are we hoping to learn from this experiment?”


“The difference between theory and practice is larger in practice than the difference between theory and practice in theory.” — Jan L.A. van de Snepscheut

We already discussed that A/B testing is a way to compare two versions of something to figure out which performs better. Now, let’s look at A/B/n and Multivariate tests a bit closer:

A/B/n testing is an extension of A/B testing where more than two versions are compared against each other at once. We use it when we have more than one proposed variation to test our hypothesis. A major advantage of this is that, it…


Put light against light — you have nothing. Put dark against dark — you have nothing. It’s the contrast of light and dark that each give the other one meaning. — Bob Ross

Sky is the limit! We can either test just a single element or the full-page redesign or everything in between depending on our hypothesis and goal!

For example, if we have a broad hypothesis like a page is confusing, hard to navigate and looks outdated from users’ point of view, and we want to address this by helping our users to better understand the main interaction points…


“Almost any question can be answered cheaply, quickly and finally, by a test campaign. And that’s the way to answer them, not by arguments around a table.” — Claude Hopkins

There are a variety of things we can test with A/B testing ranging from;

to less user visible backend changes like;

However, A/B testing has also its limitations and there are some cases where it might not be the best or most straightforward tool to use. Here are the two most prominent use cases:


“The fewer the facts, the stronger the opinion.”
— Arnold H. Glasow

Fact vs opinion
Fact vs opinion

The main reason is that, as data-driven and learning organizations, we want to make data-informed product and design decisions rather than just relying on guesswork and intuition.

We often tend to take our assumptions as facts and believe in our gut feeling and intuition while making decisions. However, we don’t take the consequences of making wrong assumptions into account. Just because of these wrong assumptions a majority of new startups and even some established corporations fail.

So, rather than taking our assumptions as facts, we should create hypotheses based on our assumptions and test these hypotheses to make better product…


The analogy between the user flow of a price comparison website and the double diamond model

Image of the abstraction of the double diamond user flow model
Image of the abstraction of the double diamond user flow model
Image: Abstraction of the double diamond user flow model

I had set out to create the user journey map of a price comparison website where a journey map is defined as the visualization of the process that a person goes through in order to accomplish a goal. It helps us to learn more about our users and how they interact with our product. If we are open to their feedback, at the end it is a great tool for uncovering opportunities for improvement to create a better experience for our users.

While creating the journey map, I struggled a lot to come up with one, since there are lots…


“Experimentation is the least arrogant method of gaining knowledge.”
— Isaac Asimov

Let’s lay the foundations of A/B testing with its definition and brief history.

A/B testing, at its simplest, is a way to compare two versions of something, in our digital product case it is a web page or an app, to figure out which performs better.

In A/B tests, users are randomly split between the two groups where one group sees the original version (Control) and the other sees the new version (Variation) as shown in the figure below:

A simple illustration of an A/B test with two variations where 50% of the users see the A variation and the other half the B variation
A simple illustration of an A/B test with two variations where 50% of the users see the A variation and the other half the B variation
Figure 1: A simple illustration of an A/B test

During the tests, we collect the necessary metrics and in the end we prove or disprove our hypothesis by analyzing the…


Squiggly lines illustrating the messy middle.
Squiggly lines illustrating the messy middle.

Recently I came across Google’s report about the research they conducted on the buyer decision making process, which is a really interesting topic not only for marketers but also for us product people.

I really enjoyed reading the report and also found it quite insightful. Since I am working at an e-commerce price comparison company, it was even more fascinating to me because our users have lots of different options they can explore even without having to leave the page they are on.

That’s why I decided to write my next piece on this where I summarize the 100-page report…


As product people, we are basically problem solvers. Most of the time the problems we are given by our users or stakeholders are just symptoms, or side effects, rather than the root cause leading to the problem. And sometimes we aren’t given a problem at all. That’s why we always need to start with identifying the right problems which starts with asking the right questions. Because at the end of the day solving a wrong problem might be worse than having no solution at all.

This is where I find design thinking quite valuable within the product management process and…

Aybala Coskun

Product Owner for A/B Testing @idealo

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store