Selecting Between A/B and Multivariate Tests

Whether an A/B or multivariate test is used, product managers gain insight from experiments as to whether their proposed changes will have the hypothesized effect. This makes experiments an important tool in the product manager toolkit.

A/B test

An A/B test is an experiment to seeing if a single change has an effect compared to what is currently in place. An A/B test works in four steps:

1. Decide what change you want to test.

2. Decide which group of users will see the control and which group of users will see the test.

3. Run the experiment. You want the experiment to run long enough that enough people can see it.

4. Measure the results and see if the experiment results in KPI improvement.

Multivariate tests

Multivariate tests follow the same steps of an A/B test in deciding what to test, making user groups, running an experiment, and measuring results.

However, a multivariate test can test more than one change at once. The multiple changes a product manager can test takes three main forms:

• Test a range: any variable that goes from a low number to a higher one.

• Test many, totally different versions of a feature or UI.

• Test variations of a design by only changing one element or a few elements at a time.

To calculate how many tests one has, you need to multiply the number of options in each testing element. In the video, there are two testing elements and 3 options for each element (including the original version). So 3 options for the first testing element times 3 options for the second testing element (3*3), which gives 9 tests including the control (the original design).

Pros and Cons of A/B tests

Benefits	Drawbacks
​Need fewer users to get statistically significant results since there aren't many test groups.	Can only test one change at a time Don’t have the ability to see how different changes interact together

Pros and Cons of multivariate tests

Benefits	Drawbacks
Allow testing many different variations at once Show how different changes work together and interact	Need enough users in each group to show statistical significance Running a large number of tests, including sorting users randomly into the many variations, can be difficult

That is it for this article. I hope you found this article useful, if you need any help or got any feedback for me please email me at info@nabeelansar.com

Let's connect on Twitter and LinkedIn.

👋 Thanks for reading, See you next time