Rename and duplicate

Duplicating is useful when you want to copy an experience as the basis for a new experiment or when you want to publish as a live experiment but keep a copy as a draft that you can continue working on.

Duplicating an experience variation is a good option where you wish to test the effectiveness of a small change to an experience, for example, a different image or slightly different wording in a banner.

You can also duplicate a variation more than once. This allows clients looking to undertake A/B/n testing the opportunity to compare multiple versions of a web page against each other. In this type of test, traffic is split randomly and evenly distributed between the different variations to determine which variation performs the best.

A/B tests (respectively A/B/n tests) compare one variation (respectively many variations) against a single control. Whether you want to run one A/B/n test or many A/B tests is a matter of preference.

It may seem faster to run an A/B test than an A/B/n test since your traffic will be less diluted amongst the variations. If you run a sequence of A/B tests, each A/B test in the sequence will complete faster than the A/B/n test that runs all the variations you want to test in parallel.

However, the A/B/n test will complete faster than the entire sequence of A/B tests. Indeed, it’s more efficient to run an A/B/n test, since the control group is shared by the variations. A/B/n testing also has the benefit that data in all the variations is collected during the same duration, and so the variations in the A/B/n test are all subject to the same seasonal effects.

That’s not to say you should always run A/B/n tests with as many variations as possible: diluting traffic into variations that are for all practical purposes, conducting A/A tests, is not an efficient use of traffic.

An A/A test is a test where we run two variants with no treatment whatsoever, that is, we run a test with two controls. We do not expect to observe any difference between the two variants (though this can happen by chance). It’s generally done as a sanity check for new clients or a new test, to iron out data issues.

A/B testing is a randomized experiment with two variants A and B, one of which is a control, and one of which is a treatment variant. The goal is to identify changes to web pages that increase an outcome of interest, for example conversion rate, or clickthrough rate on a banner.

This term is sometimes used in the industry to describe a type of multi-variate test. Multi-variant testing is hypothesis testing with more than two variants (that is, more than one treatment variant and a control). It’s popular in web-analytics because it’s perceived that it can be used to test lots of hypotheses quickly (though we believe it’s a bit more complicated than this).