Traffic allocation modes

This is for:

Developer

In this article we will look at the standard traffic allocation modes you can select when configuring your experiences and answer some FAQs.

Of course, you may choose to select a custom split for your experience. There’s more about this in Traffic allocation.

Note

It’s important to note that once the experience is published, you can’t switch from a custom split to a standard mode traffic allocation, even if you pause it later.
Tip

During the lifetime of an experiment, especially if an experiment shows a positive impact against the defined goals, there will be a temptation to change the traffic allocation. This will result in flipping. Increasing the traffic allocation–flipping users from the control to the variation, for example increasing from Pilot 20/80 to Split 50/50–is a recommended practice as it increases the exposure of winning variants to more users.

Pilot 20/80

Pilot mode is best used for testing high risk experiences and is often used to detect issues or bugs. The variation is only shown to 20% of users.

Split 50/50

Split mode, which evenly distributes users between the variation and control over time, is the default traffic allocation setting. It’s often used because it’s the most effective mode for A/B testing. The variation and control will receive an equal amount of users over time. Split mode reduces the amount of time between launching an experience and getting conclusive test results.

Supervised 5/95

Supervised mode is typically used after Split mode, when your test is successful, and you wish to roll it out to a larger percentage of users.

Keeping a 5% control will ensure any long-term behavioral changes remain visible.

All traffic 100

All traffic mode is designed for delivering content that has to be shown to all users, for example legal messages. This mode doesn’t support performance tracking.

FAQs

Why do pilot tests complete so much quicker than tests using other traffic allocation modes?

Pilot tests run at much lower power (they don’t have much chance of detecting uplift), so they complete much faster. Generally, they should be used to ensure that a change doesn’t result in a significant negative impact.

Why do split 50/50 tests run faster than supervised 95/5 tests?

A normal test requires 6,000 converters in each variant. If a customer has only a 5% chance of being put in the control, it will take a much longer time to get to 6,000 for that variant. 50/50 is the fastest possible A/B test.

When changing an experience from 33/33/33 to 50/50 by removing a variant what happens?

Users will be reallocated according to the new split. It should work the same way as a regular change in traffic allocation.

The way it works is that every user is assigned a random number between 0 and 1, if the split is say 33, 33, 33, then visitors with numbers between 0 and 0.33 are in the control, between 0.33 and 0.67 in variant 1, and visitors over 0.67 in the second variant. If you change to 50/50 then visitors below 0.5 are in the control and visitors over 0.5 are in the variant.

So in this case, every one in the control at the start remains in the control, half the visitors in old variant 1 are placed into the new control, the other half in new variant 1, and all visitors in the old variant 2 will be placed in the new variant 1.

In what circumstances will users see both the experience control and variant?

This can certainly happen if you change the traffic allocation for an experience whilst it’s still running. This is referred to as flipping. If an experience starts in Pilot mode then 80% of users will be in the control. If you now switch to the 50/50 Split mode, 30% of the users will move from control to variant. If you later switch to the All traffic mode, then all the previous control visitors will naturally move to the variant.

The same may happen when an experience is run in All traffic mode for a period of time before being changed to Split mode (not a recommended change). In this situation, 50% of users will of course move from the variant to the control.

Users that acquire multiple Qubit Ids can also see both the control and variant. This typically happens on gaming sites, where some people use different devices, many browsers, or go incognito.

You can include a code snippet to check for user_id in the experience triggers to minimize the number of users who enter both variations.

Finally, if a user sees an experience variation and later turns off cookies, they will then no longer see the variation.

Warning

We don’t recommend switching an experience from the All traffic mode to a split experience, as it significantly extends the time to completion. We recommend duplicating the experience and restarting it as a new one. That would also remove the issue of visitors flipping.

I’ve paused an experience and I would like to change the traffic allocation to bucket more users into the control. When I try to do this, I can’t change the allocation in the control. Why is that?

As mentioned, for many reasons, but principally in the interests of statistical robustness, we don’t allow changes to splits that will result in users flipping from the variation into the control. Instead, we recommend duplicating the experience and re-running it with the desired split.