Performance of an experience

Your Experience list page is your first port of call and a great place to get a top-level view of how each of your experiences is performing against its goals. Let’s take a look at an example:

By hovering on the uplift, we can see how the experience is performing against each of the defined goals:

Each Qubit experience will have a primary goal and secondary goals. You can add a maximum of five goals for each experience.

Performance against goals is also presented in your Stats card, shown when you open an experience. Let’s look at an example:

In this next example, we see that user has added a custom goal addToBag:clicked:

For each goal, we report the results of experience variations compared to the experience control. Variations are always compared to the experience control, so, for example, A v B, A v C, A v D, etc.

By comparing the variation to the control, you always have a solid basis to determine which variation, if any, is having the biggest impact on each of your goals, whether that be conversions, Revenue Per Visitor, or the firing of specific QP events.

Stated simply, the variation that is most successful at achieving the primary goal is the winner.

Power is the probability of a significant result given random sampling fluctuations around your actual data. In other words, it shows you the importance you can attach to a result. When power is low, there’s still a lot that could change in the experiment and it’s therefore too early to attach any significance to the result.

If we look at the following example, we would conclude that it’s far too early to attach any significance to either uplift or revenue–both could change significantly as more data is gathered:

When you hover over the power meter, we’ll provide a breakdown by goal:

This section provides a couple of case studies illustrating how you can use power forecasts to gauge present and future success of an experiment. This allows for more colorful and more justified decisions than simply terminate or keep waiting.

We have replaced sample size with something called statistical power. It shows the likelihood that a significant result was achieved and accounts for both the volume of data collected and the observed results.

Power is an improvement on sample size. Whereas sample size uses a pre-determined effect size to determine if we have reached ‘sample’, based on the observed number of visitors, power takes into account the volume of data collected AND the observed results to give a more accurate representation of the current reliability of the test results.

We were initially looking at a method of making it easier to "call" an experiment early, including faster stats, less precision, etc. Statistical power allows us to look at the volume of data collected and the currently observed results and make a prediction about whether it “is worth waiting for more data”.

No, sample size is still a very effective way of determining the results of an experiment but is less effective at helping us to make decisions earlier.

If the power meter isn’t increasing over time, you will not learn any more if the current results and volume of data stay the same. In this scenario, you might consider "calling" the experiment early or changing the traffic allocation to 100%.

See Using power forecasts to gauge success for more information about the decisions you can take in our illustrative case studies.

Since the release of Statistical Power, we no longer use the default or custom effect size to determine if we have reached 'sample based on the observed number of visitors. Power takes into account the volume of data collected AND the observed results to give a more accurate representation of the current reliability of the test results.

These metrics will not display if you have selected the All traffic allocation mode. See Traffic allocation for more information.

Experiences in pilot mode are usually “risky” experiences. They are typically run in pilot mode to validate that they aren’t driving massive downturns, rather than to detect small uplifts.

Power is the probability of detecting an uplift of the measured size, assuming that the measured uplift is real (not just a statistical fluctuation). The higher the power, the more certain you can be that you haven’t “missed out” on a significant effect in your data.

If a risky experience running in pilot mode is driving a massive negative effect, the power will become large quickly, and you should end the test once it reaches whichever power you’re comfortable with.

If this same experience is not driving a massive negative effect, and instead has a negligible or positive effect, the power will grow slowly, much slower than with a 50/50 split. Once you’ve de-risked the variation you should republish the experiment to 50/50 in order to benefit from this optimal traffic allocation.

For both of these reasons, you’ll find yourself typically ending your “pilot” tests a little earlier than your 50/50 tests.

Most tests require a large number of visitors in each variation to reach ~70% power. If a customer has only a 5% chance of being put in the control, or a 20% chance of being put in the variation, it will take a much longer time to complete that test. 50/50 is the fastest possible A/B test.

Although this can differ between clients and depends on the configuration of your property, typically the QProtocol events used to report conversions and revenue are identified in the following table:

Goals are attributed for two weeks after an experience is paused to cater for an experience influencing a visitor who purchases slightly later. We believe this is the most accurate way to handle the changes induced by iterations.

A visitor’s conversions and other goals are counted towards experience results only if achieved during the same iteration in which the visitor entered the experience.

Indeed the statistics must not carry conversions/goals across iterations because both the experience itself and the visitor’s allocation to control/variation may well have changed.

Multiple iterations can therefore delay the achievement of statistical significance.