Experience performance export

This is for:

Developer

In this article, we’ll introduce you to our experience performance export and how you can use it to get an insight into the performance of your experiences.

Reporting detail

This export provides insight into how each of your experiences is performing, for example by Conversion Rate, Revenue Per Converter, and Revenue Per Visitor.

As with all Qubit exports, you can specify the level of reporting detail to get a much greater understanding of reportables and focus on the details you are really interested in.

For this export type, you can select:

  • Which experiences to export

  • A date range to include only data covering a defined period of time

  • Exclusion dates to exclude particular days or a period of time from the data

  • Dimensions, including device type and location, which allow you to decide how to break down the data

  • Metrics that allow you to choose which metrics to include in the data

Note

All selected segments are run through the stats model in order to calculate the uplift.

Logic

To export experience performance data, Qubit has built an analytics tool to break down the results of your experiences according to the dimensions you select when configuring the export. For each dimension, the number of visitors, converters, conversions, advanced goal achievers, and the total revenue are reported.

In addition, we apply a Bayesian statistical model to determine the uplifts in goals specified for the experience. This enables users to easily identify which dimensions have significant uplifts in each goal. Dimensions include location, device, browser type, and returning visitor.

In this section, we will take a look at the logic behind this tool.

Breakdown by dimension

The first stage of the analysis involves the breakdown of those visitors that were served the experience according to the selected dimensions and the number of visitors in each experience that met the experience goals. We consider the following standard goals:

  • Conversion Rate (CR) - percentage of visitors who made a purchase during the experience

  • Revenue Per Converter (RPC) - the average spend per converter

  • Revenue Per Visitor (RPV) - the average spend per visitor

These are referred to collectively as primary conversion goals. It is also possible to analyze other goals that you might have added to an experience, including those added using emitCustomGoal. To do this, select the Advanced Goals metric.

Bayesian statistical model

Once the information for each dimension has been gathered, we apply a Bayesian statistical model to the results. Experiences at Qubit typically divide visitors according to those that were served the control and those that were served one or more variations.

For the primary conversion goals, we take the number of visitors, converters, and their spend in each variation and in each dimension, and use the statistical model to compare these numbers with the corresponding values in the control. The model returns the probability of an uplift for the goal, as well as the most likely value for that uplift.

A focus on uplift probability

An uplift of 0 means no change, a positive uplift means the value is larger in the variant than in the control, and a negative uplift means the value is lower in the variant than in the control.

The same analytical method is also used to calculate the Conversion Rate of those visitors that met the objectives of advanced goals; the increase in the percentage of those visitors in the variant who met the advanced goal is compared with the corresponding percentage for the control. There are no equivalents for RPC or RPV for advanced goals.

Results returned

The returned results contain the summarized values for each variant, including the number of visitors, converters, conversion, total revenue, and the number of visitors achieving advanced goals in the variant and the control.

Note

The number of converters, conversions, and revenue reported for advanced goals only takes into account visitors who have already achieved an advanced goal. The returned results also include the mean value of the uplift for the goals, the probability of uplift, and an indication of whether the uplift is significant, that is, the probability of uplift is greater than 90% or 95%.

Dimensions, metrics, and output

Dimensions

Name Description Output

Device Type

The device type receiving the experience

deviceType

OS

The detected operating system

browserVersion

Browser

The detected browser and version number when the experience executed

osName

Country

The detected country the visitor was in when the experience executed

country

Region

The detected region the visitor was in when the experience executed

region

Metro Area

The detected metro area the visitor was in when the experience executed

area

City

The detected city the visitor was in when the experience executed

city

Returning Visitor

Whether the visitor is new or returning

returningVisitor

Metrics

Name Description Output

Advanced Goals

Include all goals defined for an experience, otherwise only primary conversion goals

Refer to the Output table below

Filter Outliers

Remove the top spending 0.1% of converters from the experience.

n/a

Output

Additional fields not included in the dimensions and metrics tables above:

Field Description

experience_name

The name of the experience

variation

The name of the variation

dimension

The dimension type (for example, Country)

dimension_label

The dimension label (for example, United States)

goal_name

Which goal is being measured. Conversion and revenue goals are labeled Primary Conversion

variant_visitors

The number of visitors in this segment the saw the variant

control_visitors

The number of visitors in this segment that saw the control

variant_converters

The number of visitors that converted in the variant. For advanced goals, only converters that already met the goal are counted

control_converters

The number of visitors that converted in the control. For advanced goals, only converters that already met the goal are counted

variant_conversions

The number of conversions in the variant. For advanced goals, only the conversion by visitors that already met the goal is counted

control_conversions

The number of conversions in the control. For advanced goals, only the conversion by visitors that already met the goal is counted

variant_advanced_goal_achieved

The number of times the advanced goal was met in the variant

control_advanced_goal_achieved

The number of times the advanced goal was met in the control

variant_revenue

The revenue from the variant For advanced goals, only revenue from visitors that already met the goal is counted

control_revenue

The revenue from the control For advanced goals, only revenue from visitors that already met the goal is counted

cr_variant

The most probable Conversion Rate for the variant

cr_variant_range_lower

Lower credible value for the variant’s Conversion Rate (95% probability the Conversion Rate is greater than this)

cr_variant_range_upper

Upper credible value for the variant’s Conversion Rate (95% probability the Conversion Rate is less than this)

cr_control

The most probable Conversion Rate for the control

cr_control_range_lower

Lower credible value for the control’s Conversion Rate (95% probability the Conversion Rate is greater than this)

cr_control_range_upper

Upper credible value for the control’s Conversion Rate (95% probability the Conversion Rate is less than this)

cr_uplift_probability

The Conversion Rate uplift probability

cr_uplift

The mean Conversion Rate uplift according to the stats-model

cr_uplift_range_lower

Lower credible value for the Conversion Rate uplift (95% probability the uplift is greater than this)

cr_uplift_range_upper

Upper credible value for the Conversion Rate uplift (95% probability the uplift is less than this)

cr_significant_90pc

Uplift significant at 90% confidence interval

cr_significant_95pc

Uplift significant at 95% confidence interval

rpc_variant

The most probable RPC for the variant

rpc_variant_range_lower

Lower credible value for the variant’s RPC (95% probability the RPC is greater than this).

rpc_variant_range_upper

Upper credible value for the variant’s RPC (95% probability the RPC is less than this)

rpc_control

The most probable Conversion Rate for the control

rpc_control_range_lower

Lower credible value for the control’s RPC (95% probability the RPC is greater than this)

rpc_control_range_upper

Upper credible value for the control’s RPC (95% probability the RPC is less than this)

rpc_uplift_probability

The Revenue Per Converter uplift probability

rpc_uplift

The mean RPC uplift according to the stats-model

rpc_uplift_range_lower

Lower credible value for the PRC uplift (95% probability the uplift is greater than this)

rpc_uplift_range_lower

Upper credible value for the PRC uplift (95% probability the uplift is less than this)

rpc_significant_90pc

Uplift significant at 90% confidence interval

rpc_significant_95pc

Uplift significant at 95% confidence interval

rpv_variant

The most probable RPV for the variant

rpv_variant_range_lower

Lower credible value for the variant’s RPV (95% probability the RPV is greater than this)

rpv_variant_range_upper

Upper credible value for the variant’s RPV (95% probability the RPV is less than this)

rpv_control

The most probable RPV for the control

rpv_control_range_lower

Lower credible value for the control’s RPV (95% probability the RPV is greater than this)

rpv_control_range_upper

Upper credible value for the control’s RPV (95% probability the RPV is less than this)

rpv_uplift_probability

The Revenue Per Visitor uplift probability

rpv_uplift

The mean RPV uplift according to the stats-model

rpv_uplift_range_lower

Lower credible value for the PRV uplift (95% probability the uplift is greater than this)

rpv_uplift_range_lower

Upper credible value for the PRV uplift (95% probability the uplift is less than this)

rpv_significant_90pc

Uplift significant at 90% confidence interval

rpv_significant_95pc

Uplift significant at 95% confidence interval

FAQs

Why are the uplifts and probabilities in the exported data slightly different to those shown on the experiences dashboard?

Qubit’s statistics model is a highly sophisticated Bayesian network, however it can be very slow. To produce exports in a timely fashion, a simplified stats model is used, which may produce slightly different results from time to time.

Why isn’t the Conversion Rate uplift just converters over visitors for the variant divided by converters over visitors for the control?

Experiences at Qubit are divided up into chunks of time called iterations. Sometimes things change between iterations, such as the amount of traffic allocated to each variant or the control. This can lead to very misleading results if the simple calculation outlined above is used. Our statistical model accurately takes these changes into account.

Why are there fewer converters and conversions, and lower revenue for the advanced goal rows?

The converters, conversions, and revenue columns for the advanced_goal row show those values for visitors who have already achieved the advanced goal. The converters row shows the number of visitors who have achieved the advanced goal and then go on to convert.

Why are there never any RPC or RPV results for advanced goals?

Advanced goals track the fraction of visitors who perform some action, such as selecting a particular link or adding an item to a basket. There’s no equivalent RPC or RPV for advanced goals.