Introduction to experiences

This is for:

Developer

In this article, we’ll introduce you to the basic experience flow and then show you how to find out what’s going on with your experiences, both property-wide and individually.

The basic flow

Once your experience has been published, you can expect it to be live on your site within a few minutes. At this point, once triggers have passed, qualifying visitors (as defined in the segments you are targeting) will join the experience and start to interact with it. This interaction generates events.

Important among these events are qubit.experience–emitted when a visitor joins the experience, and goal events–emitted when a visitor achieves an experience goal.

We use these events and the associated data to not only notify you about the health of an experience but also to report on the value the experience is delivering and its performance against goals.

Being able to analyze the value and performance of your experiences is a critical pathway in the personalization journey and key to delivering against your personalization goals.

So how do you go about finding and understanding the metrics associated with value and performance?

Intro page

You can get a top-level view of key experience metrics across your property in your Overview page:

overview-page
Note

Incremental revenue on the Overview page is site-wide and aggregates the revenue from all live, paused, and archived experiences for your property.

Experience list page

In your list of experiences, we take these metrics down to the individual experience level. Using this view allows you to get a heads up on performance and see if any experiences are having a negative impact on revenue or are showing signs of poor health:

experience-page

Searching for experiences

To help you find the right experience, when you perform a search, we’ll group the matching experiences according to state: Live, Scheduled, Drafts, Archived and favorites.

In the following example, the user has searched using the string "Exper". The search has returned four matches, across four different states:

experience-search

Experience details page

You can go even further into the metrics by opening an experience. This will give you the complete picture covering health, performance, and value:

experience-view

FAQs

In the Health card, I see Visitors and Conversions. What do these metrics mean?

These metrics are an aggregation of the visitors exposed to your experience–this includes the control and any variations.

Let’s explain each of these metrics:

  • Visitors - An aggregate total of the number of visitors that saw the experience, across all iterations, over either the last 24hrs or since the experience went live

  • Conversions - An aggregate total of the number of visitors that saw the experience and then converted, across all iterations, over the last 24hrs or since the experience went live

For my experience, no metrics are being shown. What is the likely cause?

The Health card is your first port of call for diagnosing issues with your experience.

When no metrics are being down, the likely cause is that visitors are not seeing your experience. This might be because your segment conditions are too restrictive or that the trigger conditions are not being met.

As a first step, we recommend working with a member of your dev team to check segmentation and triggers are working as you expect.

We also recommend using the Qubit Explorer to make sure that the correct events are being emitted and that experience is being triggered as you would expect.

You can read more about the Qubit Explorer and validating your QP setup in Validating Your Setup.

What is a converter?

If your primary goal is Conversions, a converter could be a visitor that purchased on your site, but this is not always the case, because it depends on how conversion has been defined for your property.

Conversion could be transaction based but equally could be based on phone calls received or form submissions. For more information, reach out to your CSM at Qubit.

I’ve paused an experience but I continue to see conversions, why is that?

Any conversions that occurred before the experience was paused will be counted for two weeks after the experience was paused.

What do the columns on the Experiences page mean?

On the Experience page, we display some basic information to give you a quick overview of what is happening with your experiences:

  • Visitors - A real-time count of visitors that have been exposed to the experience. If the count is greater than 1000, we round to the nearest hundred. For example, we would round 70,895 to 70,900

  • Exposure - The traffic split for the experience. See Traffic allocation for more information

  • Updated - A timestamp for the last change that was made to the experience and the user that made that change

  • Upift - The observed change between the control and variations for the primary and any secondary goals. A 0% uplift means there is no difference, and a negative uplift (or downlift) means that the control actually did better than a variation:

    uplift
Note

At Qubit we express this change as a percentage. So if the control has a Conversion Rate of 4%, and the variation has a Conversion Rate of 5%, we would say this is a 25% percent uplift, since 5 is 125% of 4

  • Power - The probability that a meaningful result was achieved. See Power

  • Revenue - The estimated incremental value of the experience, factoring in power

Why is the Incremental Revenue card not shown?

Right now, the card is being activated on a per-customer basis. If you don’t see it, reach out to your CSM at Qubit.

Why is the revenue metric not displayed for an experience?

We report incremental revenue for experiences that meet the following criteria:

  1. Less than 100% exposure - Experiences running at 100% exposure cannot be included in the revenue calculations because it’s not possible to make a comparison between the control and the variation if there is no control group

  2. RPV goal - We use RPV to compute incremental revenue

  3. Minimum of 500 converters per variation - For split traffic experiences, we have a minimum threshold of 500 converters that needs to be passed before

  4. Been live for a minimum of seven days - To report revenue, with the highest level of confidence, we require an experience to have been live for at least seven days. This ensure that enough data has been calculated to mitigate against short term seasonal effects that could be misleading when reporting on revenue

What inferences can I take from other metrics whilst I’m waiting for an experience to report incremental revenue?

For guidance of performance over shorter time periods, you can take inferences from other metrics such as Conversion Rate and Revenue Per Visitor. Bear in mind however that these are not wholly indicative of the actual incremental revenue that has been driven.

What experiences will I find in the different views: Live, Scheduled, Paused, etc?

Tip
Leading-practice

It is worth remembering that an experience can display in multiple views. For example, a live experience that was published according to a schedule, will display in both the Live view and in the Scheduled view. In addition, if there are unpublished changes for that experience, it will also display in the Drafts view.

Live

  • Currently live on your site or mobile app

  • Contributes to the weight of Smartserve

Scheduled

  • A live experience with a future end date/time. On end, the experience is moved to Paused

  • An unpublished experience with a scheduled start date/time. On start, the experience is also shown in Live

Paused

  • Previously live on your site but has since been paused either manually or by a scheduled end date/time

  • When recently paused, contributes to the weight of Smartserve

  • After sixty days, paused experiences are moved to the Archived view

Draft

  • Being worked on and not yet published

  • Currently live with unpublished changes

  • Does not contribute to the weight of Smartserve

Archived

  • Does not contribute to the weight of Smartserve

  • Can be permanently deleted or unarchived

Favorites

If you have some experiences that you regularly use, you can select them as favorites. When you do this, the experience will also display in the Favorites list. We will cover experience status and how you select an experience as a favorite in more detail later on.

  • Shows any experiences that you have selected as a favorite

  • Provides an easy way to access those experiences you use most often

To add an experience as a favorite, select more against your experience and then select Add to favorites. You can remove it by Remove from favorites

Will visitors continue to see experiences after it has been paused?

Yes, this is possible. Qubit re-publishes the Smartserve script when an experience is paused but because we have no control over the browser, it is possible that a cached version of Smartserve will continue to be used. Until the browser cache is cleared, whether manually or automatically, visitors may continue to be served an experience.