Impressions and clickthroughs

This is for:

Developer

This article will cover impression and clickthrough events and how they are used to measure the impact of a campaign. You’ll also see best practices for avoiding bias when implementing these events.

Measuring impact through impressions and clickthroughs

After running a campaign, merchandisers will want to know what kind of impact it had. Qubit measures impact using an A/B test (a randomized controlled test). The Placement will collect the data for this measurement.

To understand the impact, you need to know who saw it, what they did afterward, and then compare this to people who did not see the campaign:

  • To understand who was eligible to see the campaign, Coveo Merchandising Hu (CMH) has Placement impression events emitted when you call onImpression.

  • To measure engagement, CMH have Placement interaction events emitted when you call onClickthrough.

  • To determine what happened afterward, CMH uses your qubit protocol implementation (and particularly ecBasketTransactionSummary for conversions).

There are both generic and product-specific impression and interaction events. See below for details.

For a meaningful comparison between people that saw the campaign and people that did not, they must be allocated randomly. This random allocation ensures there’s no bias and that any differences measured will be because caused by the campaign itself and not because the two groups exhibit inherently different behaviors.

Placements should therefore call onImpression not only for people that saw the campaign but also for people that would have seen the campaign if they had not been randomly allocated to the control group.

While the examples below refer to Placements injected via the Smartserve script, the principles apply equally to Placements retrieved via API and reported on using callback URLs directly.

Introducing bias

Here are some examples of how inadvertent bias might be introduced within your Placement implementation:

In the following example, if there is content, the onImpression event is emitted after the user has scrolled. However, if there is no content, the onImpression event is emitted regardless of whether they scroll.

module.exports = function renderPlacement ({
  elements: [$el],
  content,
  onImpression,
  onEnterViewport
}) {
  // An implementation that introduces bias and therefore invalidates your data.
  // Don't do this!
  if (content) {
    // When there is content, the visitor is registered only when they scroll
    onEnterViewport($el, onImpression)
  } else {
    // When there is no content, all visitorsvare registered
    onImpression()
  }
}

In other words, for this campaign, you will be comparing people that scroll down to our element with the general population on site. These groups are not allocated randomly and are likely to exhibit inherently different behaviors.

The problem with this approach is that if you see a difference, it may not be because of the campaign but because you are comparing highly engaged users (people who scrolled to the campaign) with everyone else.

Similarly, the following example compares users who scroll a lot with users who scroll a little. Again, you can’t be sure whether any difference you see in the reporting will be because of the campaign or because the two groups exhibit inherently different behaviors:

module.exports = function renderPlacement ({
  elements: [$header, $footer],
  content,
  onImpression,
  onEnterViewport
}) {
  // An implementation that introduces bias and therefore invalidates your data.
  // Don't do this!
  if (content) {
    // Here we only show the variant to users who scroll a little bit
    onEnterViewport($header, onImpression)
  } else {
    // Here we only show the variant to users who scroll a lot
    onEnterViewport($footer, onImpression)
  }
}

Best practices for avoiding bias

Set up impression events upfront before branching into control and variant (content and no content)

Doing this has several advantages. Firstly, it will make it much harder to introduce conditions that specifically bias either the control or the variant. Secondly, it should make it much easier to review the code and determine whether it introduces bias.

Emit impression events before performing any rendering logic

In addition to the advantages stated above, emitting the impression events first ensures that any errors or bugs in the rendering logic will not affect whether or not we send impression events. This approach also means errors can be caught and reported separately and help avoid compounding rendering problems by introducing statistical bias.

Create a common element to detect entering the viewport

A good way of ensuring that you are scrolling to an equivalent point in the page in the control and variation (content and no content) is to create a common element and inject it regardless of where you intend to render your content.

Important

Do not use the same common element for rendering the content and tracking impressions.

Instead, use one common element for content and another common element for emitting the impression events.

You can then use that element to trigger your onImpression event. Here’s a simple example of that approach:

const React = require('preact')
const {
  style,
  onEvent,
  insertBefore,
  onEnterViewport,
  restoreAll
} = require('@qubit/utils/dom')()

module.exports = function renderPlacement ({
  content,
  onImpression,
  onClickthrough,
  onRemove,
  elements: [target]
}) {
  onRemove(restoreAll)

  // Create a dummy element here that you will use to track who saw this Placement
  // Using the same element regardless of content ensures consistency in your tracking
  const element = document.createElement('div')
  insertBefore(target, element)

  // Call onImpression when our element enters the viewport
  // This ensures you are not only reporting on people who have seen the banner,
  // but also on people in the control and scrolled to where the banner would have been if they were in the variant.
  // Calling onImpression upfront helps to ensure you don't unintentionally exclude visitors when performing an AB test
  onEnterViewport(element, onImpression)

  if (content) {
    // Render content
  } else {
    // Add some logic for control
  }
}

Report clickthroughs for control when possible

If you are injecting a new element on the page, you won’t be able to do this. However, if you are replacing an existing banner, for example, that has click handlers, you’ll need to make sure you call onClickthrough when that banner is clicked on too. This approach will ensure you can fairly compare the clicks on our variant to the ones on our control.

Example:

const React = require('preact')
const {
  style,
  onEvent,
  insertBefore,
  onEnterViewport,
  restoreAll,
  replace
} = require('@qubit/utils/dom')()

module.exports = function renderPlacement ({
  content,
  onImpression,
  onClickthrough,
  onRemove,
  elements: [target]
}) {
  onRemove(restoreAll)

  // Here it's assumed our content is configured to return a cta
  const { cta } = content

  // `onImpression` logic hidden for conciseness

  if (content) {
    // Ensure your new component calls `onClickthrough`
    const banner = <div><a onClick={onClickthrough}>{cta}</a></div>
    // Using a helper to replace the website's banner with our new one
    replace(target, banner)
  } else {
    // Here we use another helper called `onEvent`
    // and make sure our existing banner also calls `onClickthrough` for users that don't get content.
    onEvent(target.querySelector('a'), 'click', onClickthrough)
  }
}

Product events

In product recommendations Placements, it’s recommended that you also report on what products are being shown. This approach will give merchandisers some insight into what is being recommended for both variant and control and what people are clicking on.

To do that, you should pass 'product' as the first argument to the function, and the product Id or product Ids as the second.

Example:

const React = require('preact')
const {
  style,
  onEvent,
  insertBefore,
  onEnterViewport,
  restoreAll,
  replace
} = require('@qubit/utils/dom')()
const { map, forEach } = require('slapdash')

module.exports = function renderPlacement ({
  content,
  onImpression,
  onClickthrough,
  onRemove,
  elements: [target]
}) {
  onRemove(restoreAll)

  // This is the default schema for a Recommendation Placement
  const { headline, recs } = content

  const element = document.createElement('div')
  insertBefore(target, element)

  // Keeping this onImpression event here will ensure we are not causing bias with the generic (non-product) event
  // It also makes it easier to debug and reason about your code
  onEnterViewport(element, onImpression)

  if (content) {
    const productIds = recs.map(rec => rec.details.id)
    // Both `onImpression` and `onClickthrough` can handle a string or an array of strings as the second argument.
    // This way you only need to call this function once to report on all the ids being showed.
    // You could add extra logic if you have products that are not immediately shown (hidden on other pages of the carousel) if you prefer.
    onEnterViewport(element, () => onImpression('product', productIds))

    // A simple carousel that returns a clickable image for each of the products recommended in our content
    const carousel = (
      <div>{renderRecs(recs, onClickthrough)}</div>
    )

    // Using a helper to replace the website's banner with our new one
    replace(target, banner)
  } else {
    // If you are replacing an existing carousel, you can also report on the products that were shown, much like point 3 above.

    const prodEls = target.querySelectorAll('.product_element')
    // You'll need to find a way of getting the product ids from the dom. In the example below, it's assumed that  your elements have a data-product-id attribute
    const productIds = map(prodEls, el => el.dataset.productId)
    onEnterViewport(element, () => onImpression('product', productIds))

    // The `onClickthrough` events is also appended to each of the products already on the carousel
    forEach(prodEls, el => {
      const prodId = el.dataset.productId
      const anchor = el.querySelector('a')
      onEvent(anchor, 'click', () => onClickthrough('product', prodId))
    })
  }
}

function renderRecs(recs, onClickthrough) {
  const clickHandler = (id) => onClickthrough('product', rec.details.id)

  return recs.map((rec) => (
    // Here the report only includes the specific ID the user clicked on.
    <a href={rec.details.url} onClick={clickHandler(rec.details.id)}>
      <img src={rec.details.image_url} />
    </a>
  ))
}