Manage A/B tests

This is for:

Developer System Administrator

The A/B Test tab of a query pipeline configuration allows members with the required privileges to evaluate the impact of query pipeline rules changes by pairing query pipelines in one or more A/B tests.

Common use-cases

You’ll typically perform A/B tests to evaluate the impact of query pipeline rule changes.

During a test, you compare two versions of a pipeline by splitting the web traffic between both versions. After the test, you can measure which version provides the best results based on metrics that are significant to your Coveo organization. You can then set the winning pipeline as the effective one for your search interface.

You may want to use the A/B Test tab of a query pipeline configuration in the following scenarios:

Explore potential solutions

Consider the hypothesis you want to resolve before creating your test. Begin by identifying a problem in your organization search usage, and then create an A/B test to experiment potential solutions with a segment of your web traffic.

The test results then allow you to compare the performance of your original pipeline in its actual state against its Test scenario version, in which you tweaked a query pipeline rule to observe its effects on your implementation.

Example

By analyzing the performance of your relevance metrics, you realize that you need to improve your Average Click Rank to reach your relevance objectives.

You then decide to create a new query pipeline rule, hoping this modification will improve your Average Click Rank metric.

You therefore create an A/B test on your query pipeline and add the new rule in the Test scenario version of the pipeline.

To test the efficiency of that new rule, you check the values listed in the Key Performance Indicators section of the A/B Test tab to compare the metrics generated by both pipelines (the Original pipeline VS Test scenario).

After reviewing the results of the A/B test, you notice that the new rule has improved your Average result rank metric.

You then set the Test scenario version of the pipeline as the pipeline in which you want all of the traffic that meets the condition to be redirected.

Adjust the traffic ratio

The A/B Test tab allows you to adjust the traffic ratio you want to send to the Test scenario version of the pipeline, even when the A/B test is already started. You may want to use this tool to speed up or slow down data gathering when experimenting with query pipeline rule modifications.

Example

You create an A/B test that sends 50% of your traffic to your Original pipeline and the other 50% to the Test scenario version of the pipeline.

After a moment, you realize that the Test scenario version of the pipeline isn’t collecting enough data to determine whether the modifications significantly improved your search interface relevance.

You therefore modify the traffic ratio to send 70% of your traffic in the Test scenario version of the pipeline.

About A/B tests

How are A/B tests split

Once you activate an A/B test in your search interface and a query is performed by an end user, the Search API sets a cookie in their web browser to track the session. To determine in which query pipeline (Original pipeline or Test scenario) the query will be sent, the Search API may use the cookie’s session information or the client ID provided in the API request.

Example

In the A/B test configuration panel, you set the A/B test ratio to route 60% of your traffic to the Original pipeline and 40% to its Test scenario version.

Cookies have a 30-minute duration to avoid end users being routed to both the Original pipeline and its Test scenario version in a single visit. When a user makes a new request, the browser includes the cookie in the Search API request, which then sends the cookie to the browser again, resetting the 30-minute duration (see What’s a user visit).

However, if a user exceeds the 30-minute delay without performing a query, the Search API generates a new cookie. According to the A/B test ratio, the user will then be assigned to the Original pipeline or its Test scenario for the duration of that new cookie.

Important

If the end user’s browser blocks third-party cookies, the Search API uses the value of the visitorId query parameter to identify the user and ensure their queries are routed to the right query pipeline (for example, Original pipeline or Test scenario).

Query pipeline routing condition management in A/B tests

A query must be routed to the Original pipeline to trigger an A/B test (see Routing rules). If this condition is met, the Search API then routes end users to the appropriate query pipeline following the A/B test ratio.

Note

In the context of an A/B test, the condition applied to the Test scenario version of the pipeline is ignored.
Example

You create an A/B test with a ratio routing 20% of your traffic to the Original pipeline and 80% to its Test scenario.

You’re using the condition-based routing mechanism, and queries are routed to the original query pipeline when the following condition is satisfied:

Search hub is communitySearch.

An end user sends a query from the community search interface, which triggers the A/B test (because the query pipeline condition is satisfied).

At this point, the query has an 80% chance of being routed to the Test scenario of the query pipeline (ignoring its condition, if any), and a 20% chance of being routed to the original query pipeline.

Create an A/B test

  1. On the Query Pipelines (platform-ca | platform-eu | platform-au) page, click the query pipeline in which you want to create an A/B test, and then click Edit components in the Action bar.

  2. In the A/B Test tab, click Configure A/B Test.

  3. In the Configuration section, under Traffic Split, move the slider to choose the traffic ratio you want to send to the Original pipeline versus its Test scenario version.

  4. In the Test scenario section, click Edit to manage query pipeline components of the Test scenario version of the pipeline.

  5. On the Test Scenario page that opens, add, edit, or delete query pipeline rules to build your A/B test.

  6. Once you’re done adding or editing rules, click A/B test on the top of the page to return to the A/B test tab.

    A/B test tab | Coveo

  7. In the A/B Test tab, click Start to activate the A/B test.

You can pause or stop your A/B test at any moment.

Note

When you create an A/B test, a mirror of the original pipeline is created and is used to store the configuration of the Test scenario version of the pipeline. This mirror pipeline isn’t visible on the Query Pipelines page, but you can find it on the Content Browser (platform-ca | platform-eu | platform-au) page, where you can use it to test the pipeline and investigate search relevance issues.

Mirror pipeline in Content Browser | Coveo

The mirror pipeline remains visible in the Content Browser when the A/B test is active or paused, and is automatically deleted when you stop the test.

Edit an A/B test

  1. On the Query Pipelines (platform-ca | platform-eu | platform-au) page, click the query pipeline in which you want to edit an A/B test, and then click Edit components in the Action bar.

  2. In the A/B Test tab, in the Configuration section, click Edit A/B Test.

  3. In the Edit A/B Test Configuration panel that opens, in the Configuration section, under Traffic Split, move the slider to choose the traffic ratio you want to send to the Original pipeline and its Test scenario version.

  4. In the Edit A/B Test Configuration panel that opens, in the Configuration section, under Traffic Split, move the slider to choose the traffic ratio you want to send to the Original pipeline and its Test scenario version.

  5. In the Test scenario section, click Edit to manage query pipeline components of the Test scenario version of the pipeline.

  6. On the Test Scenario page that opens, add, edit, or delete query pipeline rules to build your A/B test.

  7. Once you’re done adding or editing rules, click A/B test on the top of the page to return to the A/B test tab.

    A/B test tab | Coveo

  8. In the A/B Test tab, click Start to activate the A/B test.

Pause an A/B test

You may want to temporarily stop sending traffic to the Test scenario version of your original pipeline.

  1. On the Query Pipelines (platform-ca | platform-eu | platform-au) page, click the query pipeline in which you have an active A/B test, and then click Edit components in the Action bar.

  2. In the A/B Test tab, click Pause.

You can restart the A/B test at any moment by clicking Start.

Stop an A/B test

After running an A/B test, you’ll be either satisfied or unsatisfied with the changes brought to the Test scenario version of the pipeline. Whatever the case, you need to end the A/B test in order to stop sending traffic to the Test scenario version of the original pipeline.

  1. On the Query Pipelines (platform-ca | platform-eu | platform-au) page, click the query pipeline in which you have an active A/B test, and then click Edit components in the Action bar.

  2. In the A/B Test tab, click Stop.

  3. In the Stop A/B Test panel that appears, select either:

    • Discard test scenario if you want the rules of the Original pipeline to still apply.

    • Use test scenario if you want the rules of the Test scenario version of the pipeline to apply from now on in your pipeline.

    • Create a new pipeline with test scenario if you want to create a new query pipeline with the configuration of the Test scenario. This new query pipeline will appear on the Query Pipelines (platform-ca | platform-eu | platform-au) page with the following name: [name of the original pipeline]-A/B-test-mirror.

      New query pipeline on the Query Pipelines page

  4. Click Confirm.

The A/B test is now stopped. No more traffic will be sent to the Test scenario version of the pipeline.

Review A/B tests performance

You have two avenues for reviewing the performance of your A/B tests: the A/B Test tab and the built-in A/B Test dashboard report.

The A/B Test tab displays key metrics that allow you to analyze the performance of the Original pipeline against that of the Test scenario. These metrics include Average Result Rank, Search Clickthrough, Average Number of Results, and Queries Without Results.

  1. On the Query Pipelines (platform-ca | platform-eu | platform-au) page, click the query pipeline in which you have an active A/B test, and then click Edit components in the Action bar.

  2. In the A/B Test tab, if an A/B test is running, in the Key Performance Indicators section, review the available metrics.

Based on shown metric results for your Original Pipeline and the Test scenario, you can see which pipeline provided the best search relevance.

While this tab provides a quick snapshot, it analyzes a maximum of 60 days' worth of data. For a more comprehensive and accurate analysis, we recommend using the built-in A/B Test dashboard report available in the Reports (platform-ca | platform-eu | platform-au) page of the Administration Console. This dashboard allows for a more granular view of your metrics and doesn’t have the 60-day data analysis limitation.

Leading practices

When managing A/B tests, consider the following recommendations and tips:

  • Your A/B tests should only contain subtle changes between the Original pipeline and its Test scenario so that you always know the reason for a positive outcome.

  • Set yourself achievable goals in order to improve your relevance metrics.

    Example

    Improve the Search Event Clickthrough (%) ratio of the 10 queries with lowest Relevance Index by 10%.

  • Your test results must be significant for you to conclude whether the Test scenario version of the pipeline is better than the Original pipeline. You can use tools like SurveyMonkey to calculate an A/B test statistical significance.

  • Often the results of A/B tests can be surprising. You should thus not reject an A/B test result based on your arbitrary judgment.

  • The difference between the Original pipeline and its Test scenario, as well as the sample size, are two key factors to look into before drawing any conclusion on an A/B test.

    Example

    You want to test your results regarding the clickthrough ratio with a couple of changes you made in your default pipeline. You get a clickthrough ratio of 30% with the Original pipeline and 35% with its Test scenario. The base size of your two samples is 1,000 queries. With these numbers, your result is statistically significant 19 times out of 20 (95% confidence level), so you make the Test scenario version of the pipeline effective on your search page.

  • Make your A/B test consistent across your entire search page.

    Example

    When a user searches for char and you enter a thesaurus rule that replaces char with characters, this behavior should be the same across every search box of your search interface.

  • When you want to deploy a query pipeline change in the interface to which your main traffic is directed, we recommend that you first assign a very small percentage of your audience to the Test scenario version of the pipeline to test the change behavior. If the change behaves as expected, you can then gradually increase the percentage routed to the Test scenario version of the pipeline.

  • We recommend that you perform many A/B tests so that you may compile the positive results to boost your expected outcome.

Reference

"Key Performance Indicators" section

This section allows you to review the performance of the Original pipeline compared to that of the Test scenario. Available metrics are:

Metric Definition

Average result rank

The average position of opened items from the search results.

Search clickthrough

The percentage of search events followed by a click.

Average number of results

The average number of results for a query.

Query without results

The number of queries that returned no results.

Required privileges

By default, members with the required privileges can view and edit elements of the Query Pipelines (platform-ca | platform-eu | platform-au) page.

The following table indicates the required privileges to view or edit A/B tests (see Manage privileges and Privilege reference).

Action Service - Domain Required access level

View A/B tests

Organization - Organization
Search - Query pipelines
Analytics - Analytics data

View

Edit A/B tests

Organization - Organization
Analytics - Analytics data

View

Search - Query pipelines

Edit

Search - Execute queries

Allowed