Manage A/B Tests

The A/B Test tab of a query pipeline configuration allows users with the required privileges to evaluate the impact of query pipeline rules changes by pairing query pipelines in one or more A/B tests.

Common Use-Cases

You will typically perform A/B tests to evaluate the impact of query pipeline rule changes.

During a test, you compare two versions of a pipeline by splitting the web traffic between both versions. After the test, you can measure which version provides the best results based on metrics that are significant to your Coveo organization. You can then set the winning pipeline as the effective one for your search interface.

You may want to use the A/B Test tab of a query pipeline configuration in the following scenarios:

Explore Potential Solutions

Consider the hypothesis you want to resolve before creating your test. Begin by identifying a problem in your organization search usage, and then create an A/B test to experiment potential solutions with a segment of your web traffic.

The test results then allow you to compare the performance of your original pipeline in its actual state against its Test scenario version, in which you tweaked a query pipeline rule to observe its effects on your implementation.

EXAMPLE

By analyzing the performance of your relevance metrics, you realize that you need to improve your Average Click Rank to reach your relevance objectives.

You then decide to create a new query pipeline rule, hoping this modification will improve your Average Click Rank metric.

You therefore create an A/B test on your query pipeline and add the new rule in the Test scenario version of the pipeline.

To test the efficiency of that new rule, you check the values listed in the Key Performance Indicators section of the A/B Test tab to compare the metrics generated by both pipelines (the Original pipeline VS Test scenario).

After reviewing the results of the A/B test, you notice that the new rule has improved your Average result rank metric.

You then set the Test scenario version of the pipeline as the pipeline in which you want all of the traffic that meets the condition to be redirected.

Adjust the Traffic Ratio

The A/B Test tab allows you to adjust the traffic ratio you want to send to the Test scenario version of the pipeline, even when the A/B test is already started. You may want to use this tool to speed up or slow down data gathering when experimenting with query pipeline rule modifications.

EXAMPLE

You create an A/B test that sends 50% of your traffic to your Original pipeline and the other 50% to the Test scenario version of the pipeline.

After a moment, you realize that the Test scenario version of the pipeline isn’t collecting enough data to determine whether the modifications significantly improved your search interface relevance.

You therefore modify the traffic ratio to send 70% of your traffic in the Test scenario version of the pipeline.

About A/B Tests

How Are A/B Tests Split

Once you activate an A/B test in your search interface and a query is performed by an end user, the Search API sets a cookie in their web browser. The Search API then uses this cookie to determine in which query pipeline (Original pipeline or Test scenario) the query will be sent.

EXAMPLE

In the A/B test configuration panel, you set the A/B test ratio to route 60% of your traffic to the Original pipeline and 40% to its Test scenario version.

Cookies have a 30-minute duration to avoid end users being routed to both the Original pipeline and its Test scenario version in a single visit. When a user makes a new request, the browser includes the cookie in the Search API request, which then resends the cookie to the browser, resetting the 30-minute duration (see What’s a User Visit).

However, if a user exceeds the 30-minute delay without performing a query, the Search API generates a new cookie. According to the A/B test ratio, the user will then be assigned to the Original pipeline or its Test scenario for the duration of that new cookie.

A small percentage of users might have disabled cookies in their browser settings or might use an extension that blocks cookies. Therefore, they could switch from both the Original pipeline and its Test scenario during a single session. On average these users shouldn’t impact aggregated metrics, but if you look closely into the details, you could see abnormal values.

Query Pipeline Routing Condition Management in A/B Tests

A query must be routed to the Original pipeline to trigger an A/B test (see Routing Rules). If this condition is met, the Search API then routes end users to the appropriate query pipeline following the A/B test ratio.

In the context of an A/B test, the condition applied to the Test scenario version of the pipeline is ignored.

EXAMPLE

You create an A/B test with a ratio routing 20% of your traffic to the Original pipeline and 80% to its Test scenario.

You’re using the condition-based routing mechanism, and queries are routed to the original query pipeline when the following condition is satisfied:

Search hub is communitySearch.

An end user sends a query from the community search interface, which triggers the A/B test (because the query pipeline condition is satisfied).

At this point, the query has an 80% chance of being routed to the Test scenario of the query pipeline (ignoring its condition, if any), and a 20% chance of being routed to the original query pipeline.

Create an A/B Test

  1. On the Query Pipelines page, click the query pipeline in which you want to create an A/B test, and then in the Action bar, click Edit Components.

  2. In the A/B Test tab, click Configure A/B Test.

  3. In the Configuration section, under Traffic Split, move the slider to choose the traffic ratio you want to send to the Original pipeline versus its Test scenario version.

  4. In the Test scenario section, click Edit to manage query pipeline components of the Test scenario version of the pipeline.

  5. In the Test Scenario panel that opens, add, edit, or delete query pipeline rules to build your A/B test.

  6. Once you’re done managing the query pipeline rules of the Test Scenario version of the pipeline, at the bottom of the panel, click Close.

  7. Back in the A/B Test tab, click Start to activate the A/B test.

You can pause or stop your A/B test at any moment.

Edit an A/B Test

  1. On the Query Pipelines page, click the query pipeline in which you want to edit an A/B test, and then in the Action bar, click Edit Components.

  2. In the A/B Test tab, in the Configuration section, click Edit A/B Test.

  3. In the Edit A/B Test Configuration panel that opens, in the Configuration section, under Traffic Split, move the slider to choose the traffic ratio you want to send to the Original pipeline and its Test scenario version.

  4. In the Test scenario section, click Edit to manage query pipeline components of the Test scenario version of the pipeline.

  5. In the Test Scenario panel that opens, add, edit, or delete query pipeline rules to build your A/B test.

  6. Once you’re done managing the query pipeline rules of the Test Scenario version of the pipeline, at the bottom of the panel, click Close.

  7. Back in the Edit A/B Test Configuration panel, click Confirm to save your changes.

You can pause or stop your A/B test at any moment.

Pause an A/B Test

You may want to temporarily stop sending traffic to the Test scenario version of your original pipeline.

  1. On the Query Pipelines page, click the query pipeline in which you have an active A/B test, and then in the Action bar, click Edit Components.

  2. In the A/B Test tab, click Pause.

You can restart the A/B test at any moment by clicking Start.

Stop an A/B Test

After running an A/B test, you will be either satisfied or unsatisfied with the changes brought to the Test scenario version of the pipeline. Whatever the case, you need to end the A/B test in order to stop sending traffic to the Test scenario version of the original pipeline.

  1. On the Query Pipelines page, click the query pipeline in which you have an active A/B test, and then in the Action bar, click Edit Components.

  2. In the A/B Test tab, click Stop.

  3. In the Stop A/B Test panel that appears, select either:

    • Discard test scenario if you want the rules of the Original pipeline to still apply.

    • Use test scenario if you want the rules of the Test scenario version of the pipeline to apply from now on in your pipeline.

    • Create a new pipeline with test scenario if you want to create a new query pipeline with the configuration of the Test scenario. This new query pipeline will appear on the Query Pipelines page with the following name: [name of the original pipeline]-A/B-test-mirror.

      New query pipeline in the query pipelines page
  4. Click Confirm.

The A/B test is now stopped. No more traffic will be sent to the Test scenario version of the pipeline.

Review A/B Tests Performance

While it’s possible to use the default "A/B Testing" dashboard template on the Reports page of the Administration Console to report on your A/B tests performance, the A/B Test tab displays key metrics allowing you to analyze the performance of the Original pipeline against that of the Test scenario.

  1. On the Query Pipelines page, click the query pipeline in which you have an active A/B test, and then in the Action bar, click Edit Components.

  2. In the A/B Test tab, if an A/B test is running, in the Key Performance Indicators section, review the available metrics.

Based on shown metric results for your Original Pipeline and the Test scenario, you can see which pipeline provided the best search relevance.

Leading Practices

When managing A/B tests, consider the following recommendations and tips:

  • Your A/B tests should only contain subtle changes between the Original pipeline and its Test scenario so that you always know the reason for a positive outcome.

  • Set yourself achievable goals in order to improve your relevance metrics.

    EXAMPLE

    Improve the Search Event Click-Through (%) ratio of the 10 queries with lowest Relevance Index by 10%.

  • Your test results must be significant for you to conclude whether the Test scenario version of the pipeline is better than the Original pipeline. You can use tools like SurveyMonkey to calculate an A/B test statistical significance.

  • Often the results of A/B tests can be surprising. You should thus not reject an A/B test result based on your arbitrary judgment.

  • The difference between the Original pipeline and its Test scenario, as well as the sample size, are two key factors to look into before drawing any conclusion on an A/B test.

    EXAMPLE

    You want to test your results regarding the click-through ratio with a couple of changes you made in your default pipeline. You get a click-through ratio of 30% with the Original pipeline and 35% with its Test scenario. The base size of your two samples is 1,000 queries. With these numbers, your result is statistically significant 19 times out of 20 (95% confidence level), so you make the Test scenario version of the pipeline effective on your search page.

  • Make your A/B test consistent across your entire search page.

    EXAMPLE

    When a user searches for char and you enter a thesaurus rule that replaces char with characters, this behavior should be the same across every search box of your search interface.

  • When you want to deploy a query pipeline change in the interface to which your main traffic is directed, we recommend that you first assign a very small percentage of your audience to the Test scenario version of the pipeline to test the change behavior. If the change behaves as expected, you can then gradually increase the percentage routed to the Test scenario version of the pipeline.

  • We recommend that you perform many A/B tests so that you may compile the positive results to boost your expected outcome.

  • (When you use a proxy) Before activating an A/B test, ensure that the cookies used by Coveo are forwarded. If not, you need to add those cookies to the list of forwarded cookies.

Reference

"Key Performance Indicators" Section

This section allows you to review the performance of the Original pipeline compared to that of the Test scenario. Available metrics are:

Metric Definition

Average result rank

The average position of opened items from the search results.

Search click-through

The percentage of search events followed by a click.

Average number of results

The average number of results for a query.

Query without results

The number of queries that returned no results.

Required Privileges

By default, members of the Administrators and Relevance Managers built-in groups can view and edit elements of the Query Pipelines page.

The following table indicates the required privileges to view or edit elements of the Query Pipelines page (see Manage Privileges and Privilege Reference).

Action Service - Domain Required access level

View A/B tests

Search - Query pipelines

View

Edit A/B tests

Search - Query pipelines

Edit

Recommended Articles