About A/B Tests

You can use A/B tests to compare two versions of a query pipeline. You typically use an A/B test to measure the effectiveness of a change made within a specific query pipeline. Once the A/B test is completed, you should be able to tell whether your modification has improved the relevance of the search experience you’re providing (see Query Pipeline A/B Test Management).

In Coveo organizations that have gone through the query pipeline migration process, you can’t configure A/B tests using the A/B Tests page of the Administration Console.

You must use the “A/B Test” tab of a pipeline configuration instead.

How Are A/B Tests Split

Once you activate an A/B test in your search interface and a query is performed by an end-user, the Search API sets a cookie in their web browser. The Search API then uses this cookie to determine in which of the two tested query pipelines the query will be sent (see Manage A/B Tests - Create an A/B Test).

In the A/B test configuration panel, you set the A/B test ratio to route 60% of your traffic to pipeline A and 40% to pipeline B.

Cookies have a 30 minutes duration to avoid end users being routed to both pipeline A and pipeline B in a single visit. When a user makes a new request, the browser includes the cookie in the Search API request, which then resends the cookie to the browser, resetting the 30 minutes duration (see What’s a User Visit).

However, if a user exceeds the 30 minutes delay without performing a query, the Search API generates a new cookie. According to the A/B test ratio, the user will then be assigned pipeline A or pipeline B for the duration of that new cookie.

A small percentage of users might have disabled cookies in their browser settings or might use an extension that blocks cookies. Therefore, they could switch from both pipeline A and B during a single session. On average these users shouldn’t impact aggregated metrics, but if you look closely into the details, you could see abnormal values (see Analyze the Performance of Pipeline A Vs Pipeline B - People to exclude).

Query Pipeline Routing Condition Management in A/B Tests

A query must be routed to pipeline A to trigger an A/B test (see Routing Rules). If this condition is met, the Search API then routes end users to the appropriate query pipeline following the A/B test ratio.

In the context of an A/B test, the condition on query pipeline B is ignored.

You decide to compare Production (A) and Test (B) query pipelines in an A/B test.

You set the A/B test ratio to route 20% of your traffic to the Production pipeline and 80% to the Test pipeline.

You’re using the condition-based routing mechanism, and queries are routed to the Production query pipeline when the following condition is satisfied:

  • Search hub is communitySearch.

An end user sends a query from the community search interface, which triggers the A/B test (because the Production query pipeline condition is satisfied).

At this point, the query has 80% chances of being routed to the Test query pipeline (ignoring its condition, if any), and 20% chances of being routed to the Production query pipeline.

Recommended Articles