Manage A/B Tests

You can use the A/B Tests page1 in the Coveo Cloud Administration Console navigation menu to efficiently evaluate the impact of query pipeline rule changes by pairing query pipelines in one or more A/B tests.

1: In Coveo organizations that have gone through the query pipeline migration process, you can’t configure A/B tests using the A/B Tests page of the Administration Console.

You must use the “A/B Test” tab of a pipeline configuration instead.

During a test, you subject two versions of a pipeline to experimentation simultaneously by splitting the web traffic between both versions. In the end, you can measure which version has the best results based on metrics that are significant to your Coveo organization, and then set the winning pipeline as the effective one for your search interface.

By analyzing your relevance metrics performance, you realize that you need to improve your Average Click Rank to reach your relevance objectives.

You then decide to create a new query pipeline rule, hoping this modification will improve your Average Click Rank metric.

You therefore duplicate the pipeline and add the rule to the duplicated version (see Leading Practices).

To test the efficiency of that new rule, you create an A/B test in which you compare both versions of the query pipeline, i.e., the original production pipeline (A) and the pipeline with the new rule to test (B).

After reviewing the results of the A/B test, you notice that the new rule has indeed improved your Average Click Rank metric (see Analyze the Performance of Pipeline A Vs Pipeline B).

You therefore copy the new rule from the test pipeline (B) to the production pipeline (A) (see Copy a Rule to Another Pipeline).

You deactivate and delete the A/B test, and then delete pipeline B (see Manage A/B Tests and Delete a Query Pipeline).

Create an A/B Test

  1. Prepare the following A/B test requirements, if not already done:

    1. Duplicate the pipeline that could be improved (see Manage Query Pipelines).

    2. In the navigation bar on the left, under Search, select Query Pipelines.

    3. On the Query Pipelines page, click the duplicate pipeline name, and then in the Action bar, click Edit Components.

    4. Based on the problem you try to solve, add, remove, or modify query pipeline rules (see Manage Query Pipeline Rules and Rule Conditions From Tabs).

    Don’t select or recreate the same condition as the original pipeline while configuring the duplicate pipeline. Doing so will prevent you from creating an A/B test since a specific condition can only be assigned to one pipeline.

  2. On the A/B Tests page, click Add Test.

  3. In the Add an A/B Test panel that appears:

    1. In the first box, enter a name for the A/B test.

    2. In the Pipeline A and Pipeline B drop-down menus, select two different pipelines.

      • If you test two versions of the same pipeline, select the original as Pipeline A and the duplicate (containing your changed rules) as Pipeline B.

      • Since you can’t perform chain tests (e.g., pipeline x paired with pipeline y and pipeline y paired with pipeline z), the lists only contain available pipelines.

    3. (Optional) Move the ratio slider left or right to set the proportion (%) of traffic sent to each pipeline.

      • By default, each pipeline is used 50% of the time.

      • We recommend giving less traffic to Pipeline B (e.g., 10%) when you have big traffic on your web page to only impact a small percentage of your users if changes to Pipeline A don’t achieve the desired outcome. On the other hand, when you have little traffic, you should increase the traffic to Pipeline B (e.g., 90%), to evaluate more quickly the effectiveness of your changes.

      • Have in mind that a small percentage of users doesn’t accept cookies. They might have disabled cookies in their browser settings or they might use an extension that blocks cookies. Therefore, they could switch from both Pipeline A and B during a single session since the Coveo Search API uses cookies to direct users into pipelines. On average these users shouldn’t impact aggregated metrics, but if you look closely into details, you could see abnormal values (see About A/B Tests).

    4. Click Add Test.

      Tests are inactive by default. Activate them to launch the testing process.

Manage A/B Tests

Members with the privilege to view query pipelines (i.e., the View all or the Custom access level on the Query Pipelines domain) can click View pipeline A or View pipeline B to review the pipeline components associated with the A/B tests in read-only mode (see Manage Privileges and Query Pipelines Domain).

Deactivate/Activate an A/B Test

  • More than one A/B test can be active at a time.

  • (When you use a proxy) Before activating an A/B test, ensure that the cookies used by Coveo are forwarded. If not, you need to add those cookies to the list of forwarded cookies.

  1. On the A/B Tests page, click the A/B test you want to activate/deactivate.

  2. In the Action bar, click More, and then select Activate test or Deactivate test.

Access a Pipeline Page to Modify Rules and Conditions

  1. On the A/B Tests page, click the A/B test containing the pipeline that you want to modify.

  2. In the Action bar, click Pipeline A or Pipeline B to access the [Pipeline Name] page.

  3. On the [Pipeline Name] page, manage pipeline components rules and conditions (see Manage Query Pipeline Rules and Rule Conditions From Tabs).

Delete an A/B Test

  1. Click the A/B test you want to delete.

  2. In the Action bar, click More, and then select Delete.

  3. Click Delete to confirm.

Required Privileges

The following table indicates the required privileges to view and edit elements of the A/B Tests page and associated panels (see Manage Privileges and Privilege Reference).

Action Service - Domain Required access level
View A/B tests

Search - Query pipelines

Edit A/B tests Search - Query pipelines Edit

What’s Next?

Test the efficiency of your A/B test (see Analyze the Performance of Pipeline A Vs Pipeline B).

Recommended Articles