Adding and Managing A/B Tests

You can use the A/B Tests page in the Coveo Cloud administration console navigation menu to efficiently evaluate the impact of query pipeline rule changes by pairing query pipelines in one or more A/B tests.

During a test, you subject two versions of a pipeline to experimentation simultaneously by splitting the web traffic between both versions. In the end, you can measure which version has the best results based on metrics that are significant to your Coveo Cloud organization, and then set the winning pipeline as the effective one for their search page.

By analyzing your relevance metrics performance, you realize that you need to improve your Average Click Rank to reach your relevance objectives.

You then decide to modify your query pipeline by adding a new rule, hoping this modification will improve your Average Click Rank metric.

You then duplicate the pipeline and add the rule to the duplicated version (see Leading Practices).

In order to test the efficiency of that new rule, you create an A/B test in which you compare both versions of the query pipeline, i.e., the original production pipeline (A) and the pipeline with the new rule to test (B).

After reviewing the results of the A/B test, you notice that the new rule has indeed improved your Average Click Rank metric (see Analyzing the Performance of Pipeline A Vs Pipeline B).

You therefore copy the new rule from the test pipeline (B) to the production pipeline (A) (see Copy a Rule to Another Pipeline).

You deactivate and delete the A/B test, and then delete pipeline B (see Manage A/B Tests and Delete a Query Pipeline).

Access the “A/B Tests” Page

  1. If not already done, log in to the Coveo Cloud platform as a member of a group with the required privileges to manage A/B tests in the target Coveo Cloud organization.

  2. In the main menu on the left, under Search, select A/B Tests.

Create an A/B Test

  1. If not already done, prepare the following A/B test requirements:

    1. Duplicate the pipeline that could be improved (see Adding and Managing Query Pipelines).

    2. In the navigation bar on the left, under Search, select Query Pipelines.

    3. On the Query Pipelines page, double-click the [Pipeline Duplicate Name].

    4. Based on the problem you try to solve, add, remove, or modify query pipeline rules (see Adding and Managing Query Pipeline Rules and Rule Conditions From Tabs).

    Do not select or recreate the same condition as the original pipeline while configuring the duplicate pipeline. Doing so will prevent you from creating an A/B test since a specific condition can only be assigned to one pipeline.

  2. On the A/B Tests page, click Add Test.

    If the Add Test button is grayed and unresponsive, you do not have all required privileges to perform this action.

  3. In the Add an A/B Test dialog that appears:

    1. In the first box, enter a name for the A/B test.

    2. In the Pipeline A and Pipeline B drop-down menus, select two different pipelines.

      • If you test two versions of the same pipeline, select the original as Pipeline A and the duplicate (containing your changed rules) as Pipeline B.

      • Since you cannot perform chain tests (e.g., pipeline x paired with pipeline y and pipeline y paired with pipeline z), the lists only contain available pipelines.

    3. (Optional) Move the ratio slider left or right to set the proportion (%) of traffic sent to each pipeline.

      • By default, each pipeline is used 50% of the time.

      • It is recommended to give less traffic to Pipeline B (e.g., 10%) when you have big traffic on your web page to only impact a small percentage of your users if changes to Pipeline A do not achieve the desired outcome. On the other hand, when you have little traffic, you should increase the traffic to Pipeline B (e.g., 90%), to evaluate more quickly the effectiveness of your changes.

      • Have in mind that a small percentage of users does not accept cookies. They might have disabled cookies in their browser settings or they might use an extension that blocks cookies. Therefore, they could switch from both Pipeline A and B during a single session since the Coveo Search API uses cookies to direct users into pipelines. On average these users should not impact aggregated metrics, but if you look closely into details, you could see abnormal values (see Understanding A/B Tests).

    4. Click Add Test.

      Tests are inactive by default. Activate them to launch the testing process (see Manage A/B Tests).

Search A/B Tests

On the A/B Tests page, in the right section of the Action bar, type keywords in the Filter box. You can search tests by name, status, and tested pipeline name.

Manage A/B Tests

Members with the privilege to view query pipelines (i.e., the View all or the Custom access level on the Query Pipelines domain) can click View pipeline A or View pipeline B to review the pipeline components associated with the A/B tests in read-only mode (see Privilege Management and Query Pipelines Domain).

  1. On the A/B Tests page:

    • Deactivate/Activate an A/B test.

      • More than one A/B test can be active at a time.

      • (When you use a proxy) Before activating an A/B test, ensure that the cookies used by Coveo are forwarded. If not, you need to add those cookies to the list of forwarded cookies.

      1. Click the A/B test you want to activate/deactivate.

      2. In the Action bar, click More, and then select Activate test or Deactivate test.

    • Access a pipeline page to modify rules and conditions.

      1. Click the A/B test containing the pipeline that you want to modify.

      2. In the Action bar, click Pipeline A or Pipeline B to access the [Pipeline Name] page.

      3. On the [Pipeline Name] page, manage pipeline components rules and conditions (see Adding and Managing Query Pipeline Rules and Rule Conditions From Tabs).

    • Delete an A/B test.

      1. Click the A/B test you want to delete.

      2. In the Action bar, click More, and then select Delete.

      3. Next to Are you sure?, click Delete.

Required Privileges

The following table indicates the required privileges to view and edit elements of the A/B Tests page and associated panels (see Privilege Management and Privilege Reference).

Action Service - Privilege Required access level
View A/B tests

Organization - Organization

Search - Query pipelines

View
Edit A/B tests

Organization - Organization

View
Search - Query pipelines Edit

What’s Next?

Test the efficiency of your A/B test (see Analyzing the Performance of Pipeline A Vs Pipeline B).