A/B Tests Leading Practices

In Coveo organizations that have gone through the query pipeline migration process, you can’t configure A/B tests using the A/B Tests page of the Administration Console.

You must use the “A/B Test” tab of a pipeline configuration instead.

When managing A/B tests, consider the following recommendations and tips:

  • All A/B tests should be done with a purpose starting with a hypothesis to resolve. You can start by identifying a problem of your organization search usage, and then create one or more pipeline rules that can solve it (see Leverage Coveo Usage Analytics).

  • Your A/B tests should only contain subtle changes between pipeline A and B so that you always know the reason of a positive outcome. Therefore, it’s recommended to start from an existing pipeline, and then modify or add only a couple of pipeline rules before conducting your tests (see Duplicate a Query Pipeline).

  • Set a goal to achieve regarding the relevance metrics you want to improve.

    For example, improving the Search Event Click-Through (%) ratio of the 10 queries with lowest Relevance Index by 10% (see Usage Analytics Metrics).

  • Both pipelines are always tested simultaneously to ensure that the validity of your tests since many factors can differ between two periods of time, such as the number of visits, clicks and queries.

  • Your test results must be significant for you to conclude that a pipeline is better than another.

  • Often the results of A/B tests can be surprising or against logic. Therefore, you shouldn’t reject an A/B test result based on your arbitrary judgment.

  • The difference between the original and the modified pipeline as well as the sample size are two key factors to look into before drawing any conclusion regarding an A/B test.

    You can test whether your results are statistically significant on websites such as Survey Star.

    For example, you want to test your results regarding the click-through ratio with a couple of changes you made in your default pipeline. You get a click-through ratio of .30 with Pipeline A and 0.35 with Pipeline B. The base size of your two samples is 1,000 queries. With these numbers, your result is statistically significant 19 times out of 20 (95% confidence level), so you make Pipeline B effective on your search page.

  • Make your A/B test consistent across your whole search page.

    For example, when a user searches for char and you enter a thesaurus rule that replaces char with characters, this behavior should be the same across every search box of your search interface.

  • We recommend that you perform many A/B tests so that you can take positive results and add them together to boost whatever outcome you expected in the first place.

  • (When you use a proxy) Before activating an A/B test, ensure that the cookies used by Coveo are forwarded. If not, you need to add those cookies to the list of forwarded cookies.

Recommended Articles