A/B Tests Leading Practices

When managing A/B tests, consider the following recommendations and tips:

  • All A/B tests should be done with a purpose starting with a hypothesis to resolve. You can start by identifying a problem of your organization search usage, and then create one or more pipeline rules that can possibly solve it (see Leveraging the Coveo Cloud Usage Analytics).

  • Your A/B tests should only contain subtle changes between pipeline A and B so that you always know the reason of a positive outcome. It is thus recommended to start from an existing pipeline, and then modify or add only a couple of pipeline rules before conducting your tests (see Duplicate a Query Pipeline).

  • Set a goal to achieve regarding the relevance metrics you want to improve.

    Improve the Query Click-Through (%) ratio of the 10 queries with lowest Relevance Index by 10% (see Usage Analytics Metrics).

  • Both pipelines are always tested simultaneously to ensure the validity of your tests since many factors can differ between two periods of time, such as the number of visits, clicks and queries.

  • Your test results must be significant for you to conclude that a pipeline is better than another.

  • Often the results of A/B tests can be surprising or against logic. You should thus not reject an A/B test result based on your arbitrary judgment.

  • The difference between the original and the modified pipeline as well as the sample size are two key factors to look into before drawing any conclusion regarding an A/B test.

    You can test whether your results are statistically significant on websites such as Survey Star.

    You want to test your results regarding the click-through ratio with a couple of changes you made in your default pipeline. You get a click-through ratio of .30 with Pipeline A and 0.35 with Pipeline B. The base size of your two samples is 1 000 queries. With these numbers, your result is statistically significant 19 times out of 20 (95% confidence level), so you make Pipeline B effective on your search page.

  • Make your A/B test consistent across your whole search page.

    When a user searches for char and you enter a thesaurus rule that replaces char with characters, this behavior should be the same across every search box of your search interface.

  • It is recommended to perform many A/B tests so that you can take positive results and add them together to boost whatever outcome you expected in the first place.

  • (When you use a proxy) Before activating an A/B test, ensure that the cookies used by Coveo are forwarded. If not, you need to add those cookies to the list of forwarded cookies.