Automatic Relevance Tuning (ART) model card

This is for:

Developer System Administrator

What’s a model card?

A model card is a document that provides a summary of key information about a Coveo Machine Learning (Coveo ML) model. It details the model’s purpose, intended use, performance, and limitations.

Model details

The Coveo Machine Learning (Coveo ML) Automatic Relevance Tuning (ART) model dynamically adjusts the ranking of search results based on end-user interactions and query behavior. The ART model continuously learns from search analytics to optimize results relevance without requiring manual tuning. ART leverages behavioral signals, including queries and clicks, to elevate search results that have previously demonstrated strong user engagement and satisfaction for similar queries.

  • Development team: Coveo ML team

  • Initial release date: 2015. Major changes can occur and are communicated via Coveo release notes.

  • Activation: The ART model is created and assigned to query pipelines using the Coveo Administration Console.

Intended use

  • Intended purpose: To enhance the end-user’s search experience by automatically boosting search results based on historical engagement patterns for similar queries.

  • Intended input: The end user’s search query and context.

  • Intended output: A ranked list of top items (typically between 5 to 50) with corresponding ranking score boosts for the current query. These boosts are passed to the index and applied in the final ranking function.

  • Intended users: End users of Coveo customers.

Factors

The ART model enhances search relevance by leveraging aggregated analytics data collected on behalf of each customer. It considers search queries and clicked results, as well as contextual factors in which these interactions occurred. The model is trained to optimize the search result ranking for each query. A distinct model is trained for each supported language. By default, ART also records the search interface in which the query is performed, which allows for query-time filtering based on the search interface.

Training data

The data used to train each customer’s ART model is derived from that customer’s end-user interactions with Coveo’s platform.

It includes the following:

  • Events received via API calls, such as query information, client ID, user ID, and clicks.

  • Data automatically collected by Coveo’s platform, such as the user agent, IP address, or any information derived from the IP address like country or city.

  • Optional custom context information provided by customers to enrich the search context, such as the specific page, specific search hub, or specific user segment.

  • In Coveo for Commerce implementations, ART can also learn from commerce-specific user actions such as cart and purchase events.

All data collection and processing are performed on behalf of each customer, in compliance with Coveo’s privacy and security commitments.

Performance

The quality and performance of the ART model can be assessed by looking at the search performance metrics like clickthrough rate and average click rank, which typically improve as ART learns from historical interactions. Customers can also fine-tune the ART model’s performance by adjusting the training dataset, and retraining frequency. You can use the Relevance Inspector to identify and validate search results promoted by ART.

Limitations

Certain conditions may reduce the effectiveness of the ART model:

  • Data volume: The effectiveness of the ART model relies on the volume of historical analytics data. As a result, new queries or low-traffic queries may yield less accurate results. A query needs to have occurred multiple times for ART to learn which search results should be boosted.

  • Feedback type: The ART model doesn’t directly consider explicit end-user feedback, such as thumbs up and thumbs down.

Best practices

Best practices for the ART model are documented here.

All usage must comply with Coveo’s Acceptable Use Policy.