Relevance Index metric

This is for:

Business Analyst System Administrator

The exact formula is part of Coveo’s secret sauce to determine the relevance of queries and items. This index combines many indicators measuring relevance such as the clickthrough ratio (low is bad), the Average Click Rank (high is bad), and how frequently the item is queried or the query submitted (high frequency has more impact on relevance).

Lower Relevance Index values mean users aren’t finding what they’re looking for. Sorting queries or items by ascending Relevance Index is our recommended approach to focus on high-impact queries or items with relevance issues.

The Relevance Index and Relevance Index (Legacy) values are colored to show if they’re good, bad, or neutral.

The predefined thresholds for each metric are the following:

  • Relevance Index: < 0.3 = bad (red), >= 0.3 and <= 0.7 = neutral (black), and > 0.7 = good (green)

  • Relevance Index (Legacy): < 0.5 = bad (red), from 0.5 to 0.7 = neutral (black), and > 0.7 = good (green)

Main differences between Relevance Index and Relevance Index (Legacy)

Concept of confidence

The new Relevance Index introduces the concept of confidence. For the same proportion of Search Event Clickthrough and the same Average Click Rank, the new Relevance Index value will increase proportionally to the number of times a user query is performed. In the same scenario, the Relevance Index (Legacy) value would have decreased.

The confidence goes both ways: good queries get a score higher than 0.5 that increases with the frequency of the query, and bad queries receive a score lower than 0.5 that decreases with the frequency of the query. The higher the frequency, the farther the Relevance Index is from the 0.5 mark.

For the same Search Event Clickthrough (that is, 40 %) and an Average Click Rank of 3, the table below shows how a positive change in the Search Event Count metric impacts the Relevance Index and Relevance Index (Legacy) values.

UA-RelevanceIndexMetricsEx3

Search Event Count Clickthrough (%) Average Click Rank Relevance Index Relevance Index (Legacy)
5 40 3 0.63 0.70
10 40 3 0.68 0.65
50 40 3 0.80 0.56
100 40 3 0.84 0.52
500 40 3 0.87 0.46

Average Click Rank value impact

An Average Click Rank value change has less impact for the same number of queries and clickthrough ratio in the new Relevance Index formula. The value changes gradually.

For the same Search Event Count (that is, 500) and a clickthrough of 40%, the table below shows how a change in the Average Click Rank metric impacts the Relevance Index and Relevance Index (Legacy) values.

UA-RelevanceIndexMetricsEx2

Search Event Count Clickthrough (%) Average Click Rank Relevance Index Relevance Index (Legacy)
500 40 1 0.87 0.82
500 40 4 0.85 0.42
500 40 8 0.78 0.35
500 40 12 0.64 0.32
500 40 16 0.56 0.30

Score of queries with a Clickthrough of 0%

The new Relevance Index attributes a low score to queries with a clickthrough of 0 %.

For the same Search Event Count (that is, 500) and an Average Click Rank of 3, the table below shows how a change in the Search Event Clickthrough metric impacts the Relevance Index and Relevance Index (Legacy) values.

UA-RelevanceIndexMetricsEx1

Search Event Count Clickthrough (%) Average Click Rank Relevance Index Relevance Index (Legacy)
500 0 0 0 0.72
500 1 3 0.08 0.53
500 10 3 0.41 0.43
500 25 3 0.7 0.44
500 50 3 0.91 0.48