Relevance Index metric

This is for:

Business Analyst System Administrator

The Relevance Index metric evaluates how well search results satisfy user queries. It combines several behavioral signals, including the clickthrough rate, the Average Click Rank, and how frequently a query is performed.

These signals are combined to produce a score between 0 and 1, where higher values indicate stronger search performance. In general, higher clickthrough rates and lower average click ranks contribute to better scores.

Lower Relevance Index values mean users aren’t finding what they’re looking for. Sorting queries or items by ascending Relevance Index is our recommended approach to focus on high-impact queries or items with relevance issues.

Note

The Relevance Index is intended as a relative metric used to compare queries and identify potential relevance issues, rather than an absolute measure of search performance. The formula and parameters may evolve over time as the metric continues to be refined.

How the Relevance Index is calculated

The Relevance Index calculation relies on three behavioral signals:

  • Clickthrough rate

  • Average click rank

  • Number of queries

These signals are combined into two components:

  • Relevance, which reflects user engagement signals.

  • Confidence, which reflects how frequently a query is performed.

The calculation can be summarized as follows:

Score = Relevance x Confidence
Relevance Index = (Score + 1) / 2

The normalization step converts the score from a range of -1 to 1 into a 0 to 1 scale used in reports.

Relevance is computed using a weighted combination of the clickthrough rate and click rank. Confidence increases with the number of queries, meaning that queries with higher frequency produce more reliable scores.

Note

In Coveo Analytics reports, the Relevance Index and Relevance Index (Legacy) values are color-coded to show whether they’re good, bad, or neutral.

The predefined thresholds for each metric are the following:

  • Relevance Index: < 0.3 = bad (red), >= 0.3 and <= 0.7 = neutral (black), and > 0.7 = good (green)

  • Relevance Index (Legacy): < 0.5 = bad (red), from 0.5 to 0.7 = neutral (black), and > 0.7 = good (green)

Concept of confidence

The Relevance Index uses the concept of confidence. For the same proportion of Search Event Clickthrough and the same Average Click Rank, the current Relevance Index value will increase proportionally to the number of times a user query is performed. In the same scenario, the Relevance Index (Legacy) value would have decreased.

Confidence amplifies the result. Queries performing well receive scores increasingly above 0.5 as their frequency grows, while poorly performing queries receive scores increasingly below 0.5.

Example

For the same Search Event Clickthrough (that is, 40 %) and an Average Click Rank of 3, the table below shows how a positive change in the Search Event Count metric impacts the Relevance Index and Relevance Index (Legacy) values.

Search Event Count Clickthrough (%) Average Click Rank Relevance Index Relevance Index (Legacy)

5

40

3

0.63

0.70

10

40

3

0.68

0.65

50

40

3

0.80

0.56

100

40

3

0.84

0.52

500

40

3

0.87

0.46

Average Click Rank value impact

An Average Click Rank value change has less impact for the same number of queries and clickthrough rate in the Relevance Index formula. The value changes gradually.

Example

For the same Search Event Count (that is, 500) and a clickthrough of 40%, the table below shows how a change in the Average Click Rank metric impacts the Relevance Index and Relevance Index (Legacy) values.

Search Event Count Clickthrough (%) Average Click Rank Relevance Index Relevance Index (Legacy)

500

40

1

0.87

0.82

500

40

4

0.85

0.42

500

40

8

0.78

0.35

500

40

12

0.64

0.32

500

40

16

0.56

0.30

Score of queries with a Clickthrough of 0%

The Relevance Index attributes a low score to queries with a clickthrough of 0 %.

Example

For the same Search Event Count (that is, 500) and an Average Click Rank of 3, the table below shows how a change in the Search Event Clickthrough metric impacts the Relevance Index and Relevance Index (Legacy) values.

Search Event Count Clickthrough (%) Average Click Rank Relevance Index Relevance Index (Legacy)

500

0

0

0

0.72

500

1

3

0.08

0.53

500

10

3

0.41

0.43

500

25

3

0.7

0.44

500

50

3

0.91

0.48