How Is the Relevance Index Calculated and How Should It Be Used?

The exact formula is part of our secret sauce to determine the relevance of queries and items. This index combines many indicators measuring relevance such as the Click-Through ratio (low is bad), the Average Click Rank (high is bad) and how frequently the item is queried or the query submitted (high frequency has more impact on relevance). Lower Relevance Index values mean users are not finding what they are looking for. Sorting queries or items by ascending Relevance Index is our recommended approach to focus on high-impact queries or items with relevance issues (see Identifying Relevance Issues).

The Relevance Index and Relevance Index (Legacy) values are colored to show if they are good, bad, or neutral.

The predefined thresholds for each metric are the following:

  • Relevance Index: < 0.3 = bad (red), >= 0.3 and <= 0.7 = neutral (black), and > 0.7 = good (green)

  • Relevance Index (Legacy): < 0.5 = bad (red), 0.5-0.7 = neutral (black), and > 0.7 = good (green)

Main Differences Between Relevance Index and Relevance Index (Legacy)

  • The new Relevance Index introduces the concept of confidence. For the same proportion of Click-Through and the same Average Click Rank, the new Relevance Index value will increase proportionally to the number of times a query is performed. In the same scenario, the Relevance Index (Legacy) value would have decrease.

    The confidence goes both way: good queries get a score higher than 0.5 that increases with the frequency of the query, and bad queries receive a score lower than 0.5 that decreases with the frequency of the query. The higher the frequency, the farther the Relevance Index is from the 0.5 mark.

    For the same Click-Through (i.e., 40 %) and an Average Click Rank of 3, the table below shows how a positive change in the Query Count metric impacts the Relevance Index and Relevance Index (Legacy) values.

    UA-RelevanceIndexMetricsEx3

    Query Count Click-Through (%) Average Click Rank Relevance Index Relevance Index (Legacy)
    5 40 3 0.63 0.70
    10 40 3 0.68 0.65
    50 40 3 0.80 0.56
    100 40 3 0.84 0.52
    500 40 3 0.87 0.46
  • An Average Click Rank value change has less impact for the same number of queries and Click-Through ratio in the new Relevance Index formula. The value changes gradually.

    For the same Query Count (i.e., 500) and a Click-Through of 40%, the table below shows how a change in the Average Click Rank metric impacts the Relevance Index and Relevance Index (Legacy) values.

    UA-RelevanceIndexMetricsEx2

    Query Count Click-Through (%) Average Click Rank Relevance Index Relevance Index (Legacy)
    500 40 1 0.87 0.82
    500 40 4 0.85 0.42
    500 40 8 0.78 0.35
    500 40 12 0.64 0.32
    500 40 16 0.56 0.30
  • The new Relevance Index attributes a low score to queries with a Click-Through of 0 %.

    For the same Query Count (i.e., 500) and an Average Click Rank of 3, the table below shows how a change in the Click-Through metric impacts the Relevance Index and Relevance Index (Legacy) values.

    UA-RelevanceIndexMetricsEx1

    Query Count Click-Through (%) Average Click Rank Relevance Index Relevance Index (Legacy)
    500 0 0 0 0.72
    500 1 3 0.08 0.53
    500 10 3 0.41 0.43
    500 25 3 0.7 0.44
    500 50 3 0.91 0.48
People also viewed