Leveraging Custom Contexts in Coveo Machine Learning Features

Coveo Machine Learning (Coveo ML) models learn from user behavior to automatically improve Coveo-powered content relevance (see Coveo Machine Learning Features).

You can take the relevance of the Coveo ML feature one step further by defining and using custom contexts that are appropriate to your business.

Coveo ML models favor (or boost the ranking score of) content that matches or is accessed more often in the specified custom context. Because Coveo ML models do not filter content based on this custom context, they can still provide output when there are few or no context matches.

Consider specifying a custom context for Coveo ML models when most queries should return different results depending on the context.

Users are authenticated on your product documentation site. From their profiles, you can get their roles relative to your products (i.e., end-user, administrator, or developer). This means that user role could be a good custom context.

Often, for similar queries (e.g., entering only a feature name), users with different roles have different intentions and are looking for different kinds of information:

  • Developers - how to customize the product with code samples.

  • Administrators - look for installation and configuration articles.

  • End users - instructions on how to use the product.

With the user role as a custom context, Coveo ML features can promote content that is more relevant to each user’s context.

You can also use custom contexts in query pipeline conditions, and then apply these conditions to query pipeline elements (see Sending Custom Context Information).

How Contexts and Coveo Machine Learning Models Work

In practice, custom contexts are key/value pairs that must be sent as custom data with standard or custom Coveo Usage Analytics (Coveo UA) events to build a contextualized activity history.

When a user makes a query and a context is specified, the matching relevant results with the highest activity history in this context are promoted. More technically, Coveo ML models use context like an intelligent query ranking expression (QRE), boosting results that match the query and the context. Coveo ML models automatically take into account custom usage analytics metadata keys that start with the context_ prefix.

Other relevant results which are not seen in a specified context are not promoted by Coveo ML models, but their overall ranking score may still be higher than the contextually promoted ones when they are popular.

A query can carry more than one context key/value pair, in which case the ranking score boosts associated with each key/value context pair are combined.

Custom Context Leveraging Leading Practices

Consider the following criteria for a good custom context:

  • For the same queries, successful results are significantly different depending on the context key/value pairs.

    The context can be the user role. The authenticated users on your software product documentation site play one of the following role relative to your product:

    • End user - Actually use the product.

    • Administrator - Install, configure, and support the product.

    • Developer - Write code to customize or interact with your product.

    Because users are authenticated, at query time you have a method to get the user role

  • You can clearly identify and send the appropriate static or dynamic key/value pair with each query (see Sending Custom Context Information).

  • While contexts can be combined, use only the few most significant context keys, typically less than 5 gives better results.

    • On your client community, you can send the following contexts since they are different for each client:

      • User role

      • User department

      • Product categories they already own

      • User country (useful when items are different from one country to the other)

    • You have many employees using your agent panel. Therefore, you want to pass information about the role, department, and level of your agents to categorize or group them.

      When logging usage analytics events, you thus ensure that your developer include, in the customData property of the request body, key-value pairs such as:

      • "context_role":"<context_role>"

      • "context_department":"<context_department">

      • "<context_level>":"<context_level">

      where:

      • <context_role> is the role of the agent sending the request.

      • <context_department> is the department of the agent sending the request.

      • <context_level> is the level of the agent sending the request.

    Increasing the number of context combinations dilutes the context-specific effect. Reduce the number of key/value pairs even more when the level of search activity is low.

    Your Coveo-powered customer portal search page serves about 500,000 search events per day, which provides plenty of activity from which Coveo ML models can learn for a few contexts, each with a few values.

    By contrast, your Coveo-powered employee portal search page serves less than 1,000 search events per day (this is normal, as only a few hundred employees can access it). With much less data to learn from, Coveo ML models may only be effective for one significant context key, with only a few values.

  • Each context key should have a limited number of possible values, less than 10 being recommended.

    Your product popularity vary notably from one country to the other, so you guess using the search page user country of origin as a context will help Coveo ML models return more relevant content for customers of each country.

    The effect of the context will probably be notable if your customers are from only a few countries, but may be diluted if they come from 100 countries.

  • DO NOT use the following aspects as custom context keys, because they are passed by the Coveo JavaScript search interfaces with each query and and are taken into account by out-of-the-box Coveo ML models:

    • Language - The language of the search interface.

    • Search Hub - The name you give to a specific search instance.

    • Interface (Tab) - The different search page tabs in your search hub.

    • Is Guest

    • Is Anonymous

    Coveo powers the search pages in your English-only employee portal as well as in your multi-language customer portal.

    In the Spanish customer portal, the output of the Coveo ML models should only take into account what other customers have previously searched for in this portal, not searches made in the employee portal.

Leveraging Custom Context Steps

  1. Define the context key/value pairs that you plan to leverage.

    It is important to pick key/value custom context pairs for which you can get appropriate values in your code and provide good search result differentiation (see Custom Context Leveraging Leading Practices).

  2. Ensure that appropriate context key/value pairs that you defined are passed by your search interface with each query:

    • Standard search usage analytics events (for Coveo ML Automatic Relevance Tuning and Dynamic Navigation Experience)

      Coveo JavaScript Search Framework 1.1276 (September 2016) In Coveo JavaScript Search Framework search interfaces, you must add code to specify the custom context in which each search is performed (see Sending Custom Context Information).

    • View usage analytics events (for Recommendation components)

      For view usage analytics events which are required by Event Recommendation (ER) models, you must pass the custom context key/value pairs along with usage analytics parameters, and ensure that your custom context key names start with the context_ prefix (see Sending Usage Analytics View Events).

  3. Create or edit Coveo ML models

    Existing or new Coveo ML models automatically take advantage of custom context values (passed as described in the previous steps) when the are trained or retrained, so there is no custom-context-specific configuration to do on Coveo ML models.

    If not already done, create Coveo ML models (see Adding and Managing Coveo Machine Learning Models).

  4. Attempt to measure the impact of custom contexts

    There are no direct methods to measure the impact of the custom contexts on the output of Coveo ML models. However, you can create analytics reports to look at certain quality metrics (e.g., Click-Through or Click Rank) as a function of context dimensions.

    1. Create a dimension for each of your custom context keys (see Adding and Managing Dimensions on Custom Metadata).

      When some custom context key/value pairs have been passed by your search interface, in the Add a Dimension panel, the API name parameter automatically proposes custom context keys (the key is prefixed with context_).

      When your custom context key is userRole, the proposed API name value is context_userRole.

    2. Validate that your Coveo ML model has been trained since you started to send custom context.

    3. Create a report with the metrics of your choice filtered by your context dimensions.