Adding Machine Learning Automatic Relevance Tuning Models

Coveo™ Machine Learning (Coveo ML) is a service that leverages usage analytics data (see Coveo Machine Learning). Automatic relevance tuning (ART), is a Coveo ML feature that optimizes search results relevance based on user search behavior (see Automatic Relevance Tuning (ART) Feature).

In short, ART looks at end-user query and search result click behavior made over a given period to learn what the best search results for each query are. When a learned query is performed again, ART recommends the best learned search results by boosting their ranking score so they appear among the top 10 results.

  • ART recommends the 5 best learned search results, but may recommend less than 5 when:

    • A query is not very frequent and less than 5 items were learned for that query.

    • Some of the recommendations are secured and not accessible to the user performing the query.

    • Some of the recommendations are filtered out because they do not match filters such as the current facet selections.

  • On an empty query, ART returns the most clicked documents during the data period of the model (see Data Period).

ART can inject items that would not normally be included in search results because they were learned to be relevant even if they do not contain some or all of the searched keywords. This is one of the key ART benefits as a user can find the most useful items without having to type the right keywords or the specific synonym contained these items. This is the default behavior (Match the query parameter is false).

ART currently ignores all special characters or operators in the user entered query to only keep the keywords and therefore ignores the special behavior described in Using Special Characters in Queries or Search Prefixes and Operators.

This ART behavior may lead to unexpected search results when users want to take advantage of more advanced query syntax.

A user is searching for items containing a specific phrase by entering the phrase enclosed in double-quotes (see Searching for a Phrase) and no items contain the phrase. The user would expect to get no search results, but ART can inject items that were clicked for similar queries made of one or some of the searched keywords (not in a phrase search).

You configure ART from the administration console for a query pipeline defined in your Coveo Organization by adding a machine learning model (see Coveo Machine Learning Models and Who Can Perform the Page Actions). The default values for available parameters are typically appropriate in most cases, but the following procedure indicates why or when you may want to change them.

Add an Automatic Relevance Tuning (ART) Model

  1. If not already done, log in to your Coveo Organization with an Admin or Relevance Analyst user role account.

  2. Ensure that your Coveo Organization collects usage analytics data for the search hub on which you want to activate ART (see Coveo Cloud Usage Analytics).

    ART predictive models are based on Coveo Cloud usage analytics data. If no usage analytics data is available, there will be no recommendations, but recommendations will improve as more data becomes available each time the model is retrained (see Training and Retraining).

  3. Ensure that your Coveo Cloud organization has collected enough data before creating an ART model (see Evaluating If Your Search Interface Produces Enough Data for ART).

    OR

    When your search interface has more than 55 visits per day in which a manual query is followed by a click for a specific language, wait 24 hours, and then create a model with specific settings (detailed later in this topic) until enough data is collected.

  4. In the navigation bar on the left, under Search Optimization, select Query Pipelines.

  5. In the Query Pipelines page:

    Duplicate your production pipeline, activate ART on this secondary test pipeline, and test ART before putting the test pipeline in production (see Testing Query Pipeline Changes).

    1. Click the Actions icon (Icon-SourceActions) at the end of the row of the pipeline for which you want to manage automatic relevance tuning (typically the one used by the search interface for which you want to activate ART), and then select Duplicate.

      A Copy of [PipelineName] pipeline is created.

    2. In the Name column, click the query pipeline copy.

  6. In the Machine Learning tab:

    If the Machine Learning tab is not visible, contact Coveo Support to request to enable Coveo ML in your Coveo Cloud organization.

    • Create an ART model by clicking Add Model.

      Because an ART model automatically includes sub-models for hubs, interfaces, and languages, you typically need only one ART model in your query pipeline (see Sub-Models).

      However, you need to create separate models only when you have cases where any of the model parameters described below must have a different value in each case.

      Your Community search page serves thousands of queries every day. You create a first ART model based on the last 3 months of usage analytics data that is retrained weekly to include ART recommendations in the search results. You also want to add in your search page an ART-powered Trending items list taking recommendations from a second model that is rather based the last week of data and retrained daily.

      OR

    • (For developers only) Create an ART model in Query Pipeline Language (QPL):

      You can directly enter a configuration rule that respects the Query Pipeline Language (QPL) syntax for an Automatic Relevance Tuning model.

      1. Click Add Model with Code.

      2. In the form editor box that appears, enter an Automatic Relevance Tuning (topClicks) model configuration rule (see REST Search API Models).

      3. Click Save.

      OR

    • Modify an existing model by clicking the model that you want to change or, clicking the Actions icon (Icon-SourceActions) and then selecting Edit model.

      Currently, when you edit an existing ART model, the automatic relevance tuning service is interrupted for about one hour while the model is trained with the new configuration. The recommended workaround is rather to duplicate and customize the model, and make the new model effective when it is ready (see Editing a Coveo Machine Learning Model Without Service Interruption).

  7. In the Add a Machine Learning Model or Edit a Machine Learning Model panel, in the left section:

    1. (In the Add a Machine Learning Model panel only) Click the Select model type drop-down menu, and then select Tune relevance.

    2. In the User note input, enter useful information such as the creation or edition date and the purpose of the model.

      John Smith - 2016-09-17 - Improve the search result relevance of our online help

  8. In the Add a Machine Learning Model or Edit a Machine Learning Model panel, in the right section, evaluate if you need to change some of the default values:

    1. In the Training set section:

      1. In the Update frequency drop-down menu, select the time interval at which the model is retrained between the following values: Daily, Weekly, or Monthly. The default value is Weekly for the default 3 months Data period (see Training and Retraining).

      2. In the Data period drop-down menu, select the usage analytics data time interval on which the model will be based. Available data periods depend on the Update frequency you select (see Training and Retraining). The default value is 3 months, meaning the last 90 days from today.

        As a general guide, the Data period should include a minimum of 5,000 queries in 3 months. When you have different hubs, interfaces, and languages, a higher number of queries would be better so that each sub-model has access to a significant data set.

        You can evaluate the available volume of queries from usage analytics reports (see Usage Analytics Reports).

        Consider selecting a longer period when your search interface serves a small number of queries and want to improve the recommendations relevancy by using a larger data set.

        Consider selecting a shorter period when your search hub serves a large number of queries and you want the recommendations to be more responsive to user behavior trends.

      3. (Optional) When you want to train a model on a given search hub, and then use this model on another search hub, add a hub filter.

        Your Community search page has more traffic than your agent hub, so you want to use the Community model in your agent hub, so that the model influences results on both hubs.

        You can also add a hub filter when you want to use a model on all search hubs.

        To do so:

        1. Under Train on events, select the Specific radio button.

        2. Click Origin 1 (Page/Hub).

        3. Select the Is operator.

        4. In the Enter filter value input, select or type the name for your search hub.

        5. Click Add Filter.

      4. When you want to train the ART model with usage analytics data only from a specific geographic origin, consider creating Country and or Region filters.

        Your Community search page is used by customers from the USA and the UK, but your products have different names, features, and documentation in these two countries. You can create one model for each country and add a corresponding Country data filter.

    2. In the Impact on relevancy section:

      1. With the Ranking modifier slider, the Default value of 250 typically modifies the ranking score of the 5 ART recommendations so that they appear among the top 10 search results.

        Consider reducing the value when ART recommendations ranking scores significantly exceed that of the normal top results, such as when ART recommendations are always appearing as the top 5 results. You generally do not want ART recommendations to completely override the normal search results ranking, but rather bring them among the first top 10 results. The typical range of values is between 50 and 250.

        Consider increasing the value when the ART recommendations generally do not appear within the first search result page. Increase the value until the ART recommendations surface among the first 10 top results.

        ART recommendations can be buried lower in search results when other query pipeline rules such as Featured Results and Ranking Expressions, or hard-coded query ranking expressions significantly increase the top result ranking scores. When this is the case, reevaluate and consider eliminating such query pipeline rules or hard-coded expression when they are trying to do what ART can do automatically and probably much better.

      2. Under Inject search results, the Match the query option default value is false because it is typically desirable for ART to be allowed to inject search results that do not contain some or all of the keywords entered in the search box by the user.

        • Even if a query contains a typographical mistake, ART still recommends results.

        • The Match the query option allows Coveo ML to associate keywords used by user VS those used in the documentation.

          Your users are searching for broken clip, but your knowledge articles referred a clip as a latch. If the Match the query check box is cleared, Coveo ML will learn that clip and latch are the same object.

        • Depending on your use case, you could want to match the queries.

          In the People tab of your internal search interface where employees search for people names, you want results to match the name they entered in the search box.

        Many customers on a Community search page start seeking help on a popular product by entering the part number. However, many useful items available about this product do not include that part number. Consequently, the initial search results do not return relevant items, but customers keep adapting their keywords or navigating in search results until they find an appropriate item for the product.

        Because ART learns that this item was useful in relation with this part number, when the Match the query option is false, ART can now recommend it immediately, even if the part number is not present in the item.

        You could achieve the similar outcome using a query pipeline thesaurus rule expanding the part number with a product name that you know is present in most useful items about this product, but you would probably have to spend a significant amount of time to analyze usage analytics data to identify the issue and manually enter an appropriate thesaurus rule. ART does that continuously and automatically for countless user behavior issues.

        The scope broadening bonus coming with the injection of search results not matching the query may have a downside. Users may be confused in some cases (such as when someone searches for a specific keyword or an exact phrase), because the ART boosted results do not contain some or all of the keywords, these keywords are not highlighted in the excerpt, and are not navigable in the Quick View. The user may think that something is wrong with the search result ranking.

        Consider changing the Match the query option value to true only if the negative side of ART appears to be more important than the benefits or when instructed to do so by Coveo Support.

      3. Under Inject search results, the Match the advanced query option default value is true because it is typically desirable for ART recommendations to match search interface scope, facet selection, and other hidden or user selected filters.

        ART recommended items are typically not distinguishable from other results in search interfaces. If ART is allowed to inject results outside of the expected scope, users may be confused or even think that there are bugs in the search interface.

        A customer is looking for a specific knowledge article about a product called XYZ. He goes to the Knowledge interface of the Community search hub, enters appropriate keywords in the search box, and than selects the XYZ value in the Product facet. With the Match the advanced query option set to true, ART only recommends knowledge articles about the XYZ product. If the value was false, matching items of other types and about other products could be recommended.

        Consider changing the Match the advanced query option value to false only for rare cases when instructed to do so by Coveo Support.

      4. Under Inject search results, select the Enable Intelligent Term Detection (ITD) check box when you want long user queries (essentially support case descriptions) to be refined by ART before being sent to the index.

        When selected, ART processes queries by only keeping the most relevant words. By default, the selection is based not only on the average importance (weight) of each word but also on the longest substring from your search interface user queries that is part of a list of top queries. The list contains the top 2500 queries that have been submitted at least five times each in your search interface.

        This option is particularly useful when the Coveo ML model is used in case deflection interfaces that receive long case descriptions as queries.

    3. In the Condition to be applied section, when you want or need to activate ART recommendations conditionally for incoming queries, apply a condition to your model:

      When you train a model with the events of a given search hub (see Data filter) and want to use this model on another search hub, add a model condition so that the recommendations for the first model are also returned on the second hub.

      Search Hub is [secondHub]

      In the list of available conditions, select a condition.

      • At the bottom-right of the list, click the left and right arrow icons, or a page number to navigate through pages.

      • At the bottom-left of the list, select 10 or 100. By default, the table shows 10 conditions per page.

      OR

      Create a new condition by clicking Add Condition (see Create a Condition).

      OR

      Edit an existing condition:

      1. Click Manage Conditions.

        You lose all unsaved changes if you clicked Confirm in the Are You Sure confirmation prompt that appears. It is thus recommended to Save the model before creating a condition.

      2. In the Conditions page, edit a condition (see Modify a Condition).

      3. Access again the Condition to be applied section of a new model or the model you edited.

      4. In the list of available conditions, select the condition you edited.

    4. At the bottom of the Add a Machine Learning Model or Edit a Machine Learning Model panel, click Add or Save.

      The instructions to create or update the model are sent to the Coveo ML service.

  9. When you edit a model, in the Are You Sure? dialog box, click Confirm only when it is acceptable for you to get an ART service interruption, which will not be noticed by end-users as only the ranking of a few results may be affected.

    Otherwise, click Cancel and use the other method (see Editing a Coveo Machine Learning Model Without Service Interruption).

    CCV2AC_ARTAreYouSure2

  10. Back in the Machine Learning tab of your pipeline page, on the model row, in the Status column, the value is most probably In progress.

    The value changes to Ready when the model creation or modification is completed, typically within one hour, the delay increasing with the amount of usage analytics data to process. ART can return recommendations only when the model has a Ready status.

    If you just started collecting usage analytics data, the model may appear Ready, even if no or insufficient usage analytics data is available, in which case, no recommendations will be made until the model is retrained according to the Training Frequency value.

  11. Validate that ART recommendations appear in search results:

  12. Create an A/B test to review the effectiveness of your ART model (see Managing A/B Tests).

    Before distributing all your search interface traffic to the new pipeline, perform an A/B test for at least a week:

  13. When the A/B test is conclusive, distribute 100% of the traffic on the new pipeline by swapping your production (A) pipeline with your test (B) pipeline (see Managing Query Pipelines).

    You can also copy the changed rule from the B to the A pipeline (see Copy a Rule to Another Pipeline).

  14. Optionally, plan the usage of custom contexts to improve ART relevance:

    While ART without custom context does an amazing job, using custom contexts appropriate for your business can take ART relevance one step further.

    If you are new to Coveo ML ART, you may skip this step for now to more quickly and easily get the ART benefits, and then consider using custom contexts in a second phase.

    You can define custom contexts and then pass appropriate ones along with usage analytics events and queries to allow Coveo ML ART to take them into account (see Leveraging Custom Contexts in Coveo Machine Learning Features).

Who Can Perform the Page Actions

The required roles are:

  • Administrator

    OR

  • Relevance Analyst

What’s Next?

If the query pipeline in which you configure ART contains thesaurus rules, help train ART to learn those rules (see Helping Train Coveo Machine Learning With Thesaurus Rules).