Manage stop word rules

This is for:

Developer System Administrator

Stop words are words that are filtered out from a query before it’s sent to the index. By filtering out certain words, other keywords are given more importance, which helps in providing users with more relevant search results.

By default, Coveo search results contain all searched words, but the Coveo ranking algorithm gives less importance to frequently used words such as articles (for example, a, the), prepositions (for example, of, in), and pronouns (for example, my, them). Consequently, the presence of these words in an item doesn’t significantly increase the item ranking in search results.

Depending on your specific needs, you may want to add a list of words to ignore from queries. For example, industry-specific words that appear in many of your items, and offer little in the way of differentiation between items in search results.

The list of stop words for the index of a Coveo organization is empty by default, but members with the required privileges can add stop word rules, which are defined independently for each query pipeline.

Example

A user enters the following natural language query:

Expert doctor treatment for patient with headaches

In a clinical setting, you may want to create a list of stop words such as doctor, patient, and expert, as they’re frequently repeated across many documents and don’t add much value.

Adding these common words as stop words, and therefore filtering them out of the query, gives more importance to the more meaningful keywords of the query: (treatment and headaches).

When these common words are included as stop words, the query sent to the index is: treatment headaches

The search results for only these two keywords will most likely include more relevant items.

Prerequisites

Before creating a rule, first make sure that you have the following:

Once you meet these requirements, you can create a rule on the Query Pipelines (platform-ca | platform-eu | platform-au) page. To test the rule, use the A/B test feature to compare the results of the rule with the results of the original pipeline.

Order of execution

The following diagram illustrates the overall order of execution of query pipeline features:

diagram showing order of execution

Leading practices

When managing stop word rules, consider the following recommendations and tips:

  • Use stop words sparingly:

    • Avoid adding more than a dozen stop words. Too many stop words can negatively impact search results since most of them may still convey some meaning and provide syntactical information used by the search engine to better match content.

    • The index assigns a semantic value to every word by taking into account their frequency in the index. Very frequent words in indexed items are considered to carry less meaning. Consequently, the index already attributes minimal ranking weight for the occurrence of stop words in search results. Therefore, it’s recommended to add stop word rules only for specific use cases.

      Note

      You might consider adding stop word rules to exclude bad keywords from queries to not impact the Coveo ML model learning process. However, if end users perform queries containing only banned words, the model learning process could be affected depending on the returned search results (if any) (see Stop word special cases).

      For more information on the management of the blocklist words, see Blocklists.

  • Apply conditions:

    Typically, stop word rules should only apply when a certain condition is fulfilled.

    In general, you should ensure that this is the case by associating such a rule, and/or the query pipeline it’s defined in, to a query pipeline condition.

  • Consider using the partial match feature:

    For implementations where users tend to enter lengthy natural language queries, employing the partial match can simplify search results by eliminating the need for exhaustive stop word definitions. This option lets you define a minimum number of keywords to be found in a search result before this search result is returned. As a result, the index favors the most important keywords and stop words become optional.

  • Consider how stop words affect ART models:

    When creating stop word rules, the words specified in the rule don’t get passed to the ART models, which means that the Automatic Relevance Tuning (ART) models don’t learn from these words and won’t associate them with user interactions. However, it’s important to note that these models also have their own stop word list that can be managed through the Advanced model configuration API. These stop words help reduce the impact of common words when generating recommendations, however, a model only starts excluding them after accumulating sufficient interaction data.

    Therefore, it’s recommended to regularly review ML model training data to ensure that stop word rules aren’t adversely affecting the quality of the recommendations.

  • Test your stop words:

    When creating stop word rules, you should always perform tests to ensure that your stop words don’t negatively impact the search experience in cases other than the one you’re trying to improve.

Stop word special cases

Stop word rules ensure that certain words are removed only when they’re part of an AND (explicit or implicit) or OR expression, or sub-expression. They aren’t removed when they occur alone, because otherwise the removal could create an invalid expression.

Coveo indexes all words contained in your source items, including the stop words. This allows Coveo to manage exceptions and keep stop words in the query in the following cases:

  • Stop words within a phrase search.

    Example

    A user is looking for an item that contains a very specific phrase and encloses the phrase between double-quotes in the search box:

    "in the plan for year 2025"

    The words in, the, and for are stop words, but since the phrase is enclosed in double-quotes, all keywords in this query are sent to the index, so only items containing these keywords in the same order and as contiguous occurrences are returned.

  • A query containing only stop words.

    Example

    A user searches for:

    to be or not to be

    If all the keywords of this query are stop words, they’re all kept and sent to the index.

  • A stop word is an argument of the NOT or NEAR operators.

    Examples

    • A user searches for:

      how NEAR:10 export, meaning the user is looking for items containing both how and export occurring within ten words from each other.

      The word how has been set as a stop word. However, since it’s an argument of the NEAR operator, it will be kept to return items containing both how and export within ten words from each other.

    • A user searches for:

      (NOT how export), meaning the user is looking for items containing export but not how.

      The word how has been set as a stop word.

    If how is a stop word, because it’s an argument of the NOT operator, and the NOT operator has precedence over the implicit AND operator, it will be kept to return items containing both how and export.

  • A stop word also has a plural form.

    Example

    An online store sells a variety of wristwatches. Since the majority of their products contain the word watch or watches, searching for these words doesn’t provide relevant results, as customers are looking for either specific brands or types of watches. Therefore, you add both watch and watches to your stop words list to ensure that the search results are more relevant. You must add both forms since the plural form isn’t automatically removed when the singular form is a stop word.

Add or edit stop word rules

You can add stop word rules to a query pipeline to filter out specific words from queries.

  1. On the Query Pipelines (platform-ca | platform-eu | platform-au) page, click the query pipeline you want to modify, then select Edit components in the Action bar.

  2. On the page that opens, select the Search Terms tab.

  3. In the Search Terms tab, on the left side of the page, select Stop words.

    • If you’re creating a new rule, click Add stop word.

    • If you’re editing an existing rule, click the rule you want to edit.

  4. Under Stop word, add or edit the words that will be ignored when they appear in queries by entering one or more words separated by commas, and then press Enter.

  5. (Optional) Click Add condition to set a condition for when the stop word rule applies, or click an existing condition to edit.

    1. In the Select a condition panel that opens, in the Select a condition dropdown menu, select one of the available conditions, or create a new one by clicking Create a new condition.

    2. Click Apply Condition.

Your rule is now active.

Duplicate stop word rules

You can duplicate a stop word rule within the same query pipeline to create a slightly different rule.

  1. On the Query Pipelines (platform-ca | platform-eu | platform-au) page, click the query pipeline in which you want to duplicate a rule, and then click Edit components in the Action bar.

  2. On the page that opens, select the Search Terms tab.

  3. In the Search Terms tab, on the left side of the page, select Stop words.

  4. In the Stop words subtab, for the rule you want to duplicate, click dots, and then select Duplicate.

  5. The duplicated rule appears at the bottom of the list. Modify the duplicated rule as required.

Tip

You can verify who created or last modified a stop word rule by inspecting the Details column of the Stop Words subtab. The Details column also indicates the hour and date the rule was created or last modified.

Delete stop word rules

You can delete a stop word rule from a query pipeline.

  1. On the Query Pipelines (platform-ca | platform-eu | platform-au) page, click the query pipeline that contains the stop word you want to delete, and then click Edit components in the Action bar.

  2. On the page that opens, select the Search Terms tab.

  3. In the Search Terms tab, on the left side of the page, select Stop words.

  4. In the Stop words subtab, for the rule you want to delete, click dots, and then select Delete.

Required privileges

By default, members with the required privileges can view and edit elements of the Query Pipelines (platform-ca | platform-eu | platform-au) page.

The following table indicates the required privileges to view or edit stop word rules (see Manage privileges and Privilege reference).

Action Service - Domain Required access level

View stop word rules

Organization - Organization
Search - Query pipelines

View

Edit stop word rules

Organization - Organization

View

Search - Query pipelines

Edit