Filter - Query pipeline feature

This is for:

Developer System Administrator
In this article

A query pipeline statement expressing the filter query pipeline feature adds one or more query expressions to the advanced (aq), constant (cq), disjunction (dq), large (lq), or basic (q) part of the query expression before the query is sent to the index.

Tip
Leading practice

Typically, a statement expressing the filter feature should only apply when a certain condition is fulfilled.

In general, you should ensure that this is the case by associating such a statement, and/or the pipeline it’s defined in, to a global condition.

Moreover, when basing a filter statement on the value of one or more $context keys, you should use a condition to assert that those keys contain values. Otherwise, you risk injecting invalid expressions in the target part of the query expression (that is, aq, cq, etc.).

Example

Condition

when $context[favoriteBookGenre] matches /*.\S.*/

Statement

filter aq `@bookgenre==$context.favoriteBookGenre`
Note

In the Coveo Administration Console, you can manage statements expressing the filter feature from the Filters tab (see Manage filter rules).

The following diagram highlights when statements expressing the filter feature are applied in the query pipeline (see Order of execution of query pipeline features).

Apply filter statements
Important

Filters, by themselves, don’t prevent the exposure of filtered content. We strongly advise against creating a source whose content is accessible to everyone ("sourceVisibility": "PUBLIC") and using a pipeline filter to exclude sensitive information.

In the following cases, sensitive content from a source whose content is accessible to everyone ("sourceVisibility": "PUBLIC") could be exposed:

  • A filter rule based on the aq query parameter is set on a query pipeline that contains an ART model for which the Match the advanced query option is disabled.

  • Using other pipelines not having a similar filter from other search interfaces or directly from the API.

  • A colleague not understanding the reason for the filter could modify or remove the filter.

You can ensure security by enforcing the search hub at the search token level (see Search token authentication). Moreover, search hubs defined on the client-side that are used as conditions in pipelines don’t safeguard the security of the filtered content.

Syntax

Use the following query pipeline language (QPL) syntax to define a statement expressing the filter feature:

filter <queryExpressionPart> <expressions>

<queryExpressionPart>

A string indicating the part of the query expression to which one or more query expressions should be appended. Must be one of:

  • aq: The advanced part

  • cq: The constant part

  • dq: the disjunction part

  • lq: the large part

  • q: the basic part

Notes

  • The combined query expression is: ((q AND aq) OR dq) AND cq.

  • When an Automatic Relevance Tuning (ART) (that is, topClicks) model is configured and ready in the query pipeline, the Coveo Machine Learning (Coveo ML) Intelligent Term Detection (ITD) feature extracts relevant keywords from the large part of the query expression (lq). Otherwise, the Search API converts lq to a partial match expression (see the lqPartialMatchKeywords and lqPartialMatchThreshold query parameters). In both instances, the subset of keywords processed from lq is ultimately injected in the basic part of the query expression (q).

<expressions>

A comma-separated list of query expressions (for example, foo`, @bar=="baz"). Each of those query expressions will be appended to the current value of the target `<queryExpressionPart> using an implicit AND operator (that is, a whitespace character).

Important

The result set of the constant part (cq) of the combined query expression is kept in a special cache.

Consequently, you should avoid creating filter statements that add dynamic content to cq, otherwise you risk filling the cache with useless data, which might have a negative impact on performance.

Example

You create a global condition with the following QPL definition:

Global condition:

when $searchHub is "Community"

In an empty query pipeline named Testing Filter, you create three distinct statements, each expressing the filter feature, with the following QPL definitions:

Statement 1:

filter cq `@language==en`

Statement 2:

filter aq `@source=="Public Content"`

Statement 3:

filter dq `@date=today`

You associate each of those three statements to the global condition you created before.

A user performs a query with the following payload:

{
  "aq": "@tags==(\"Query pipeline\", \"Search API\")",
  "context": {
    "userRole": "Developer"
  },
  "cq": "@filetype==html",
  "pipeline": "Testing Filter",
  "q": "filter statement",
  "searchHub": "Community"
}

Since this query goes through the Testing Filter query pipeline and satisfies the condition of each statement in that pipeline, those statements apply. As a result, the different parts of the query expression are modified as follows:

  • cq (that is, $constantQuery) becomes @filetype==html @language==en.

  • aq (that is, $advancedQuery) becomes @tags==("Query pipeline", "Search API") @source=="Public Content".

  • dq (that is, $disjunctionQuery) becomes @date=today.

  • q (that is, $query) remains filter statement.

The combined query expression (that is, ((q AND aq) OR dq) AND cq) means that to be included in the result set of this query, an item must mandatorily:

  • Be an HTML file (because of the original cq query parameter).

  • Be in English (because of Statement 1, which modifies the cq expression).

In addition, this item must:

  • Match the filter statement basic query expression (because of the q query parameter).

  • Be tagged as being about query pipelines or the Search API (because of the original aq query parameter).

  • Reside within the source named Public Content (because of Statement 2, which modifies the aq expression).

    OR

  • Have been updated today (because of Statement 3, which modifies the dq expression).