Filter - Query Pipeline Feature

A query pipeline statement expressing the filter query pipeline feature adds one or more query expressions to the advanced (aq), constant (cq), disjunction (dq), large (lq), or basic (q) part of the query expression before the query is sent to the index.

Typically, a statement expressing the filter feature should only apply when a certain condition is fulfilled.

In general, you should ensure that this is the case by associating such a statement, and/or the pipeline it’s defined in, to a global condition.

Moreover, when basing a filter statement on the value of one or more $context keys, you should use a condition to assert that those keys contain values. Otherwise, you risk injecting invalid expressions in the target part of the query expression (i.e., aq, cq, etc.).

Condition

when $context[favoriteBookGenre] matches /*.\S.*/

Statement

filter aq `@bookgenre==$context.favoriteBookGenre`

In the Coveo Administration Console, you can manage statements expressing the filter feature from the Filters tab (see Manage Filter Rules).

The following diagram highlights when statements expressing the filter feature are applied in the query pipeline (see Order of Execution of Query Pipeline Features).

Apply filter statements

Filters, by themselves, don’t protect the security of filtered content. We strongly advise against creating a source whose content is accessible to everyone ("sourceVisibility": "PUBLIC") and using a pipeline filter to exclude sensitive information.

In the following cases, sensitive content from a source whose content is accessible to everyone ("sourceVisibility": "PUBLIC") could be exposed:

A colleague not understanding the reason for the filter could modify or remove the filter.
Using other pipelines not having a similar filter from other search interfaces or directly from the API.

You can ensure security by enforcing the search hub at the search token level (see Search Token Authentication). Moreover, search hubs defined on the client-side that are used as conditions in pipelines do not safeguard the security of your filters.

Syntax

Use the following query pipeline language (QPL) syntax to define a statement expressing the filter feature:

filter <queryExpressionPart> <expressions>

`<queryExpressionPart>`

A string indicating the part of the query expression to which one or more query expressions should be appended. Must be one of:

aq: The advanced part
cq: The constant part
dq: the disjunction part
lq: the large part
q: the basic part

The combined query expression is: ((q AND aq) OR dq) AND cq.
When an Automatic Relevance Tuning (ART) (i.e., topClicks) model is configured and ready in the query pipeline, the Coveo Machine Learning (Coveo ML) Intelligent Term Detection (ITD) feature extracts relevant keywords from the large part of the query expression (lq). Otherwise, the Search API converts lq to a partial match expression (see the lqPartialMatchKeywords and lqPartialMatchThreshold query parameters). In both instances, the subset of keywords processed from lq is ultimately injected in the basic part of the query expression (q).

`<expressions>`

A comma-separated list of query expressions (e.g., `foo`, `@bar=="baz"`). Each of those query expressions will be appended to the current value of the target <queryExpressionPart> using an implicit AND operator (i.e., a whitespace character).

The result set of the constant part (cq) of the combined query expression is kept in a special cache.

Consequently, you should avoid creating filter statements that add dynamic content to cq, otherwise you risk filling the cache with useless data, which might have a negative impact on performance.

Example

You create a global condition with the following QPL definition:

Global condition:

when $searchHub is "Community"

In an empty query pipeline named Testing Filter, you create three distinct statements, each expressing the filter feature, with the following QPL definitions:

Statement 1:

filter cq `@language==en`

Statement 2:

filter aq `@source=="Public Content"`

Statement 3:

filter dq `@date=today`

You associate each of those three statements to the global condition you created before.

A user performs a query with the following payload:

{
  "aq": "@tags==(\"Query pipeline\", \"Search API\")",
  "context": {
    "userRole": "Developer"
  },
  "cq": "@filetype==html",
  "pipeline": "Testing Filter",
  "q": "filter statement",
  "searchHub": "Community"
}

Since this query goes through the Testing Filter query pipeline and satisfies the condition of each statement in that pipeline, those statements apply. As a result, the different parts of the query expression are modified as follows:

cq (i.e., $constantQuery) becomes @filetype==html @language==en.
aq (i.e., $advancedQuery) becomes @tags==("Query pipeline", "Search API") @source=="Public Content".
dq (i.e., $disjunctionQuery) becomes @date=today.
q (i.e., $query) remains filter statement.

The combined query expression (i.e., ((q AND aq) OR dq) AND cq) means that to be included in the result set of this query, an item must mandatorily:

Be an HTML file (because of the original cq query parameter).
Be in English (because of Statement 1, which modifies the cq expression).

In addition, this item must:

Match the filter statement basic query expression (because of the q query parameter).
Be tagged as being about query pipelines or the Search API (because of the original aq query parameter).
Reside within the source named Public Content (because of Statement 2, which modifies the aq expression).

OR
Have been updated today (because of Statement 3, which modifies the dq expression).

What's Next for Me?