Standard query extensions

This is for:

Developer

The standard query extensions are built-in elements of the query extension language that you can use in queries. This article describes and provides examples for the available standard query extensions.

The query extensions described in this article are usable in the basic (q), advanced (aq), constant (cq), or disjunction (dq) parts of the query expression.

However, query extensions returning the sequence type (e.g., $valuesOfField, $joinValues, $splitValues, etc.) are only meant to be used as arguments of other query extensions.

For example, you want the results to be ordered by descending date (rather than relevance, the default). You inject a $sort query extension in the advanced query (aq) so that it’s transparent to the user.

/rest/search?q=user%20keywords&aq=$sort(criteria:'datedescending')

General extensions

$q

Injects the basic query expression inside another query expression.

This extension is useful for example when another part of the query (advanced or disjunction) should contain the keywords entered by the end user. A typical usage is to use it inside nested queries to match additional content.

The following advanced expression will expand to foo bar baz if the basic expression is bar: foo $q() baz

$qf

Evaluates a query function (QF) against each item in the result set and stores the resulting values in a dynamic, temporary field that’s generated at query time. These values can then be used for almost any normal numerical field operation, including further query functions.

  • For each item in the query result set, evaluate the distance between that item and a specific location, and store the resulting value in the @distance dynamic field.

    $qf(function: 'dist(@latitude, @longitude, 46.8167, -71.2167)', fieldName: 'distance')

    where:

    • @latitude and @longitude are floating point numerical fields available in your index.

    • 46.8167 and -71.2167 are the latitude and longitude of the current reference position.

  • For each item in the query result set, divide the byte size of that item by 1024, and store the resulting value in the @sizekb dynamic field.

    $qf(function: '@size/1024', fieldName: 'sizekb')

You can also use the queryFunctions parameter to include QFs in a query.

Parameters:

function (scalar [string])

The mathematical expression to evaluate against each item in the query result set.

  • QF expressions support the geolocation distance function (dist) and the C++ Mathematical Expression Library syntax (see ExprTk). However, the following statements have been disabled:

    • if/else

    • switch

    • while

    • repeat until

  • If your QF expression references certain numeric fields, you should ensure that the useCacheForComputedFacet option is enabled for each of those fields to speed up evaluation (see Available Boolean Field Options).

Sample value: 'dist(@latitude, @longitude, 46.8167, -71.2167)'

fieldName (scalar [string])

The name of the dynamic, temporary field in which to store the values resulting from evaluating the function.

Must start with a lowercase alphabetical character, and may only contain lowercase alphanumeric and/or underscore characters.

  • You can’t use dynamic field values to override existing field values (e.g., $qf(function: 'sqrt(@size)', fieldName: '@size') has no effect).

  • In a Group By operation, you can use a dynamic field as the field value. However, when doing so, setting generateAutomaticRanges to true has no effect (i.e., you must explicitly specify rangeValues). You can also use a dynamic field as the field value of a computed field.

Sample value: 'distance'

$qre

Injects a query ranking expression (QRE) in the query. A QRE applies a specific ranking score modifier to items in a specific result set.

Returns a result set.

  • Increase the ranking score of all book items by 1000 (cache the query result set).

    $qre(expression: @documenttype==Book, modifier: '100', isConstant: true)

  • Increase the ranking score of all items already ranked high enough by the index, and whose concepts contain any of the basic query expression keywords, by 100 (don’t cache the query result set).

    $qre(expression: @concepts=$splitValues(text: $query, separator: '\s'), modifier: 10, applyToEveryResults: 'false')

You can also configure conditional QREs in your query pipelines (see Ranking - Query Pipeline Feature).

Parameters:

expression (result set)

The query expression whose result set items the QRE should apply to.

Sample value: @documenttype==Book

modifier (scalar [integer])

The ranking score modifier to apply to each item in the expression query result set. A positive value increases ranking scores, whereas a negative one reduces them.

Must be in the range [-1000000, 1000000].

The modifier value is correlated to the ranking score of a query result set item by a 1 to 10 ratio (e.g., a modifier value of '100' adds 1000 to the ranking score of each item in the expression query result set).

Typically, you should use a modifier value between '-100' and '100', unless you want to completely override the index ranking scores.

Sample value: '100'

isConstant (scalar [boolean], optional)

Whether to treat the expression as a constant query expression (cq) (and cache its result set).

You should leave this option to 'false' if the expression is based on end-user input (e.g., @concepts=$splitValues(text: $query, separator: '\s')) or if it contains a nested query.

Default value: 'false'

applyToEveryResults (scalar [boolean], optional)

Whether to apply the QRE to every item in the expression query result set, regardless of its current ranking score. When this option is set to 'false', the QRE only applies to query result set items whose current ranking score is considered high enough by the index.

Default value: 'true'

$qrf

Injects a query ranking function (QRF) in the query, effectively creating a custom ranking algorithm for that query.

Returns a result set.

  • For each item in the query result set, increase the ranking score of that item by the square root of its current view count.

    $qrf(expression: 'sqrt(@viewcount)')

  • For each item in the query result set, add a normalized boost of 0 to 600 to the ranking score of that item based on whichever is higher between its last update date and a date that corresponds to four weeks ago.

    $qrf(expression: 'max(@date, (NOW - (WEEK * 4)))', normalizeWeight: 'true')

  • For each item in the query result set, add a normalized boost of 0 to 1000 to the ranking score of that item depending on how good its rating/price ratio is.

    $qrf(expression: '@rating / @price', normalizeWeight: 'true', modifier: '1000')

You can also use the rankingFunctions query parameter to include QRFs in a query.

Parameters:

expression (scalar [string])

The mathematical expression to evaluate against each item in the query result set. The result of this expression for a given item generates a boost which is then added to the ranking score of that item.

  • QRF expressions support the geolocation distance function (dist) and the C++ Mathematical Expression Library syntax (see ExprTk). However, the following statements have been disabled:

    • if/else

    • while

    • repeat until

    • switch

  • The following time constants are available in QRF expressions:

    • NOW: A value corresponding to the date and time of the precise moment when the query was performed.

    • YEAR: A value corresponding to the duration of a year (i.e., 365.25 days).

    • WEEK: A value corresponding to the duration of a week.

    • DAY: A value corresponding to the duration of a day.

  • If your QRF expression references certain numeric fields, you should ensure that the useCacheForComputedFacet option is enabled for each of those fields to speed up evaluation (see Available Boolean Field Options).

Sample value: 'sqrt(@viewcount)'

normalizeWeight (scalar [boolean], optional)

Whether to normalize the ranking score boosts resulting from the evaluation of this query ranking function using the standard index scale.

Default value: 'false'

Unless you want to completely override the index ranking and use the results of this query ranking function directly to boost the ranking scores of query results, you should set this parameter to 'true'.

modifier (scalar [unsigned integer], optional)

The maximum boost this query ranking function can add to the ranking score of any given query result. This property only has a meaning if normalizeWeight is set to 'true'.

Unlike the $qre modifier, this boost is not correlated to the ranking score of a query result set item by a 1 to 10 ratio.

Must be in range [0, 1000000].

Default value: '600'

$weight

Adjusts the weight of a specific ranking factor for the query.

Returns a result set.

  • Slightly increase the weight of the TFIDF ranking factor for the current query.

    $weight(name: 'TFIDF', value: '6')

  • Reduce the weight of the concept ranking factor to its minimum value for the current query.

    $weight(name: 'Concept', value: '0')

  • Increase the weight of the docDate ranking factor to its maximum value for the current query.

    $weight(name: 'DocDate', value: '9')

You can also configure conditional ranking weights rules in your query pipelines (see Rankingweight - Query Pipeline Feature).

Parameters:

name (scalar [string])

The name of the ranking factor whose weight should be altered.

Allowed values:

  • Adjacency: The proximity of query terms to one another in the item.

  • Concept: Query terms in the automatically populated @concepts field for the item.

  • CustomDocumentWeight: Custom weight assigned through an indexing pipeline extension for the item.

  • DocDate: How recently the item was modified.

  • Formatted: How query terms are formatted in the item (e.g., heading level, bold, large, etc.).

  • Language: Whether the item is in the language of the search interface from which the query originates.

  • Quality: The proximity of the item to the root of the indexed system.

  • SourceReputation: The rating of the source the item resides in.

  • Summary: Query terms in the summary of the item.

  • TermCorrelation: Query term correlations within stemming classes in the item.

  • TermCasing: Query term casing in the item.

  • TFIDF: Term frequency-inverse document frequency.

  • Title: Query terms in the title of the item.

  • URI: Query terms in the URI of the item.

The allowed name values are case-sensitive.

value (scalar [integer])

How much to alter the target ranking factor weight (as determined by the name argument) relative to its default weight.

Must be in range [0, 9].

A value of 5 applies the default ranking weight, as determined by the index. A value below 5 reduces the ranking factor weight, whereas a value over 5 increases it.

$sort

Sorts the result set according to the specified criterion.

Returns a result set.

  • Sort the result in descending order based on their last update date.

    $sort(criteria: 'datedescending')

  • Sort the result set items in ascending order based on their respective price.

    $sort(criteria: 'fieldascending', field: '@price')

You can also use the sortCriteria query parameter to specify how to sort the result set of a query.

Parameters:

criteria (scalar [string])

The criteria to use to sort the result set.

Allowed values:

  • relevancy: Use configured ranking weights as well as any specified ranking expressions to sort the result set.

  • dateascending/datedescending: Use the @date field to sort the result set. This field typically contains the last modification date of an item in the index.

  • rankingexpressions: Sort the result set using only the weights applied through query ranking expressions (QREs). This is similar to relevancy, except that ranking factor weights (adjacency, TFIDF, etc.), aren’t computed.

  • noranking: Don’t sort the result set. The index will return the result set items in an essentially random order.

  • fieldascending/fielddescending: Sort the result set using the values of a specified, sortable field.

Sample value: 'fieldascending'

field (scalar [string], optional)

The name of the field on which to sort the result set. Only required when using fieldascending or fielddescending as a criteria. If specified, must reference an existing, sortable field in the index.

If the field being used isn’t numeric (i.e., string), ensure that its sort option is set to true; otherwise, sorting will fail (see Available Boolean Field Options).

Sample value: '@price'

$fold

Groups up to a specific number of items matching the query and sharing the same value for a specific field. Grouped items are folded under the item in that same group that would appear first in the sorted query result set.

Returns a result set.

Fold up to five items with the same conversation ID under the same result.

$fold(field: '@conversationid', range: '5')

You can also use the filterField and filterFieldRange to request folded query results.

Parameters:

field (scalar [string])

A field that has the same value for all of the items that you want to group/fold. Must reference an existing field in the index.

The values of the field must only contain alphanumeric characters. Trying to fold items on a field whose values contain non-indexable characters (such as underscores) will fail.

Trying to fold items on a field containing values such as id_123 will fail.

The field should contain values such as id123 instead.

Sample value: '@conversationid'

range (scalar [unsigned integer])

The maximum number of items to fold under the childResults array of the group item that would appear first in the sorted query result set.

Sample value: '5'.

$loadParent

Load an item which is the parent of another item. The parent-child relationship is determined by the presence of the parent ID value among the fields of the child item.

Returns a result set.

Loads the parent result that has the ID specified in the child item.

$loadParent(parent:'@messageid', child:'@parentmessageid')

You can also use the parentField and childField to identify parent-child relationships in the result set of a query.

parent (scalar [string])

The name of the field to use to identify the parent item. This should correspond to a field whose value can uniquely identify the parent item. Any item whose child field value is the same as the parent field value of another item is considered a child of that item.

Sample value: '@messageid'

child (scalar [string])

The name of the field to use to identify an item as a child of another item. Whenever an item is a child of another item, its child field value should be the same as the parent field value of its parent.

Sample value: '@parentmessageid'

$quoteVar

Allows you to put the value between double quotes ("value"). This is useful when you want to set a value between double quotes in a basic, advanced, constant, or disjunction query. It’s also useful in query pipeline Ranking Expressions.

  • $quoteVar(value: myvalue) returns "myvalue"

  • @title=$quoteVar(value: This is my title) returns @title="This is my title".

  • $quoteVar helps when string concatenation is present in a field query expression. For example, none of the following would produce the expected @myfield=="some static text $query" output:

    • @myfield==some static text $query

    • @myfield==(some static text $query)

    • @myfield=="some static text $query"

    However, what does work with $quoteVar is:

    • @myfield==$quoteVar(value: some static text $query)

$stripXml

Removes all XML tags from an expression.

Consider the following context:

{
  "context": {
    "Case_Subject": "<h1>Hello</h1>"
  }
}

The following QueryParamOverride: override query q: \"<@+ $stripXml(value: $context.Case_Subject) +@>\" will result in q containing “Hello”.

Argument Description
value The value to filter.

Extensions for Lists of Values

$valuesOfField

Lists all the distinct values of a field in a result set.

The following query extension returns all the work emails associated with the result set specified by the Contacts alias.

$valuesOfField(field: '@sysworkemail', resultSet: @firstname==john)

Argument Description
resultSet The result set for which to compute the list of values.
field The name of the field whose values should be retrieved.
aggregateConditions Optional. Aggregate conditions to use to filter the values based on a computed field operation (see Aggregate Conditions).
maximum Optional. The maximum number of values to retrieve. If not specified, all the values are retrieved.
sortOrder Optional. The sort order to use when retrieving values (see Group By Sort Orders). The default value is donotsort.

$keepMatchingValues

Filters a list to select only the values matching a regular expression.

The following extension returns only email addresses from the company.com domain.

$keepMatchingValues(regex: '.*@company.com', values: {{addresses}})

Argument Description
regex

The regular expression to use.

Use the Java regular expression (REGEX) syntax (see Regular Expressions ).
values The list of values to filter.

$removeMatchingValues

Filters a list to remove all values matching a regular expression.

The following extension returns all email addresses except those ending with @company.com.

$removeMatchingValues(regex: '.*@company.com', values: {{addresses}})
Argument Description
regex

The regular expression to use.

Use the Java regular expression (REGEX) syntax (see Regular Expressions ).
values The list of values to filter.

$removeEmptyValues

Filters a list to remove all the empty values.

The following extension eliminates empty values from the values returned by {{addresses}}.

$removeEmptyValues(values: {{addresses}})
Argument Description
values The list of values to filter.

$removeDuplicateValues

Filters a list to remove all the duplicate values.

The following extension eliminates duplicate values from the values returned by {{addresses}}.

$removeDuplicateValues(values: {{addresses}})
Argument Description
values The list of values to filter.

$replaceInValues

Transforms a list using a regular expression.

In the list of values returned by {{addresses}}, the following extensions respectively replace occurrences of the string company with the string somethingelse, and .com value endings by .org.

$replaceInValues(values: {{addresses}}, regex: 'company', replacement: 'somethingelse')
$replaceInValues(values: {{addresses}}, regex: '(.*)\.com', replacement: '$1.org')
Argument Description
regex

The regular expression to use.

Use the Java regular expression (REGEX) syntax (see Regular Expressions ).
replacement The replacement string to use with the regular expression.
values The list of values to transform.
removeNonMatchingValues Optional. If true, all values that aren't matching the regular expressions are removed. By default, non-matching values are kept as is.
removeEmptyValues Optional. If true, all values that are empty following the replacement are removed. By default, empty values are kept.
removeDuplicateValues Optional. If true, duplicate values following the replacement are removed. By default, duplicate values are kept.

$mergeValues

Merges two lists of values together. Duplicate values are included only once.

The following query extension merges the {{addresses}} and the {{anotherListOfAddresses}} lists of values.

$mergeValues(first: {{addresses}}, second: {{anotherListOfAddresses}})
Argument Description
first The first list of values to merge.
second The second list of values to merge.

$joinValues

Joins a list of values into a string.

The following expression joins a list of values into a string:

$joinValues(values: {{values}}, separator: ',')
Argument Description
values The list of values to join.
separator The separator to put between values.

$splitValues

Splits a string into a list of values.

The following expression splits a string into a list of values:

$splitValues(text: 'foo, bar, baz', separator: ',')
Argument Description
text The string value to split.
separator The separator that separates each value.

Join Extensions

The query extensions of join type produce similar operation as a join keyword used in an SQL statement.

$valuesToResultSet

Converts a list of values to a result set matching the items where a given field is equal to a value in the list. The original order of the values in the result set is preserved using ranking expressions.

The following query extension returns a result set of items for which the @sysworkemail field contains a value present in the {{addresses}} list of values.

$valuesToResultSet(values: {{addresses}}, field: '@sysworkemail')
Argument Description
values The values to filter with.
field

The name of the field to use.
modifier Optional. The modifier to use for the ranking expressions.
minimumModifier Optional. The minimum modifier to use for the ranking expressions.

$joinOnValues

Filters a result set to only the items for which a given field is equal to one of the values contained in a list. The order of the values can optionally be preserved in the resulting result set when the modifier argument is specified. The extension returns a result set filtered on the field values. Any ranking expressions associated with the result set are kept.

The following query extension returns only items for which the email domain name is listed in the specified domains alias.

$joinOnValues(resultSet: @uri, field: '@sfemaildomainnames', values: {{domains}})
Argument Description
resultSet The result set to filter.
field

The name of the field to use.
values The values to filter with.
modifier Optional. The highest modifier to use for the ranking expressions.
minimumModifier Optional. The minimum modifier to use for the ranking expressions.

$join

Filters a result set to only the items for which a given field is equal to a value of another field in another result set. If the toResultSet contains ranking expressions or sort orders, they will be included in the resulting result set. The order of the values can optionally be preserved in the resulting result set when the modifier argument is specified.

The following query extension returns only accounts with a won opportunities for which the opportunity amount sum is greater or equal to the specified amount alias.

$join(fromResultSet:  $type(name: 'Opportunity') @sfopportunitystagename=won,
      toResultSet: $type(name: 'Account'),
      fromAggregateConditions: $sum(@sfopportunityamount)>={{amount}},
      field: '@sfaccountid')
Argument Description
fromResultSet The result set from which the list of values is extracted.
toResultSet The result set to filter.
fromField Required if field isn’t set. The name of the field to use in the fromResultSet.
toField Required if field isn’t set. The name of the field to use in the toResultSet.
field Required if fromField and toField aren’t set. The name of the field to use in both result sets. This is equivalent to specifying both fromField and toField with the same value.
fromAggregateConditions Optional. Aggregate conditions to use to filter the from result set before performing the join (see Aggregate Conditions).
maximumValues Optional. The maximum number of values to filter on. By default, all values are used.
sortOrder Optional. The sort criteria to use when retrieving values from the fromResultSet. The default value is donotsort.
modifier Optional. The highest modifier to use for the ranking expressions.
minimumModifier Optional. The minimum modifier to use for the ranking expressions.

$filterJoin

Filters a result set to only the items that can be joined in another result set using the specified fields. If the local result set contains ranking expressions or sort orders, they’re included in the resulting result set. This query extension basically performs two joins, one from the local to the external result set, and another back from the external to the local result set.

The following query extension returns accounts with won opportunities for which the total amount is greater than 100,000$.

$filterJoin(localResultSet: $type(name: 'Account'), externalResultSet: (@sfopportunitystagename=won), field: '@sfaccountid', aggregateConditions: $sum(@sfopportunityamount)>100000, maximumValues: 5)
Argument Description
localResultSet The result set to filter.
externalResultSet The result set to perform a join on in order to filter items in localResultSet.
localField Required if field isn’t set. The name of the field to use in the localResultSet.
externalField Required if field isn’t set. The name of the field to use in the externalResultSet.
field Required if localField and externalField aren’t set. The name of the field to use in both result sets. This is equivalent to specifying both localField and externalField with the same value.
aggregateConditions Optional. Aggregate conditions to use to filter the result set resulting from joining the local result set to the external one (see Aggregate Conditions).
maximumValues Optional. The maximum number of values to filter on. By default, all values are used.

Correlation extensions

$correlateListOfValues

Outputs a set of ranking expressions that boosts results having one of several values in a specified field.Arguments:

Argument Description
field The name of the field to use.
values The values to use for boosting.
modifier The maximum modifier to apply when a result matches. Items matching the first value in the list get the highest modifier, and those matching the last one get the lowest modifier.
useContainsOperator Optional. Whether to match using the containment operator (=) rather than the identity operator (==), which is used by default.
useFullText Optional. If true, query ranking expressions (QRE) will also be output for items matching the field values in full text. The default value is false.
forceOneMatch Optional. If true, a query expression will be added that ensures that all results have a least one matching field value. The default value is false.

$correlateResultSet

Outputs a set of ranking expressions that boosts results sharing a field value from another result set.

Argument Description
resultSet The result set from which to grab the list of values.
fromField Required if field isn’t set. The name of the field to use in resultSet.
toField Required if field isn’t set. The name of the field to use in the resulting result set.
field Required if fromField and toField aren’t set. The name of the field to use in both result sets. This is equivalent to specifying both fromField and toField with the same value.
maximumValues Optional. The maximum number of values to retrieve from the source result set. By default, all values are used.
sortOrder Optional. The sort criteria to use when retrieving values from resultSet. The default value is chisquare.
modifier The highest modifier to use for the ranking expressions. The maximum modifier to apply when a result matches. Items matching the first value in the list get the highest modifier, and those matching the last one gets the lowest modifier.
useFullText Optional. If true, query ranking expressions (QRE) will also be outputted for items matching the field values in full text. The default value is false .
forceOneMatch Optional. If true, a query expression will be added that ensures that all results have a least one matching field value. The default value is false .

$correlateUsingIdf

Outputs an expression that ranks up items sharing keywords with those specified. Uses the inverse document frequency (IDF) method to give a greater weight to keywords that are less frequent in the index.

You may want to apply this extension when you need to make accessible items that aren’t frequently clicked within your index (e.g., articles about less popular products).

Argument Description
keywords The string from which to extract keywords.
forceOneMatch Optional. If true, a query expression will be added that ensures that all results have at least one matching keyword. The default value is false .
removeStopWords

Optional. If true, any English stop word present in the keywords will be automatically removed. The default value is true.

This argument is deprecated. You should use the Stop Words query pipeline feature to filter out stop words from the query expression (see Manage Query Pipeline Stop Words).
noStemming

Optional. If true, keywords won't be expanded using the index stemming before matching items. The default value is false.
maximum Optional. The maximum number of keywords to use. If a larger number of keywords is provided, some keywords will be completely ignored. The default value is 100.

$some

Extracts up to a specific part of “best” terms from a string, then retrieves items matching at least a specific part of those terms from the index.

Returns a result set.

  • Retrieve items containing at least the foo, bar, and baz terms. $some(keywords: 'foo bar baz')

  • Retrieve items containing at least the one “best” term among foo, bar, and baz. $some(keywords: 'foo bar baz', best: '1')

  • Retrieve items containing at least two terms among foo, bar, and baz. $some(keywords: 'foo bar baz', match: '50%')

  • Retrieve items containing at least one of the best two terms among foo, bar, and baz. $some(keywords: 'foo bar baz', best: '50%', 'match': '1')

  • Retrieve items containing at least the terms foo and bar. $some(keywords: 'foo bar baz', maximum: '2')

  • Retrieve items containing at least one of the best two terms among foo, bar, and baz. $some(keywords: 'foo bar baz hello', best: '50%', match: '50%', maximum: '3')

keywords (scalar [string])

The string from which to extract a subset of best terms.

All original keywords terms (not only “best” terms) are highlighted in matching items.

best (scalar [string | unsigned integer], optional)

The absolute or percentage (rounded up) maximum number of “best” terms to extract from keywords. Terms occurring less frequently in the index are considered “better” than more common keywords. Applies after maximum, but before match.

Sample values:

  • '5'

  • '75%'

Default value: '100%'

match (scalar [string | unsigned integer], optional)

The absolute or percentage (rounded up) minimum number of “best” terms an item must contain to be included in the result set. Applies after best.

Sample values:

  • '5'

  • '75%'

Default value: '100%'

removeStopWorlds (scalar [boolean], optional)

Deprecated: this parameter is exposed for legacy compatibility reasons. Use query pipeline statements expressing the stop feature instead (see Stop - Query Pipeline Feature).

Whether to ignore standard English stop words (a, the, etc.) in the keywords.

Default value: 'false'

maximum (scalar [unsigned integer], optional)

The absolute maximum number of terms in keywords to consider when extracting terms (consider the string as a sequence). Excess keywords are ignored.

Default value: '100'

$removeStopWords

Removes the English stop words from a list of keywords.

Argument Description
keywords The string from which to extract keywords.

This extension is deprecated. You should use the Stop Words query pipeline feature to filter out stop words from the query expression (see Manage Stop Word Rules).

$noStemming

Arranges for a list of keywords to match without using stemming.

Argument Description
keywords The string from which to extract keywords.