Standard query extensions

This is for:

Developer

The standard query extensions are built-in elements of the query extension language that you can use in queries. This article describes and provides examples for the available standard query extensions.

Note

The query extensions described in this article are usable in the basic (q), advanced (aq), constant (cq), or disjunction (dq) parts of the query expression.

However, query extensions returning the sequence type, such as $valuesOfField, $joinValues, or $splitValues, are only meant to be used as arguments of other query extensions.

For example, you want the results to be ordered by descending date rather than the default, which is relevance. You inject a $sort query extension into the advanced query expression (aq) so that it’s transparent to the user:

/rest/search?q=user%20keywords&aq=$sort(criteria:'datedescending')

General extensions

$q

Injects the basic query expression (q) inside another query expression.

This extension is useful when another part of the query expression (either advanced or disjunction) should contain the keywords entered by the user. A typical use is inside nested queries to match additional content.

Examples

The following advanced query expression will expand to foo bar baz if the basic query expression is bar: foo $q() baz.

$qf

Evaluates a query function against each item in the result set and stores the resulting values in a dynamic, temporary field that’s generated at query time. These values can then be used for almost any normal numerical field operation, including further query functions.

Examples

  • For each item in the query result set, evaluate the distance between that item and a specific location, and store the resulting value in the @distance dynamic field.

    $qf(function: 'dist(@latitude, @longitude, 46.8167, -71.2167)', fieldName: 'distance')

    where:

    • @latitude and @longitude are floating point numerical fields available in your index.

    • 46.8167 and -71.2167 are the latitude and longitude of the current reference position.

  • For each item in the query result set, divide the byte size of that item by 1024, and store the resulting value in the @sizekb dynamic field.

    $qf(function: '@size/1024', fieldName: 'sizekb')

Note

You can also use the queryFunctions parameter to include query functions in a query.

$qf parameters:

function (scalar [string])

The mathematical expression to evaluate against each item in the query result set.

Notes

  • query function expressions support the geolocation distance function (dist) and the C++ Mathematical Expression Library syntax (see ExprTk). However, the following statements have been disabled:

    • if/else

    • switch

    • while

    • repeat until

  • If your query function expression references certain numeric fields, you should ensure that the useCacheForComputedFacet option is enabled for each of those fields to speed up evaluation.

Sample value: 'dist(@latitude, @longitude, 46.8167, -71.2167)'

fieldName (scalar [string])

The name of the dynamic, temporary field in which to store the values resulting from evaluating the function.

Must start with a lowercase alphabetical character, and may only contain lowercase alphanumeric or underscore characters.

Notes

  • You can’t use dynamic field values to override existing field values. For example, $qf(function: 'sqrt(@size)', fieldName: '@size') has no effect.

  • In a Group By operation, you can use a dynamic field as the field value. However, when doing so, setting generateAutomaticRanges to true has no effect (that is, you must explicitly specify rangeValues). You can also use a dynamic field as the field value of a computed field.

Sample value: 'distance'

$qre

Injects a query ranking expression (QRE) into the query. A QRE applies a specific ranking score modifier to the items in a specific result set.

Returns a result set.

Examples

  • Increase the ranking score of all book items by 1000 (cache the query result set):

    $qre(expression: @documenttype==Book, modifier: '100', isConstant: true)

  • Increase the ranking score of all items already ranked high enough by the index, and whose concepts contain any of the basic query expression keywords, by 100 (don’t cache the query result set):

    $qre(expression: @concepts=$splitValues(text: $query, separator: '\s'), modifier: 10, applyToEveryResults: 'false')

Note

You can also configure conditional QREs in your query pipelines (see Ranking - Query pipeline feature).

$qre parameters:

expression (result set)

The query expression whose result set items the QRE should apply to.

Sample value: @documenttype==Book

modifier (scalar [integer])

The ranking score modifier to apply to each item in the expression query result set. A positive value increases ranking scores, whereas a negative one reduces them.

Must be in the range [-1000000, 1000000].

Important

The modifier value is correlated to the ranking score of a query result set item by a 1 to 10 ratio. For example, a modifier value of '100' adds 1000 to the ranking score of each item in the expression query result set.

Typically, you should use a modifier value between '-100' and '100', unless you want to completely override the index ranking scores.

Sample value: '100'

isConstant (scalar [boolean], optional)

Whether to treat the expression as a constant query expression (cq) (and cache its result set).

Note

You should leave this option set to 'false' if the expression contains a nested query or if it’s based on user input:

@concepts=$splitValues(text: $query, separator: '\s')

Default value: 'false'

applyToEveryResults (scalar [boolean], optional)

Whether to apply the QRE to every item in the expression query result set, regardless of its current ranking score. When this option is set to 'false', the QRE only applies to query result set items whose current ranking score is considered high enough by the index.

Default value: 'true'

$qrf

Injects a query ranking function (QRF) in the query, effectively creating a custom ranking algorithm for that query.

Returns a result set.

Examples

  • For each item in the query result set, increase the ranking score of that item by the square root of its current view count:

    $qrf(expression: 'sqrt(@viewcount)')

  • For each item in the query result set, add a normalized boost of 0 to 600 to the ranking score of that item based on whichever is higher between its last update date and a date that corresponds to four weeks ago:

    $qrf(expression: 'max(@date, (NOW - (WEEK * 4)))', normalizeWeight: 'true')

  • For each item in the query result set, add a normalized boost of 0 to 1000 to the ranking score of that item depending on how good its rating or price ratio is:

    $qrf(expression: '@rating / @price', normalizeWeight: 'true', modifier: '1000')

Note

You can also use the rankingFunctions query parameter to include QRFs in a query.

$qrf parameters:

expression (scalar [string])

The mathematical expression to evaluate against each item in the query result set. The result of this expression for a given item generates a boost which is then added to the ranking score of that item.

Notes

  • QRF expressions support the geolocation distance function (dist) and the C++ Mathematical Expression Library syntax (see ExprTk). However, the following statements have been disabled:

    • if/else

    • switch

    • while

    • repeat until

  • The following time constants are available in QRF expressions:

    • NOW: A value corresponding to the date and time of the precise moment when the query was performed.

    • YEAR: A value corresponding to the duration of a year (that is, 365.25 days).

    • WEEK: A value corresponding to the duration of a week.

    • DAY: A value corresponding to the duration of a day.

  • If your QRF expression references certain numeric fields, you should ensure that the useCacheForComputedFacet option is enabled for each of those fields to speed up evaluation.

Sample value: 'sqrt(@viewcount)'

normalizeWeight (scalar [boolean], optional)

Whether to normalize the ranking score boosts resulting from the evaluation of this {query-ranking-function} using the standard index scale.

Default value: 'false'

Tip

Unless you want to completely override the index ranking and use the results of this {query-ranking function} directly to boost the ranking scores of query results, you should set this parameter to 'true'.

modifier (scalar [unsigned integer], optional)

The maximum boost that this {query-ranking-function} can add to the ranking score of any given query result. This property only has a meaning if normalizeWeight is set to 'true'.

Note

Unlike the $qre modifier, this boost isn’t correlated to the ranking score of a query result set item by a 1 to 10 ratio.

Must be in the range [0, 1000000].

Default value: '600'

$weight

Adjusts the weight of a specific ranking factor for the query.

Returns a result set.

Examples

  • Slightly increase the weight of the TFIDF ranking factor for the current query:

    $weight(name: 'TFIDF', value: '6')

  • Reduce the weight of the concept ranking factor to its minimum value for the current query:

    $weight(name: 'Concept', value: '0')

  • Increase the weight of the docDate ranking factor to its maximum value for the current query:

    $weight(name: 'DocDate', value: '9')

Note

You can also configure conditional ranking weight rules in your query pipelines.

$weight parameters:

name (scalar [string])

The name of the ranking factor whose weight should be altered.

Allowed values:

  • Adjacency: The proximity of query terms to one another in the item.

  • Concept: query terms in the automatically populated @concepts field for the item.

  • CustomDocumentWeight: Custom weight assigned through an indexing pipeline extension (IPE) for the item.

  • DocDate: How recently the item was modified.

  • Formatted: How query terms are formatted in the item. Possible formatting types include heading level, bold, and large.

  • Language: Whether the item is in the language of the search interface from which the query originates.

  • Quality: The proximity of the item to the root of the indexed system.

  • SourceReputation: The rating of the source the item resides in.

  • Summary: query terms in the summary of the item.

  • TermCorrelation: query term correlations within stemming classes in the item.

  • TermCasing: query term casing in the item.

  • TFIDF: Term frequency-inverse document frequency.

  • Title: query terms in the title of the item.

  • URI: query terms in the URI of the item.

Important

The allowed name values are case-sensitive.

value (scalar [integer])

How much to alter the target ranking factor weight (as determined by the name argument), relative to its default weight.

Must be in the range [0, 9].

A value of 5 applies the default ranking weight, as determined by the index. A value of less than 5 reduces the ranking factor weight, whereas a value greater than 5 increases it.

$sort

Sorts the result set according to the specified criterion.

Returns a result set.

Examples

  • Sort the result set items in descending order based on the date of their last update:

    $sort(criteria: 'datedescending')

  • Sort the result set items in ascending order based on their respective prices:

    $sort(criteria: 'fieldascending', field: '@price')

Note

You can also use the sortCriteria query parameter to specify how to sort the result set of a query.

$sort parameters:

criteria (scalar [string])

The criteria to use to sort the result set.

Allowed values:

  • relevancy: Use configured ranking weights as well as any specified ranking expressions to sort the result set.

  • dateascending/datedescending: Use the @date field to sort the result set. This field typically contains the last modification date of an item in the index.

  • rankingexpressions: Sort the result set using only the weights applied through query ranking expressions (QREs). This is similar to relevancy, except that ranking factor weights such as adjacency and TFIDF aren’t computed.

  • noranking: Don’t sort the result set. The index will return the result set items in an essentially random order.

  • fieldascending/fielddescending: Sort the result set using the values of a specified, sortable field.

Sample value: 'fieldascending'

field (scalar [string], optional)

The name of the field on which to sort the result set. Only required when using fieldascending or fielddescending as a criteria. If specified, must reference an existing, sortable field in the index.

Note

If the field being used isn’t numeric (that is, string), ensure that its sort option is set to true. Otherwise, sorting will fail.

Sample value: '@price'

$fold

Groups up to a specific number of items matching the query and sharing the same value for a specific field. Grouped items are folded under the item in that same group that would appear first in the sorted query result set.

Returns a result set.

Example

Fold up to five items with the same conversation ID under the same result:

$fold(field: '@conversationid', range: '5')

Note

You can also use the filterField and filterFieldRange query parameters to request folded query results.

$fold parameters:

field (scalar [string])

A field that has the same value for all of the items that you want to group or fold. Must reference an existing field in the index.

Note

The values of the field must only contain alphanumeric characters. Trying to fold items on a field whose values contain non-indexable characters (such as underscores) will fail.
Example

Trying to fold items on a field containing values such as id_123 will fail.

The field should contain values such as id123 instead.

Sample value: '@conversationid'

range (scalar [unsigned integer])

The maximum number of items to fold under the childResults array of the group item that would appear first in the sorted query result set.

Sample value: '5'.

$loadParent

Load an item which is the parent of another item. The parent-child relationship is determined by the presence of the parent ID value among the fields of the child item.

Returns a result set.

Example

Loads the parent result that has the ID specified in the child item.

$loadParent(parent:'@messageid', child:'@parentmessageid')

Note

You can also use the parentField and childField query parameters to identify parent-child relationships in the result set of a query.

parent (scalar [string])

The name of the field to use to identify the parent item. This should correspond to a field whose value can uniquely identify the parent item. Any item whose child field value is the same as the parent field value of another item is considered a child of that item.

Sample value: '@messageid'

child (scalar [string])

The name of the field to use to identify an item as a child of another item. Whenever an item is a child of another item, its child field value should be the same as the parent field value of its parent.

Sample value: '@parentmessageid'

$quoteVar

Allows you to put the value between double quotes ("value"). This is useful when you want to set a value between double quotes in a basic, advanced, constant, or disjunction query. It’s also useful in query ranking expressions (QREs).

Examples

  • $quoteVar(value: myvalue) returns "myvalue".

  • @title=$quoteVar(value: This is my title) returns @title="This is my title".

  • $quoteVar helps when string concatenation is present in a field query expression. For example, none of the following would produce the expected @myfield=="some static text $query" output:

    • @myfield==some static text $query

    • @myfield==(some static text $query)

    • @myfield=="some static text $query"

    However, the following does work with $quoteVar:

    • @myfield==$quoteVar(value: some static text $query)

$stripXml

Removes all XML tags from an expression.

Example

Consider the following context:

{
  "context": {
    "Case_Subject": "<h1>Hello</h1>"
  }
}

The QueryParamOverride override query q: \"<@+ $stripXml(value: $context.Case_Subject) +@>\" will result in a q containing "Hello".

Argument Description

value

The value to filter.

Extensions for lists of values

$valuesOfField

Lists all of the distinct values of a field in a result set.

Example

The following query extension returns all of the work emails associated with the result set specified by the Contacts alias:

$valuesOfField(field: '@sysworkemail', resultSet: @firstname==john)

Argument Description

resultSet

The result set for which to compute the list of values.

field

The name of the field whose values should be retrieved.

aggregateConditions

Optional. The aggregate conditions to use to filter the values based on a computed field operation.

maximum

Optional. The maximum number of values to retrieve. If not specified, all of the values are retrieved.

sortOrder

Optional. The sort order to use when retrieving values. The default value is donotsort.

$keepMatchingValues

Filters a list to select only the values matching a regular expression.

Example

The following extension returns only email addresses from the company.com domain:

$keepMatchingValues(regex: '.*@company.com', values: {{addresses}})

Argument Description

regex

The regular expression to use.

Note

Use the Java regular expression (REGEX) syntax.

values

The list of values to filter.

$removeMatchingValues

Filters a list to remove all values matching a regular expression.

Example

The following extension returns all email addresses except those ending with @company.com:

$removeMatchingValues(regex: '.*@company.com', values: {{addresses}})

Argument Description

regex

The regular expression to use.

Note

Use the Java regular expression (REGEX) syntax.

values

The list of values to filter.

$removeEmptyValues

Filters a list to remove all of the empty values.

Example

The following extension eliminates the empty values from the values returned by {{addresses}}:

$removeEmptyValues(values: {{addresses}})

Argument Description

values

The list of values to filter.

$removeDuplicateValues

Filters a list to remove all of the duplicate values.

Example

The following extension eliminates the duplicate values from the values returned by {{addresses}}:

$removeDuplicateValues(values: {{addresses}})

Argument Description

values

The list of values to filter.

$replaceInValues

Transforms a list using a regular expression.

Example

In the list of values returned by {{addresses}}, the following extensions replace occurrences of the string company with the string somethingelse, and .com value endings with .org.

$replaceInValues(values: {{addresses}}, regex: 'company', replacement: 'somethingelse')
$replaceInValues(values: {{addresses}}, regex: '(.*)\.com', replacement: '$1.org')
Argument Description

regex

The regular expression to use.

Note

Use the Java regular expression (REGEX) syntax.

replacement

The replacement string to use with the regular expression.

values

The list of values to transform.

removeNonMatchingValues

Optional. If true, all values that don’t match the regular expression are removed. By default, non-matching values are kept as is.

removeEmptyValues

Optional. If true, all values that are empty following the replacement are removed. By default, empty values are kept.

removeDuplicateValues

Optional. If true, all duplicate values following the replacement are removed. By default, duplicate values are kept.

$mergeValues

Merges two lists of values together. Duplicate values are only included once.

Example

The following query extension merges the {{addresses}} and {{anotherListOfAddresses}} lists of values:

$mergeValues(first: {{addresses}}, second: {{anotherListOfAddresses}})

Argument Description

first

The first list of values to merge.

second

The second list of values to merge.

$joinValues

Joins a list of values into a string.

Example

The following expression joins a list of values into a string:

$joinValues(values: {{values}}, separator: ',')

Argument Description

values

The list of values to join.

separator

The separator to put between values.

$splitValues

Splits a string into a list of values.

Example

The following expression splits a string into a list of values:

$splitValues(text: 'foo, bar, baz', separator: ',')

Argument Description

text

The string value to split.

separator

The separator that separates each value.

Join extensions

The join query extensions produce similar operations as the join keyword used in an SQL statement.

$valuesToResultSet

Converts a list of values to a result set matching the items where a given field is equal to a value in the list. The original order of the values in the result set is preserved using ranking expressions.

Example

The following query extension returns a result set of items for which the @sysworkemail field contains a value present in the {{addresses}} list of values:

$valuesToResultSet(values: {{addresses}}, field: '@sysworkemail')

Argument Description

values

The values to filter with.

field

The name of the field to use.

modifier

Optional. The modifier to use for the ranking expressions.

minimumModifier

Optional. The minimum modifier to use for the ranking expressions.

$joinOnValues

Filters a result set to only the items for which a given field is equal to one of the values contained in a list. The order of the values can optionally be preserved in the resulting result set when the modifier argument is specified. The extension returns a result set filtered on the field values. Any ranking expressions associated with the result set are kept.

Example

The following query extension returns only items for which the email domain name is listed in the specified domains alias:

$joinOnValues(resultSet: @uri, field: '@sfemaildomainnames', values: {domains})

Argument Description

resultSet

The result set to filter.

field

The name of the field to use.

values

The values to filter with.

modifier

Optional. The highest modifier to use for the ranking expressions.

minimumModifier

Optional. The minimum modifier to use for the ranking expressions.

$join

Filters a result set to only the items for which a given field is equal to a value of another field in another result set. If the toResultSet contains ranking expressions or sort orders, they will be included in the resulting result set. The order of the values can optionally be preserved in the resulting result set when the modifier argument is specified.

Example

The following query extension returns only accounts with won opportunities for which the opportunity amount sum is greater or equal to the specified amount alias:

$join(fromResultSet: $type(name: 'Opportunity') @sfopportunitystagename=won, toResultSet: $type(name: 'Account'), fromAggregateConditions: $sum(@sfopportunityamount)>={{amount}}, field: '@sfaccountid')

Argument Description

fromResultSet

The result set from which the list of values is extracted.

toResultSet

The result set to filter.

fromField

Required if field isn’t set. The name of the field to use in the fromResultSet.

toField

Required if field isn’t set. The name of the field to use in the toResultSet.

field

Required if fromField and toField aren’t set. The name of the field to use in both result sets. This is equivalent to specifying both fromField and toField with the same value.

fromAggregateConditions

Optional. The aggregate conditions to use to filter the from result set before performing the join.

maximumValues

Optional. The maximum number of values to filter on. By default, all values are used.

sortOrder

Optional. The sort criteria to use when retrieving values from the fromResultSet. The default value is donotsort.

modifier

Optional. The highest modifier to use for the ranking expressions.

minimumModifier

Optional. The minimum modifier to use for the ranking expressions.

$filterJoin

Filters a result set to only the items that can be joined in another result set using the specified fields. If the local result set contains ranking expressions or sort orders, they’re included in the resulting result set. This query extension basically performs two joins, one from the local to the external result set, and another back from the external to the local result set.

Example

The following query extension returns accounts with won opportunities for which the total amount is greater than $100,000:

$filterJoin(localResultSet: $type(name: 'Account'), externalResultSet: (@sfopportunitystagename=won), field: '@sfaccountid', aggregateConditions: $sum(@sfopportunityamount)>100000, maximumValues: 5)

Argument Description

localResultSet

The result set to filter.

externalResultSet

The result set to perform a join on to filter items in localResultSet.

localField

Required if field isn’t set. The name of the field to use in the localResultSet.

externalField

Required if field isn’t set. The name of the field to use in the externalResultSet.

field

Required if localField and externalField aren’t set. The name of the field to use in both result sets. This is equivalent to specifying both localField and externalField with the same value.

aggregateConditions

Optional. The aggregate conditions to use to filter the result set resulting from joining the local result set to the external one.

maximumValues

Optional. The maximum number of values to filter on. By default, all values are used.

Correlation extensions

$correlateListOfValues

Outputs a set of ranking expressions that boosts results having one of several values in a specified field.Arguments.

Argument Description

field

The name of the field to use.

values

The values to use for boosting.

modifier

The maximum modifier to apply when a result matches. items that match the first value in the list get the highest modifier, and those that match the last one get the lowest modifier.

useContainsOperator

Optional. Whether to match using the containment operator (=) rather than the identity operator (==), which is used by default.

useFullText

Optional. If true, query ranking expressions (QREs) will also be output for items that match the field values in full text. The default value is false.

forceOneMatch

Optional. If true, a query expression will be added that ensures that all results have a least one matching field value. The default value is false.

$correlateResultSet

Outputs a set of ranking expressions that boosts results sharing a field value from another result set.

Argument Description

resultSet

The result set from which to grab the list of values.

fromField

Required if field isn’t set. The name of the field to use in resultSet.

toField

Required if field isn’t set. The name of the field to use in the resulting result set.

field

Required if fromField and toField aren’t set. The name of the field to use in both result sets. This is equivalent to specifying both fromField and toField with the same value.

maximumValues

Optional. The maximum number of values to retrieve from the source result set. By default, all values are used.

sortOrder

Optional. The sort criteria to use when retrieving values from resultSet. The default value is chisquare.

modifier

The highest modifier to use for the {ranking expressions}. The maximum modifier to apply when a result matches. items that match the first value in the list get the highest modifier, and those that match the last one get the lowest modifier.

useFullText

Optional. If true, query ranking expressions (QREs) will also be output for items that match the field values in full text. The default value is false.

forceOneMatch

Optional. If true, a query expression will be added that ensures that all results have a least one matching field value. The default value is false.

$correlateUsingIdf

Outputs an expression that ranks up items sharing keywords with those specified. Uses the inverse document frequency (IDF) method to give a greater weight to keywords that are less frequent in the index.

Example

You may want to apply this extension when you need to make accessible items that aren’t frequently clicked within your index. For example, articles about less popular products.

Argument Description

keywords

The string from which to extract keywords.

forceOneMatch

Optional. If true, a query expression will be added that ensures that all results have at least one matching keyword. The default value is false.

removeStopWords

Optional. If true, any English stop word present in the keywords will be automatically removed. The default value is true.

Important

This argument is deprecated.

You should use the stop words query pipeline feature to filter out stop words from the query expression.

noStemming

Optional. If true, keywords won’t be expanded using the index stemming before matching items. The default value is false.

maximum

Optional. The maximum number of keywords to use. If a larger number of keywords is provided, some keywords will be completely ignored. The default value is 100.

$some

Extracts up to a specific part of "best" terms from a string, then retrieves items that match at least a specific part of those terms from the index.

Returns a result set.

Examples

  • Retrieve items containing at least the foo, bar, and baz terms:

    $some(keywords: 'foo bar baz')

  • Retrieve items containing at least the one "best" term among foo, bar, and baz:

    $some(keywords: 'foo bar baz', best: '1')

  • Retrieve items containing at least two terms among foo, bar, and baz:

    $some(keywords: 'foo bar baz', match: '50%')

  • Retrieve items containing at least one of the best two terms among foo, bar, and baz:

    $some(keywords: 'foo bar baz', best: '50%', 'match': '1')

  • Retrieve items containing at least the terms foo and bar:

    $some(keywords: 'foo bar baz', maximum: '2')

  • Retrieve items containing at least one of the best two terms among foo, bar, and baz:

    $some(keywords: 'foo bar baz hello', best: '50%', match: '50%', maximum: '3')

keywords (scalar [string])

The string from which to extract a subset of best terms.

Note

All of the original keywords terms (not only the "best" terms) are highlighted in matching items.

best (scalar [string | unsigned integer], optional)

The absolute or percentage (rounded up) maximum number of "best" terms to extract from keywords. Terms that occur less frequently in the index are considered "better" than more common keywords. Applies after maximum, but before match.

Sample values:

  • '5'

  • '75%'

Default value: '100%'

match (scalar [string | unsigned integer], optional)

The absolute or percentage (rounded up) minimum number of "best" terms an item must contain to be included in the result set. Applies after best.

Sample values:

  • '5'

  • '75%'

Default value: '100%'

removeStopWorlds (scalar [boolean], optional)

Important

This parameter is deprecated. It’s exposed for legacy compatibility reasons.

Use query pipeline statements expressing the stop query pipeline feature instead.

Whether to ignore standard English stop words (a, the, etc.) in the keywords.

Default value: 'false'

maximum (scalar [unsigned integer], optional)

The absolute maximum number of terms in keywords to consider when extracting terms (consider the string as a sequence). Excess keywords are ignored.

Default value: '100'

$removeStopWords

Removes the English stop words from a list of keywords.

Argument Description

keywords

The string from which to extract keywords.
Important

This extension is deprecated.

You should use the stop words query pipeline feature to filter out stop words from the query expression.

$noStemming

Arranges for a list of keywords to match without using stemming.

Argument Description

keywords

The string from which to extract keywords.