REST API source reference
REST API source reference
When creating a REST API source in the Coveo Administration Console, you must provide a JSON configuration detailing which services and endpoints to crawl to retrieve the desired content, and how to retrieve items of each type.
This configuration has a single property, Services, whose value must be an array of objects representing the endpoints to crawl.
This reference article defines parameters to include in your JSON configuration.
When working on your REST API source configuration, you may also want to refer to the following articles:
In addition, you may want to use the table of contents to quickly jump to the right property.
|
Use online tools such as JSONPath Online Evaluator to test your JSON paths, and freeformatter.com to validate and escape your JSON configuration. |
Example
The following is a basic example of a REST API source configuration. It instructs Coveo to call an endpoint to retrieve a list of users.
Basic example
If you expect your API to return the following response:
[
{ "id": 1, "name": "Caroline", "updated": "2023-11-14 11:29:49" },
{ "id": 2, "name": "Marcella", "updated": "2023-11-15 13:51:24" },
{ "id": 3, "name": "Susie", "updated": "2023-11-15 13:59:44" },
{ "id": 4, "name": "Rhonda", "updated": "2023-11-15 16:09:08" },
{ "id": 5, "name": "Wendy", "updated": "2023-11-16 17:02:35" }
{ "id": 6, "name": "Barbara", "updated": "2023-11-16 17:02:35" }
{ "id": 7, "name": "Deirdre", "updated": "2023-11-16 17:02:35" }
{ "id": 8, "name": "Lynda", "updated": "2023-11-16 17:02:35" }
]Then your JSON configuration could look like this:
{
"Services": [
{
"Url": "http://example.com/api/v1",
"Headers": {
"Authorization": "Bearer @ApiKey"
},
"Endpoints": [
{
"Path": "/users",
"Method": "GET",
"ItemType": "User",
"Uri": "%[coveo_url]/users/%[id]",
"ClickableUri": "%[coveo_url]/users/%[id]",
"Title": "%[name]",
"ModifiedDate": "%[updated]",
"Metadata": {
"id": "%[id]",
"username": "%[name]"
}
}
]
}
]
}For additional, more complex examples, see JSON configuration examples. You can also use the table of contents to jump to a specific parameter.
Table of contents
To help you jump to the right section, here’s a hierarchical and alphabetical list of all supported arrays, objects, and parameters. Alternatively, click one of the main topics and capabilities below.
Show table of contents
| Hierarchical list | Alphabetical list |
|---|---|
|
|
Show main topics and capabilities
Services (array, required)
Each object in the Services array specifies a REST API service to retrieve content, along with the specific endpoints to call.
The properties that may appear in an object of the Services array are the following:
Endpoints (array, required)
Services > Endpoints
Each object in the Endpoints array details how you want Coveo to crawl an endpoint of your web application.
Moreover, in each endpoint object, the Metadata objects lists the content metadata to retrieve.
Objects of the Endpoints array support the following properties from their parent service object:
It also supports the following properties:
-
ClickableUri(required) -
ItemType(required) -
Path(required) -
Uri(required)
ClickableUri (string, required)
The ClickableUri is the URI of the page to open when a user clicks the title of an item from your REST API source in a Coveo-powered search interface.
It can be identical to the Uri.
Dynamic values are supported.
Example: "ClickableUri": "%[coveo_url]/posts/%[id]"
ItemType (string, required)
The type of item that the resource contains. Coveo uses this data to organize the indexed content.
Examples:
-
"ItemType": "Post"for a blog post -
"ItemType": "PostComment"for a comment on a post
Path (string, required)
The relative path to the desired resource endpoint.
Dynamic values are supported when specified within the SubItems, SubQueries, and PermissionSubQueries arrays only.
Dynamic time expressions are also supported.
Example: "Path": "orgs/myorg/repos" retrieves your GitHub repositories
In sub-items and subqueries, should a dynamic value be invalid or unretrievable, the source skips the item, that is, the indexing process is dropped and Coveo moves on to the next item. In a permission subquery, however, an invalid or unretrievable dynamic value causes the source to stop crawling and to display an error in the Administration Console.
Uri (string, required)
The Uri is an item’s unique identifier in Coveo.
It can be identical to the ClickableUri.
If your items aren’t available through a URI, you must still index a value that is different for each item.
Dynamic values are supported.
Example: "Uri": "%[coveo_url]/repository/posts/%[id]"
Should a dynamic value be invalid or unretrievable, the source stops crawling and displays an error in the Administration Console.
Body (string)
The path (simple path or JSONPath) to the Body content of the item in the JSON response.
The item body is displayed in the Content Browser.
Typically, the Body property contains a dynamic value, since a static, hardcoded value would make all item bodies identical.
Dynamic values and hardcoded text can also be mixed.
Example: "Body": "%[html.content]"
DateFormat (string)
The format of a date injected thanks to the tokens @Now and @RefreshDate.
All .NET formats are supported, and the default is yyyy-MM-ddTHH:mm:ssZ (ISO 8601).
Alternatively, to handle Unix timestamps, enter either of the following:
-
UnixEpochfor the number of seconds elapsed since the Unix epoch. -
UnixEpochMsfor the number of milliseconds elapsed since the Unix epoch.
Coveo also uses this format when parsing dates coming from your API.
If these dates aren’t in a .NET format, use DateFormat to specify the format to expect.
Examples:
-
"DateFormat": "yyyy-MM-ddTHH:mm:ssZ" -
"DateFormat": "\\'yyyy-MM-dd\\',\\'hh:mm:ss\\'"
IndexingAction (object)
Whether to ignore or to retrieve an item when a condition is met.
When the Condition resolves to true, the Coveo crawler applies the specified action to the item.
Possible actions are Retrieve and Ignore, that is, the crawler can either index the item or ignore it.
An item ignored with IndexingAction isn’t indexed and therefore not visible in the Content Browser or a search interface.
However, its sub-items are sent to the index, unless otherwise specified in the ProcessingAction object.
If the condition resolves to false, the Coveo crawler falls back to the default action, which is Retrieve, and the specified item is indexed.
Example:
With the following IndexingAction object, all items with an id greater than 5 are ignored.
Only items with an id of 5 or less are indexed.
"IndexingAction": {
"ActionOnItem": "Ignore",
"Condition": "%[id] > 5"
}The IndexingAction object supports the following properties:
-
ActionOnItem(required)
ActionOnItem (string, required)
Services > Endpoints > IndexingAction > ActionOnItem
Action to perform if the specified Condition resolves to true.
In the IndexingAction object, this applies to retrieved items.
Possible actions are Retrieve and Ignore, that is, the crawler can either index the item to make it searchable or ignore it.
In the ProcessingAction object, this applies to child items of an item.
Possible actions are Process and Ignore, that is, the crawler can either retrieve the subitems to make them searchable or ignore them.
Action Condition (string)
Services > Endpoints > IndexingAction > Condition
The Condition value is a condition that must resolve to true for the specified action to apply to the child items of an item.
To write your condition, use the same syntax as for indexing pipeline extension conditions.
Conditions can be assembled using the following operators: AND, OR, Exists, NOT, ==, >, and <.
Parentheses are also supported to specify operation order.
|
The |
The metadata fields in your condition must either be defined in the Metadata object or referenced with raw.
You can specify an array if the metadata refers to a multi-value field in your repository.
Examples:
-
"%[author_id]"is true if the item has anauthor_id. -
"NOT %[author_id]"is true if the item doesn’t have anauthor_id. -
"%[author_id] == 1234"is true if the itemauthor_idis1234. -
"%[author_ids] == [1,2,3,4]"is true if itemauthor_idsare1,2,3, and4. -
"%[author_id] OR %[author_name]"is true if the item has anauthor_idor anauthor_name. -
"%[author_id] AND %[author_name]"is true if the item has anauthor_idand anauthor_name. -
"%[author_id] > 123"is true if the itemauthor_idis greater than 123. -
"(%[author_id] OR %[author_sys_id]) AND %[author_name]"is true if the item has anauthor_idor anauthor_name, as well as anauthor_name.
IsBinaryBody (Boolean)
Whether the subquery retrieves binary content as the body of the item.
Default value is false.
This property is available within SubQueries objects only.
It allows you to download a file (for example, XLS or PDF document) to use as the item body in a search result Quick view.
See Supported file formats for an exhaustive list of item types that Coveo can index.
Example:
"SubQueries": [
{
"Path":"%[coveo_parent.url]",
"Method":"GET",
"IsBinaryBody":true,
"Body": "%[data.body]"
}
]
|
Notes
|
ItemPath (string)
To index items from your application’s JSON response, Coveo needs to know where to find the items in the response.
If the items aren’t at the root level of the JSON response, you must specify the path to the items using the ItemPath property, with JSONPath syntax.
By default, Coveo assumes that the items to index are at the root level of the JSON response.
In such case, you can omit ItemPath.
|
|
Note
|
Example:
If your API’s response looks like this:
{
"count": 8,
"entries": [
{
"results": [
{ "id": 1, "name": "Caroline" },
{ "id": 2, "name": "Marcella" },
{ "id": 3, "name": "Susie" },
{ "id": 4, "name": "Rhonda" },
{ "id": 5, "name": "Wendy" },
{ "id": 6, "name": "Barbara Ann" },
{ "id": 7, "name": "Deirdre" },
{ "id": 8, "name": "Lynda" }
]
}
]
}Then your source configuration should contain the following:
"ItemPath": "entries[0].results"This tells Coveo to index the results array of the first entries object in the JSON response.
Metadata (object)
A key-value list of the metadata to parse.
Each key represents the metadata name of the item, while its value is the value path (simple path or JSONPath) in the JSON response.
Typically, the value path consists of one or more dynamic values, since a static, hardcoded value would result in an identical piece of metadata for all items. However, you could choose to use a hardcoded value so that the corresponding Coveo field is filled even if the API doesn’t provide this information.
When you build your source, Coveo retrieves the desired metadata. You can then review a summary of this metadata in the Administration Console and use it to create mapping rules for your source.
Example
If your API returns a response with a single item that looks like this:
{
"id": 1,
"firstname": "Lloyd",
"lastname": "Sweeney",
"age": 26,
"address": {
"streetAddress": "6974 Rutrum Dr",
"city": "Ashburn",
"state": "VA",
"zip": "25581"
},
"email": "JWeakliem@lectus.org",
"username": "GHohmann",
"phone": "9556676434",
"created": "2023-09-12T05:29:48.688Z"
}Then your source configuration could look like this:
{
"Services": [
{
"Url": "http://example.com/api/v1",
"Endpoints": [
{
"Path": "/users/1",
"Method": "GET",
"ItemType": "User",
"Uri": "%[coveo_url]/users/%[id]",
"ClickableUri": "%[coveo_url]/users/%[id]",
"Title": "%[username]",
"ModifiedDate": "%[created]",
"Metadata": {
"id": "%[id]",
"title": "Mr.", 
"firstname": "%[firstname]",
"lastname": "%[lastname]",
"age": "%[age]",
"address": "%[address]", 
"email": "%[email]",
"phonenumber": "%[phone]"
}
}
]
}
]
} |
The title metadata is hardcoded to "Mr." for all items. |
|
In the configuration above, the address metadata contains the whole JSON object.
So, to flatten the metadata, you could also write the following Metadata object: |
{
"Metadata": {
"id": "%[id]",
"firstname": "%[firstname]",
"lastname": "%[lastname]",
"age": "%[age]",
"address.street": "%[address.streetAddress]",
"address.city": "%[address.city]",
"address.state": "%[address.state]",
"address.zip": "%[address.zip]",
"email": "%[email]",
"phonenumber": "%[phone]"
}
}
Method (string enum)
The HTTP method to use to fetch the resource.
Possible values are GET and POST.
Default value is GET.
This property is not inheritable.
See the API’s documentation to determine which one you should use.
Example: "Method": "POST"
ModifiedDate (string)
The date on which the item was last modified. Although this property isn’t required in the JSON configuration, we recommend that you include it, as the Coveo-powered search pages use it for their default Sort by date option.
Typically, ModifiedDate has a dynamic value, since a static, hardcoded value would make dates identical for all items.
Example: "ModifiedDate": "%[updated]"
PayloadJsonContent (string)
The JSON content to send as a POST request body.
The Content-Type header is automatically set to application/json.
Dynamic values are supported when specified within the SubItems and SubQueries objects only.
In sub-items and subqueries, should a dynamic value be invalid or unretrievable, the source skips the item, that is, the indexing process is dropped and Coveo moves on to the next item. In a permission subquery, however, an invalid or unretrievable dynamic value causes the source to stop crawling and to display an error in the Administration Console.
|
|
Notes
|
You want to provide the following JSON content with your request:
{
"key1": "value1",
"key2": "@field=(\"value2\")",
"key3":
{
"subkey1": "subvalue1"
}
}Your source JSON configuration therefore contains the following:
"Endpoints": [
{
"Method": "POST",
"Path": "/rest/myendpoint/",
"PayloadJsonContent": "{\"key1\": \"value1\", \"key2\": \"@field=(\\\"value2\\\")\", \"key3\": {\"subkey1\": \"subvalue1\"}}"
}
]
PayloadParameters (object)
Endpoints > PayloadParameters
A key-value list of HTTP parameters to add to the payload.
Each value can be either a number, string, or Boolean.
This property can only be used in a POST request.
The parameters are sent as URL-encoded data in the request body.
The Content-Type header is automatically set to application/x-www-form-urlencoded.
Dynamic values are supported when specified within the SubItems and SubQueries objects only.
Dynamic time expressions are also supported.
|
|
Notes
|
Examples:
Basic example
"PayloadParameters": {
"type": "post",
"expand": "true",
"id": 120
}Authentication with an API key as a payload parameter
If your API requires Coveo to authenticate with an API key as the value of the api_key payload parameter, you must enter this key in the Authentication section of the Add a REST API source panel.
Then, in your source JSON configuration, you must use the @ApiKey placeholder to refer to this key.
This ensures that your API key isn’t stored in clear text in the source JSON configuration, where other Administration Console users could see it.
When Coveo processes the source configuration, it replaces the placeholder with the actual key you entered in the panel.
Your source configuration could therefore look like this.
{
"Services": [
{
"Url": "http://example.com/api/v1",
"Endpoints": [
{
"Path": "/users",
"Method": "GET",
"ItemType": "User",
"Uri": "%[coveo_url]/users/%[id]",
"ClickableUri": "%[coveo_url]/users/%[id]",
"PayloadParameters": {
"api_key": "@ApiKey"
}
}
]
}
]
} |
api_key is the name of the payload parameter that your API expects to contain the API key. |
As a result, Coveo sends the following HTTP request to the API:
POST /users HTTP/1.1
Host: http://example.com/api/v1
api_key=Az4adfSyBhZX8fTThUJ4Kb2TC3Ax0RshgrZtUiqM |
Az4adfSyBhZX8fTThUJ4Kb2TC3Ax0RshgrZtUiqM is the API key you entered in the source panel. |
PermanentId (string)
The PermanentId is an alternative, permanent unique item identifier.
When you expect the item Uri to change, this ensures that Coveo Machine Learning (Coveo ML) recognizes the items despite their new URI and associates them to the old data.
So, in the future, if you modify or delete this source and reindex the same items, specifying the same item PermanentId format in your next configuration will ensure that ML doesn’t consider your items as new and rather picks up where it left off.
Dynamic values are supported.
Example:
{
"Uri": "%[coveo_url]/items/%[sys_id]",
"PermanentId": "%[sys_id]"
}
ProcessingAction (object)
This object works similarly to the IndexingAction object.
When the Condition resolves to true, the Coveo crawler applies the specified action on the corresponding child items of an item.
Possible actions are Process and Ignore, that is, the crawler can either index the subitems or ignore them.
A child item ignored with ProcessingAction isn’t indexed and therefore not visible in the Content Browser or a search interface.
If the condition resolves to false, it fallbacks to the default action, that is, Process.
Example:
"ProcessingAction": {
"ActionOnItem": "Ignore",
"Condition": "%[id]==1"
}Like the IndexingAction object, the ProcessingAction object supports the following properties:
-
ActionOnItem(required)
QueryParameters (object)
A key-value list of HTTP parameters to add to the query.
Each value can be either a number, string, or Boolean.
Dynamic values are supported when specified within the SubItems, SubQueries, and PermissionSubQueries arrays only.
They aren’t supported in QueryParameters objects of a refresh endpoint.
|
|
Note
In sub-items and subqueries, should a dynamic value be invalid or unretrievable, the source skips the item, that is, the indexing process is dropped and Coveo moves on to the next item. In a permission subquery, however, an invalid or unretrievable dynamic value causes the source to stop crawling and to display an error in the Administration Console. |
Dynamic time expressions are also supported.
Examples:
Basic example
"QueryParameters": {
"type": "post",
"expand": "true",
"id": 120
}Dynamic time expressions
"QueryParameters": {
"since": "@Now-6M",
"until": "@Now"
}Authentication with an API key as a query parameter
If your API requires Coveo to authenticate with an API key as the value of the api_key query parameter, you must enter this key in the Authentication section of the Add a REST API source panel.
Then, in your source JSON configuration, you must use the @ApiKey placeholder to refer to this key.
This ensures that your API key isn’t stored in clear text in the source JSON configuration, where other Administration Console users could see it.
When Coveo processes the source configuration, it replaces the placeholder with the actual key you entered in the panel.
Your source configuration could therefore look like this.
{
"Services": [
{
"Url": "http://example.com/api/v1",
"Endpoints": [
{
"Path": "/users",
"Method": "GET",
"ItemType": "User",
"Uri": "%[coveo_url]/users/%[id]",
"ClickableUri": "%[coveo_url]/users/%[id]",
"QueryParameters": {
"api_key": "@ApiKey"
}
}
]
}
]
} |
api_key is the name of the query parameter that your API expects to contain the API key. |
As a result, Coveo sends the following HTTP request to the API:
GET /users?api_key=Az4adfSyBhZX8fTThUJ4Kb2TC3Ax0RshgrZtUiqM HTTP/1.1
Host: http://example.com/api/v1Az4adfSyBhZX8fTThUJ4Kb2TC3Ax0RshgrZtUiqM is the API key you entered in the source panel.
RefreshEndpoints (array)
The RefreshEndpoints array is required for your source to perform refresh operations, which index only the items that have been added, updated, or deleted since the last update operation.
Refreshes are typically faster than rescan and rebuild operations, as they only process the items that have changed.
For more information on source update operations, see Refresh, rescan, and rebuild.
Coveo recommends you configure a RefreshEndpoints array for each endpoint in your source configuration.
Each object in the RefreshEndpoints array represents a query that Coveo will make to your API when conducting a source refresh operation.
By default, a refresh object inherits the properties defined in its parent endpoint object. So, in a refresh query, you only need to specify what should be different from the original indexing query.
Then, when you refresh your source, the properties defined in an object of the RefreshEndpoints array override the properties defined under Endpoints.
When you rescan or rebuild the source, however, the Endpoints properties apply.
Example
When refreshing your source, Coveo indexes only the published items that have been updated since the last refresh operation, as instructed by the following RefreshEndpoints array.
Since no other property is specified, Coveo will index these items exactly like during the initial indexing operation.
"Endpoints": [
{
"Path": "/api/now/table/kb_knowledge",
"Method": "GET",
"ItemPath": "result",
"ItemType": "kbknowledge",
"Uri": "%[coveo_url]/kb_knowledge/%[sys_id]",
"PermanentId": "%[sys_id]",
"ModifiedDate": "%[sys_updated_on]",
"ClickableUri": "%[coveo_url]/nav_to.do?uri=kb_knowledge.do?sys_id=%[sys_id]%26sysparam_view-ess",
"Title": "%[short_description]",
"Body": "%[text]",
"QueryParameters": {
"sysparm_query": "workflow_state=published"
},
"Metadata": {
"short_description": "%[short_description]",
"number": "%[number]",
"workflow_state": "%[workflow_state]",
},
"RefreshEndpoints": [
{
"DateFormat": "\\'yyyy-MM-dd\\',\\'hh:mm:ss\\'",
"QueryParameters": {
"sysparm_query": "workflow_state=published^sys_updated_on>javascript:gs.dateGenerate(@RefreshDate)"
}
}
]
}
] |
The DateFormat property specifies the format of the date value in the query parameters. |
Inheritance of QueryParameters works differently from other properties found in objects of the RefreshEndpoints array.
If you don’t specify any query parameter in your refresh query, the query parameters of the parent query will be inherited.
However, either all query parameters are inherited, or none is.
So, if you change one parameter, you must also redefine all other parameters that apply, even if they’re identical to those in the parent query object.
In addition, dynamic values aren’t supported in the query parameters of a RefreshEndpoints object.
|
|
Note
A source refresh operation will index a change in a sub-item only if its parent item has also changed. |
Objects of the RefreshEndpoints array support the properties of Endpoints objects, plus the following:
Example:
"RefreshEndpoints": [
{
"DateFormat": "\\'yyyy-MM-dd\\',\\'hh:mm:ss\\'",
"QueryParameters": {
"workflow_state": "published",
"since": "@RefreshDate"
}
},
{
"IsDeletionQuery": true,
"DateFormat": "\\'yyyy-MM-dd\\',\\'hh:mm:ss\\'",
"QueryParameters": {
"lastModifiedDate": "@Now-7d",
"status": "Deleted"
}
},
{
"IsDeletionQuery":true,
"DateFormat": "\\'yyyy-MM-dd\\',\\'hh:mm:ss\\'",
"Path": "/api/table/audit_delete",
"QueryParameters":{
"tablename": "kb_knowledge",
"since": "@RefreshDate"
}
},
{
"IsDeletedItem": "%[is_deleted]",
"DateFormat":"\\'yyyy-MM-dd\\',\\'hh:mm:ss\\'",
"QueryParameters":{
"since": "@RefreshDate"
}
}
]
IsDeletionQuery (Boolean)
Services > Endpoints > RefreshEndpoints > IsDeletionQuery
Whether the defined query returns items to delete from the index.
Default value is false, which means that all retrieved items are added to the index or updated.
If the query returns items to add, update, and delete all at once, use the IsDeletedItem property instead.
IsDeletedItem (Boolean)
Services > Endpoints > RefreshEndpoints > IsDeletedItem
When a refresh query returns items to add, update, and delete all at once, this property allows you to specify which of these items should be deleted.
Default value is false, but it’s considered to be true if IsDeletionQuery is set to true.
You can provide a dynamic value (for example, %[has_been_deleted_flag]) to be evaluated for each item or a static value.
Example:
"%[is_deleted]" deletes all items that have a "is_deleted": true JSON property flagging them as deleted.
{
"IsDeletedItem": "%[is_deleted]",
"DateFormat":"\\'yyyy-MM-dd\\',\\'hh:mm:ss\\'",
"QueryParameters":{
"since": "@RefreshDate"
}
}
DeleteChildren (Boolean)
Services > Endpoints > RefreshEndpoints > DeleteChildren
Whether to delete children when the retrieved parent item is deleted.
Default value is false.
If the value is set to true, child items are deleted based on their Uri.
SubItems (array)
A list of sub-items to retrieve.
Each sub-item is represented by a configuration object under Endpoints.
An item can have many different sub-items, for example, a blog post containing comments and attachments as child items.
The SubItems object establishes a parent-child relationship between items.
There’s no restriction on the number of sub-items an item can have.
In addition, sub-items can also have sub-items, for example if blog post comments also have attachments.
In the application JSON response, these sub-items are nested within each other, and so should be your SubItems arrays in your source configuration.
To refer to the metadata of a parent item, prefix the metadata name with coveo_parent.
You may also need to use raw.
Examples:
When the sub-items to index are located under a different endpoint, you provide the Path to their location.
"SubItems": [
{
"Path": "/posts/%[coveo_parent.id]/comments",
"Method": "GET",
"ItemType": "Comment",
"Uri": "%[coveo_url]/repository/posts/%[coveo_parent.id]/comments/%[id]",
"ClickableUri": "%[coveo_url]/posts/%[coveo_parent.id]/comments/%[id]",
"Title": "%[title]",
"ModifiedDate": "%[updated]",
"Body": "%[content]",
"Metadata": {
"author": "%[author]",
"createdby" :"%[coveo_parent.author]",
"id": "%[id]"
}
}
]Alternatively, when the desired sub-items are located under the same endpoint as their parent items (for example, the child item data is nested within the parent item data), you omit the Path property and provide the sub-items' location using the ItemPath property.
"SubItems": [
{
"ItemPath": "/comments",
"ItemType": "Comment",
"Uri": "%[coveo_url]/repository/posts/%[coveo_parent.id]/comments/%[id]",
"ClickableUri": "%[coveo_url]/posts/%[coveo_parent.id]/comments/%[id]",
"Title": "%[title]",
"ModifiedDate": "%[updated]",
"Body": "%[content]",
"Metadata": {
"author": "%[author]",
"createdby" :"%[coveo_parent.author]",
"id": "%[id]"
}
}
]
SubQueries (array)
Each object in the SubQueries array represents a subquery to execute on every item in order to fetch additional metadata and/or the item body.
Subqueries are especially useful when this information is located under a different path than that of the item.
To refer to the metadata of a parent item, prefix the metadata name with coveo_parent.
You may also need to use raw.
Example:
"SubQueries": [
{
"Path": "/posts/%[coveo_parent.id]/additionalinfo",
"Method": "GET",
"Body": "%[data.body]",
"Metadata": {
"created": "%[created]"
}
}
]The subquery configuration supports the following properties from the Endpoints object configuration:
In addition, it supports the following property from the Services object configuration:
It also contains the following property:
IsThumbnail (Boolean)
Services > Endpoints > SubQueries > IsThumbnail
If the SubQueries object path leads to an image, set IsThumbnail to true to show this image as the thumbnail of an item.
The default value is false.
Supported thumbnail file formats are the following:
.bmp, .emf, .exif, .gif, .icon, .jpeg, .png, .tiff, .wmf
.
|
|
Notes
|
Title (string)
The title of the item. Although this property isn’t required in the JSON configuration, we recommend that you include it, as the Content Browser displays its value.
Typically, Title contains a dynamic value, since a static, hardcoded value would make all item titles identical.
Example: "Title": "%[title]"
Url (string, required)
Services > Url
The Url value is the service URL of your web application.
Example: "Url": "http://example.com/api/v1"
Authentication (object)
Services > Authentication
If your API service requires Coveo to authenticate, you must enter the necessary credentials in the Authentication section of the Add a REST API source panel.
Then, in your source JSON configuration, you must specify how Coveo should use these credentials to authenticate with the API.
If your API uses password-based or OAuth 2.0 authentication, you must use the Authentication object, which contains the information Coveo will use to authenticate to the service.
What if my API uses API keys?
If your API uses API keys for authentication, Coveo will typically need to provide such a key in an HTTP header, as a query parameter, or as a payload parameter.
You therefore don’t need an Authentication object in your source JSON configuration.
In the Authentication object, you’ll use a placeholder starting with @ to refer to the credentials you entered in the panel.
This ensures that your credentials aren’t stored in clear text in the source JSON configuration, where other Administration Console users could see them.
Then, when Coveo processes the source configuration, it replaces the placeholders with the actual credentials you entered in the panel.
The following table shows the objects and placeholders you must include in your source configuration, depending on the type of authentication your API uses:
| Authentication type | Use placeholders in | Placeholders to use |
|---|---|---|
Basic, HTTP, Kerberos, or NTLM (password-based) |
|
|
OAuth 2.0 |
|
|
API key |
|
|
As indicated in the table above, API key placeholders don’t appear in the Authentication object.
If your API authenticates with API keys, see the sections listed in the table for examples with API key placeholders.
Authentication properties specified at the service level apply to the child endpoints, and sub-queries underneath them.
Typically, you don’t need to override the configuration you entered at the service level further down in the JSON configuration, as the same authentication method applies to the entire application content.
Examples:
Basic, HTTP, Kerberos, or NTLM
If your API uses password-based authentication, you must enter the username and password in the Authentication section of the Add a REST API source panel.
Then, in your source JSON configuration, you must use the @Username and @Password placeholders to refer to these credentials.
Your Authentication object therefore looks like this:
"Authentication": {
"Username": "@Username",
"Password": "@Password",
"ForceBasicAuthentication": "true"
}OAuth 2.0
If your API uses OAuth 2.0 authentication, you must enter the client ID, client secret, and refresh token in the Authentication section of the Add a REST API source panel.
Then, in your source JSON configuration, you must use the @ClientId, @ClientSecret, and @RefreshToken placeholders to refer to this information.
Your Authentication object could therefore look like this:
"Authentication": {
"OAuth": {
"Query": {
"RefreshUrl": "http://example.com/token",
"Method": "POST",
"Parameters": {
"grant_type": {
"Type": "Payload",
"Value": "refresh_token"
},
"client_id": {
"Type": "Payload",
"Value": "@ClientId"
},
"client_secret": {
"Type": "Payload",
"Value": "@ClientSecret"
},
"refresh_token": {
"Type": "Payload",
"Value": "@RefreshToken",
"IsRefreshToken": true
}
}
}
}
}The Authentication object supports the following properties:
-
OAuth(required if authenticating with OAuth 2.0)
Username (string)
Services > Authentication > Username
Username used to log in to the service.
You can use @Username to retrieve the value specified in the Add a REST API Source panel.
Password (string)
Services > Authentication > Password
Password used to log in to the service.
This property can be empty even though a Username is specified.
You can use @Password to retrieve the value specified in the Add a REST API Source panel.
Domain (string)
Services > Authentication > Domain
Domain name to use when authenticating to a NTLM or Kerberos protected server.
ForceBasicAuthentication (Boolean)
Services > Authentication > ForceBasicAuthentication
Whether to force a basic HTTP header in the request.
The default value is false.
OAuth (object)
Services > Authentication > OAuth
If your application requires OAuth 2.0 authentication, add an OAuth object to your source configuration.
Your OAuth object must contain a Query object.
Coveo will use the information in the Query object to make an authentication query to your application.
Optionally, your OAuth object may also contain a Response object.
In this object, you may include properties such as AccessToken, TokenType, and ExpiresIn, whose values indicate where Coveo will find the corresponding information when parsing the API’s response.
The Response object may also include the AuthorizationHeader property, whose value is the name of the authorization header that Coveo should use when querying your API for content.
|
|
Enter your client ID, client secret, and refresh token in the Add a REST API source panel and use the This allows you to keep your confidential information obfuscated, thus preventing other Administration Console users to access it. 
|
Example:
"OAuth": {
"Query": {
"RefreshUrl": "http://example.com/token",
"Method": "POST",
"Parameters": {
"grant_type": {
"Type": "Payload",
"Value": "refresh_token"
},
"refresh_token": {
"Type": "Payload",
"Value": "@RefreshToken",
"IsRefreshToken": true
},
"client_id": {
"Type": "Payload",
"Value": "@ClientId"
},
"client_secret": {
"Type": "Payload",
"Value": "@ClientSecret"
}
}
},
"Response": {
"AccessToken": "access_token",
"SupportsRefreshToken" : "true",
"RefreshToken": "refresh_token",
"ExpiresIn": "expires_in",
"TokenType": "token_type"
}
}The OAuth object supports the following properties:
-
Query(required)
Query (object, required)
Services > Authentication > OAuth > Query
If your application requires OAuth 2.0 authentication, you must add an OAuth object to your source configuration.
Your OAuth object must contain a Query object.
Coveo will use the information in the Query object to make an authentication query to your application.
|
|
Enter your client ID, client secret, and refresh token in the Add a REST API source panel and use the This allows you to keep your confidential information obfuscated, thus preventing other Administration Console users to access it.
|
Example:
"Query": {
"RefreshUrl": "http://example.com/token",
"Method": "POST",
"Parameters": {
"grant_type": {
"Type": "Payload",
"Value": "refresh_token"
},
"refresh_token": {
"Type": "Payload",
"Value": "@RefreshToken",
"IsRefreshToken": true
},
"client_id": {
"Type": "Payload",
"Value": "@ClientId"
},
"client_secret": {
"Type": "Payload",
"Value": "@ClientSecret"
}
}
}The Query object supports the following properties:
-
RefreshUrl(required)
RefreshUrl (string, required)
Services > Authentication > OAuth > Query > RefreshUrl
If your application requires OAuth 2.0 authentication, your Query object must include a RefreshUrl property.
The RefreshUrl value must be a URL that Coveo will use to refresh its access token.
Example
"Query": {
"RefreshUrl": "http://example.com/token",
"Method": "POST",
"Parameters": {
"grant_type": {
"Type": "Payload",
"Value": "refresh_token"
},
"refresh_token": {
"Type": "Payload",
"Value": "@RefreshToken",
"IsRefreshToken": true
},
"client_id": {
"Type": "Payload",
"Value": "@ClientId"
},
"client_secret": {
"Type": "Payload",
"Value": "@ClientSecret"
}
}
}OAuth query Headers (object)
Services > Authentication > OAuth > Query > Headers
If your application requires OAuth 2.0 authentication, your Query object can optionally include a Headers object.
When making an authentication query to your application, Coveo will add these HTTP headers to its request.
|
|
Note
Coveo automatically sets the Content-Type header.
Therefore, you don’t need to add it to the |
Example:
"Query": {
"RefreshUrl": "http://example.com/token",
"Headers": {
"X-Auth": "apikey1234567890abcdef"
}
}OAuth query Method (string enum)
Services > Authentication > OAuth > Query > Method
If your application requires OAuth 2.0 authentication, your Query object can optionally include a Method property.
The Method property specifies the HTTP method that Coveo should use when refreshing its token.
Default is POST.
Example
"Query": {
"RefreshUrl": "http://example.com/token",
"Method": "POST",
"Parameters": {
"grant_type": {
"Type": "Payload",
"Value": "refresh_token"
},
"refresh_token": {
"Type": "Payload",
"Value": "@RefreshToken",
"IsRefreshToken": true
},
"client_id": {
"Type": "Payload",
"Value": "@ClientId"
},
"client_secret": {
"Type": "Payload",
"Value": "@ClientSecret"
}
}
}OAuth query Parameters (object)
Services > Authentication > OAuth > Query > Parameters
If your application requires OAuth 2.0 authentication, your Query object can optionally include query parameters for Coveo to use.
Each object in the Parameters object represents a parameter to include in Coveo’s authentication request to your application.
Each object should contain a "Type" and a "Value" property.
The Type property indicates how to include the parameter in the request.
Possible values for this property are query or payload.
If you include a refresh_token object, you must add a IsRefreshToken property with a value of true to this object.
Moreover, if this refresh token is single-use, make sure to specify it by adding "UsingSingleUseRefreshToken": true to your OAuth object.
|
|
Enter your client ID, client secret, and refresh token in the Add a REST API source panel and use the This allows you to keep your confidential information obfuscated, thus preventing other Administration Console users to access it.
|
Example:
"Authentication": {
"OAuth": {
"UsingSingleUseRefreshToken": true,
"Query": {
"RefreshUrl": "https://auth.example.com/oauth/token",
"Method": "POST",
"Parameters": {
"grant_type": {
"Type": "Payload",
"Value": "refresh_token"
},
"refresh_token": {
"Type": "Payload",
"Value": "@RefreshToken",
"IsRefreshToken": true
},
"client_id": {
"Type": "Payload",
"Value": "@ClientId"
},
"client_secret": {
"Type": "Payload",
"Value": "@ClientSecret"
}
}
}
}
}
Response (object)
Services > Authentication > OAuth > Response
If your application requires OAuth 2.0 authentication, you can optionally add a Response object to your OAuth object.
The Response object contains, among other things, the properties that Coveo must extract from the authentication server’s response.
The value of each property represents the path to the corresponding value in the response.
In the Response object, you may also enter other authentication parameters, such as the AuthorizationHeader and SupportsRefreshToken properties.
Example:
If the authentication server’s response looks like this:
HTTP/1.1 200 OK
Content-Type: application/json{
"access_token": "jOs0PoLKm61sd6h49",
"refresh_token": "QpMas-tuI9bvFp01-Xxpl04gC_0m",
"token_type": "bearer",
"expires_in": 86400
}Then your Response object should look like this:
"Response": {
"AccessToken": "access_token",
"SupportsRefreshToken" : "true",
"RefreshToken": "refresh_token",
"ExpiresIn": "expires_in",
"TokenType": "token_type"
}The Response object supports the following properties:
AccessToken (string)
Services > Authentication > OAuth > Response > AccessToken
If your application requires OAuth 2.0 authentication, you can optionally add a Response object to your OAuth object.
Coveo will then extract the information from the API’s response using the values specified in the Response object.
In the Response object, you can add the AccessToken property to specify the path to the access token in the API’s response.
Default is access_token.
Example
If the authentication server’s response looks like this:
HTTP/1.1 200 OK
Content-Type: application/json{
"access_token": "jOs0PoLKm61sd6h49",
"refresh_token": "QpMas-tuI9bvFp01-Xxpl04gC_0m",
"token_type": "bearer",
"expires_in": 86400
}Then your Response object should look like this:
"Response": {
"AccessToken": "access_token",
"SupportsRefreshToken" : "true",
"RefreshToken": "refresh_token",
"ExpiresIn": "expires_in",
"TokenType": "token_type"
}
SupportsRefreshToken (Boolean)
Services > Authentication > OAuth > Response > SupportsRefreshToken
If your application requires OAuth 2.0 authentication, you can optionally add a Response object to your OAuth object.
In the Response object, you can add the SupportsRefreshToken property to indicate whether your application’s OAuth 2.0 flow supports refresh tokens.
Default is true.
Example
If the authentication server’s response looks like this:
HTTP/1.1 200 OK
Content-Type: application/json{
"access_token": "jOs0PoLKm61sd6h49",
"refresh_token": "QpMas-tuI9bvFp01-Xxpl04gC_0m",
"token_type": "bearer",
"expires_in": 86400
}Then your Response object should look like this:
"Response": {
"AccessToken": "access_token",
"SupportsRefreshToken" : "true",
"RefreshToken": "refresh_token",
"ExpiresIn": "expires_in",
"TokenType": "token_type"
}
RefreshToken (string)
Services > Authentication > OAuth > Response > RefreshToken
If your application requires OAuth 2.0 authentication, you can optionally add a Response object to your OAuth object.
Coveo will then extract the information from the API’s response using the values specified in the Response object.
In the Response object, you can add the AccessToken property to specify the path to the refresh token in the API’s response.
Default is refresh_token.
Example
If the authentication server’s response looks like this:
HTTP/1.1 200 OK
Content-Type: application/json{
"access_token": "jOs0PoLKm61sd6h49",
"refresh_token": "QpMas-tuI9bvFp01-Xxpl04gC_0m",
"token_type": "bearer",
"expires_in": 86400
}Then your Response object should look like this:
"Response": {
"AccessToken": "access_token",
"SupportsRefreshToken" : "true",
"RefreshToken": "refresh_token",
"ExpiresIn": "expires_in",
"TokenType": "token_type"
}
ExpiresIn (string)
Services > Authentication > OAuth > Response > ExpiresIn
If your application requires OAuth 2.0 authentication, you can optionally add a Response object to your OAuth object.
Coveo will then extract the information from the API’s response using the values specified in the Response object.
In the Response object, you can add the ExpiresIn property to specify the path to the refresh token expiration delay in the API’s response.
Default is expires_in.
If the API’s response doesn’t contain an expiration delay, use the ExpiresInDefaultValue property instead.
Example
If the authentication server’s response looks like this:
HTTP/1.1 200 OK
Content-Type: application/json{
"access_token": "jOs0PoLKm61sd6h49",
"refresh_token": "QpMas-tuI9bvFp01-Xxpl04gC_0m",
"token_type": "bearer",
"expires_in": 86400
}Then your Response object should look like this:
"Response": {
"AccessToken": "access_token",
"SupportsRefreshToken" : "true",
"RefreshToken": "refresh_token",
"ExpiresIn": "expires_in",
"TokenType": "token_type"
}
ExpiresInDefaultValue (number)
Services > Authentication > OAuth > Response > ExpiresInDefaultValue
If your application requires OAuth 2.0 authentication, you can optionally add a Response object to your OAuth object.
Coveo will then extract the information from the API’s response using the values specified in the Response object.
Typically, the authentication server will return an expiration delay for its refresh token, which you can extract with the ExpiresIn property.
However, if the API’s response doesn’t include an expiration delay, you can add the ExpiresInDefaultValue property to your Response object.
The ExpiresInDefaultValue value specifies the number of seconds after which Coveo should consider the refresh token expired.
Default is 3600.
Example:
"Response": {
"AccessToken": "access_token",
"SupportsRefreshToken" : "true",
"RefreshToken": "refresh_token",
"ExpiresInDefaultValue": 1800,
"TokenType": "token_type"
}
TokenType (string)
Services > Authentication > OAuth > Response > TokenType
If your application requires OAuth 2.0 authentication, you can optionally add a Response object to your OAuth object.
Coveo will then extract the information from the API’s response using the values specified in the Response object.
In the Response object, you can add the ExpiresIn property to specify the path to the token type in the API’s response.
Default is token_type.
If the response doesn’t specify a token type or if it’s invalid or inappropriate, Coveo will use Bearer by default.
If you want a different default value, use the TokenTypeDefaultValue property instead.
Example
If the authentication server’s response looks like this:
HTTP/1.1 200 OK
Content-Type: application/json{
"access_token": "jOs0PoLKm61sd6h49",
"refresh_token": "QpMas-tuI9bvFp01-Xxpl04gC_0m",
"token_type": "bearer",
"expires_in": 86400
}Then your Response object should look like this:
"Response": {
"AccessToken": "access_token",
"SupportsRefreshToken" : "true",
"RefreshToken": "refresh_token",
"ExpiresIn": "expires_in",
"TokenType": "token_type"
}
TokenTypeDefaultValue (string)
Services > Authentication > OAuth > Response > TokenTypeDefaultValue
If your application requires OAuth 2.0 authentication, you can optionally add a Response object to your OAuth object.
Coveo will then extract the information from the API’s response using the values specified in the Response object.
Typically, the authentication server will return a token type, which you can extract with the TokenType property.
If the API’s response doesn’t specify a token type, or if the token type is invalid or inappropriate, Coveo will use Bearer by default.
However, if you want Coveo to use a different default token type, you can specify it by the TokenTypeDefaultValue property to your Response object.
AuthorizationHeader (string)
Services > Authentication > OAuth > Response > AuthorizationHeader
If your application requires OAuth 2.0 authentication, you can optionally add a Response object to your OAuth object.
In the Response object, you can add the AuthorizationHeader property to indicate the name of the authorization header that Coveo should use to provide its access token when querying your API for content.
Refer to your API’s documentation for the header name to use.
Default is Authorization.
|
|
Notes
|
Example:
If your API documentation specifies that the access token must be provided in a header named X-ACCESS-TOKEN, then your Response object should specify it like this:
"Response": {
"AccessToken": "access_token",
"SupportsRefreshToken" : "true",
"RefreshToken": "refresh_token",
"ExpiresIn": "expires_in",
"TokenType": "token_type",
"AuthorizationHeader": "X-ACCESS-TOKEN"
}As a result, when making content requests to your application, Coveo will automatically include an X-ACCESS-TOKEN header with the access token value.
{
"headers": {
"Content-Type": "application/json",
"X-ACCESS-TOKEN": "123456789abcdef"
}
}
UsingSingleUseRefreshToken (Boolean)
Set UsingSingleUseRefreshToken to true to indicate that the refresh token used by the source to authenticate to your application is single-use.
Default is false.
A single-use refresh token expires after it’s been used.
When setting UsingSingleUseRefreshToken to true, make sure to include a refresh_token query parameter in your OAuth query.
Example
"Authentication": {
"OAuth": {
"UsingSingleUseRefreshToken": true,
"Query": {
"RefreshUrl": "https://auth.example.com/oauth/token",
"Method": "POST",
"Parameters": {
"grant_type": {
"Type": "Payload",
"Value": "refresh_token"
},
"client_id": {
"Type": "Payload",
"Value": "@ClientId"
},
"client_secret": {
"Type": "Payload",
"Value": "@ClientSecret"
},
"refresh_token": {
"Type": "Payload",
"Value": "@RefreshToken",
"IsRefreshToken": true
}
}
}
}
}
Headers (object)
Services > Headers
A key-value list of HTTP headers to add to the query. Each value can be either a number, string, or Boolean.
The Headers object is inheritable.
|
|
Note
Unlike a browser or a tool such as Postman, Coveo doesn’t automatically add a |
Examples:
Basic example
"Headers": {
"Custom-Header-Name": "myValue"
}Authentication with an API key in the Authorization HTTP header
If your API requires Coveo to authenticate with an API key in the Authorization HTTP header, you must enter this key in the Authentication section of the Add REST API source panel.
Then, in your source JSON configuration, you must use the @ApiKey placeholder to refer to this key.
This ensures that your API key isn’t stored in clear text in the source JSON configuration, where other Administration Console users could see it.
When Coveo processes the source configuration, it replaces the placeholder with the actual key you entered in the panel.
Your source configuration could therefore look like this.
{
"Services": [
{
"Url": "http://example.com/api/v1",
"Headers": {
"Authorization": "Bearer @ApiKey"
},
"Endpoints": [
{
"Path": "/users",
"Method": "GET",
"ItemType": "User",
"Uri": "%[coveo_url]/users/%[id]",
"ClickableUri": "%[coveo_url]/users/%[id]"
}
]
}
]
} |
Bearer is the authorization type and is most commonly used with OAuth 2.0 tokens.
If your API requires a different authorization type, replace Bearer with the appropriate type. |
As a result, Coveo sends the following HTTP request to the API:
GET /users HTTP/1.1
Host: http://example.com/api/v1
Authorization: Bearer Az4adfSyBhZX8fTThUJ4Kb2TC3Ax0RshgrZtUiqM |
Az4adfSyBhZX8fTThUJ4Kb2TC3Ax0RshgrZtUiqM is the API key you entered in the source panel. |
Authentication with an API key in a custom HTTP header
If your API requires Coveo to authenticate with an API key in the custom HTTP header X-Api-Key, you must enter this key in the Authentication section of the Add REST API source panel.
Then, in your source JSON configuration, you use the @ApiKey placeholder to refer to this key.
This ensures that your API key isn’t stored in clear text in the source JSON configuration, where other Administration Console users could see it.
When Coveo processes the source configuration, it replaces the placeholder with the actual key you entered in the panel.
Your source configuration could therefore look like this.
{
"Services": [
{
"Url": "http://example.com/api/v1",
"Headers": {
"X-Api-Key": "@ApiKey"
},
"Endpoints": [
{
"Path": "/users",
"Method": "GET",
"ItemType": "User",
"Uri": "%[coveo_url]/users/%[id]",
"ClickableUri": "%[coveo_url]/users/%[id]"
}
]
}
]
} |
X-Api-Key is the name of the custom header that your API expects to contain the API key. |
As a result, Coveo sends the following HTTP request to the API:
GET /users HTTP/1.1
Host: http://example.com/api/v1
X-Api-Key: Az4adfSyBhZX8fTThUJ4Kb2TC3Ax0RshgrZtUiqM |
Az4adfSyBhZX8fTThUJ4Kb2TC3Ax0RshgrZtUiqM is the API key you entered in the source panel. |
Paging (object)
Services > Paging
In the Paging object, specify how you want the content to be paged.
This property is inheritable, but you can disable inheritance using the DoNotInherit property.
This may help speed up the crawling process.
When you set it at the service level, it will apply to all endpoints in this service by default.
You can override the service-level configuration by setting the Paging object again at the endpoint level.
Sub-items inherit this property.
Sub-queries don’t inherit this property.
Example:
"Paging": {
"PageSize": 20,
"OffsetStart": 0,
"OffsetType": "page",
"Parameters": {
"Limit": "limit",
"Offset": "page"
},
"DoNotInherit": true
}The Paging object supports the following properties:
-
OffsetType(required) -
PageSize(required)
OffsetType (string enum, required)
When configuring your source to request paginated content, you must specify how your API paginates its content. Coveo needs this information to build its page request URLs in the appropriate format. You can typically find it in your API’s documentation.
The OffsetType property is a string enum that specifies the type of pagination your API uses.
The value to specify depends on your API’s pagination technique, just like the properties you’ll need to add to your Paging object.
| Pagination technique |
OffsetType value |
Properties to add to the Paging object |
|---|---|---|
Page-based |
|
|
Offset-based |
|
|
URL-based |
|
|
Cursor-based |
|
Examples:
Page-based pagination
Your API expects request URLs in a format such as http://example.com/api/item?limit=100&pageNumber=0 for the first page, and http://example.com/api/item?limit=100&pageNumber=1 for the second page.
With these URLs, Coveo would request the first page (page 0), and then increment the page number by 1 to request the second page, and so on.
In this case, your Paging object should look like this:
{
"Paging": {
"PageSize": 100,
"OffsetStart": 0,
"OffsetType": "page",
"Parameters": {
"Limit": "limit",
"Offset": "pageNumber"
}
}
}Offset-based pagination
Your API expects request URLs in a format such as http://example.com/api/item?limit=100&start=0 for the first page, and http://example.com/api/item?limit=100&start=100 for the second page.
With these URLs, Coveo would request the 100 first items starting from item 0, and then the 100 next items, starting from item 100.
In this case, your Paging object should look like this:
{
"Paging": {
"PageSize": 100,
"OffsetStart": 0,
"OffsetType": "item",
"Parameters": {
"Limit": "limit",
"Offset": "start"
}
}
}URL-based pagination
Your API expects request URLs in a format such as http://example.com/api/item?limit=8 for the first page, and http://example.com/api/item?page=3d170d80d8n3n2342c328s for the second page.
In this context, Coveo would request the first page, and then the next page using the URL provided in the nextPage value in the JSON response.
In this case, if your API returns the following first page:
{
"nextPage": "http://example.com/api/item?page=3d170d80d8n3n2342c328s",
"items": [
{ "id": 1, "name": "Caroline" },
{ "id": 2, "name": "Marcella" },
{ "id": 3, "name": "Susie" },
{ "id": 4, "name": "Rhonda" },
{ "id": 5, "name": "Wendy" },
{ "id": 6, "name": "Barbara" },
{ "id": 7, "name": "Deirdre" },
{ "id": 8, "name": "Lynda" }
]
}Then your Paging object should look like this:
{
"Paging": {
"PageSize": 8,
"NextPageKey": "nextPage",
"OffsetType": "url",
"Parameters": {
"Limit": "limit"
}
}
}Cursor-based pagination
Your API expects request URLs in a format such as http://example.com/api/item?limit=8 for the first page, and http://example.com/api/item?limit=8&nextPageToken=3d170d80-7b3b-4377 for the second page.
With these URLs, Coveo would request the 8 first items, and then the next page using the token provided in the NextPageId value in the JSON response.
In this case, if your API returns the following first page:
{
"nextPageId": "3d170d80-7b3b-4377",
"items": [
{ "id": 1, "name": "Caroline" },
{ "id": 2, "name": "Marcella" },
{ "id": 3, "name": "Susie" },
{ "id": 4, "name": "Rhonda" },
{ "id": 5, "name": "Wendy" },
{ "id": 6, "name": "Barbara" },
{ "id": 7, "name": "Deirdre" },
{ "id": 8, "name": "Lynda" }
]
}Then your Paging object should look like this:
{
"Paging": {
"PageSize": 8,
"NextPageKey": "NextPageId",
"OffsetType": "cursor",
"Parameters": {
"Limit": "limit",
"Offset": "nextPageToken"
}
}
}
PageSize (number, required)
The number of items to fetch per page.
Example: With paging URL https://example.com/api/item?position=0&quantity=50, your JSON configuration should include "PageSize": 50.
DoNotInherit (Boolean)
Whether to prevent your Paging configuration from being inherited.
This can be useful for preventing paging on PermissionSubqueries, which is known to slow down the crawling process.
The default value is false.
NextPageKey (string)
When the OffsetType value is "url" or "cursor", provide the path (simple path or JSONPath) to the key representing the value of the next page reference, that is, either the next page URL or the next page cursor.
The NextPageKey property isn’t required when the OffsetType value is page or item.
Example: "NextPageKey": "NextPageLink"
OffsetStart (number)
Offset of the first page to fetch.
This property is required when the OffsetType value is "page" or "item".
Example: Your first paging URL is https://example.com/api/item?position=0&quantity=50.
If you want to retrieve all items except items 1 to 50, your paging configuration must include "OffsetStart": 1.
Paging Parameters (object)
In the paging Parameters object, you must provide the names of the query string fields in your web application pagination URL that are required to support pagination.
The properties expected in this object are Limit and Offset.
However, Offset isn’t required when the OffsetType value is "url" since the complete URL is already retrieved in the JSON response.
Example:
"Parameters": {
"Limit": "limit",
"Offset": "page"
}The Parameters object supports the following properties:
-
Limit(required)
Limit (string, required)
Services > Paging > Parameters > Limit
In the query string, the field whose value is retrieved with the PageSize parameter.
Example: With paging URL https://example.com/api/item?position=0&quantity=50, your JSON configuration should include "Limit": "quantity".
FetchNextPageUntilNoResult (Boolean)
Services > Paging > Parameters > FetchNextPageUntilNoResult
The FetchNextPageUntilNoResult property indicates whether Coveo will try to fetch pages until no result is returned, or until the application API returns an error.
Default value is true.
The FetchNextPageUntilNoResult property is an alternative to the TotalCountHeaderKey and TotalCountKey properties.
|
|
Unless you set this property to |
Example:
"Paging": {
"OffsetType": "page",
"PageSize": 25,
"Parameters": {
"FetchNextPageUntilNoResult": true
}
}
Offset (string)
Services > Paging > Parameters > Offset
In the query string, the field whose value is retrieved with the OffsetStart property.
Example: Your first paging URL is https://example.com/api/item?position=0&quantity=50.
If you want to ignore items 1 to 50 and retrieve items 51 and over, your paging configuration must include "Offset": "position" and "OffsetStart": 1.
TotalCountHeaderKey (string)
The name of an HTTP header whose value represents the total number of items retrieved through the API call. This property is useful when Coveo makes an API call to a page that doesn’t exist, causing an error to be returned.
However, when using this property, you must be sure that the total always matches the actual number of items returned by your API, as Coveo’s crawler uses this total as an indication to stop. Should the expected total be lower than the actual number of items returned, Coveo won’t index the remaining items.
Therefore, if your API supports it, we recommend using the FetchNextPageUntilNoResult property instead, as a safer alternative.
On the other hand, if the expected total is higher than the actual number of items returned, Coveo’s crawler will throw an error after 10 empty pages.
In such a case, you can download the activity logs to review the TotalCountKey value that Coveo retrieved, and then adjust your source configuration accordingly.
Example: "TotalCountHeaderKey": "x-total-count"
TotalCountKey (string)
The path (simple path or JSONPath) to a response body property representing the total number of items retrieved through the API call. This property is useful when Coveo makes an API call to a page that doesn’t exist, causing an error to be returned.
However, when using this property, you must be sure that the total always matches the actual number of items returned by your API, as Coveo’s crawler uses this total as an indication to stop. Should the expected total be lower than the actual number of items returned, Coveo won’t index the remaining items.
Therefore, if your API supports it, we recommend using the FetchNextPageUntilNoResult property instead, as a safer alternative.
On the other hand, if the expected total is higher than the actual number of items returned, Coveo’s crawler will throw an error after 10 empty pages.
In such a case, you can download the activity logs to review the TotalCountKey value that Coveo retrieved, and then adjust your source configuration accordingly.
Example: "TotalCountKey": "retrievedContent.totalCount"
Permissions (array)
Services > Permissions
Each object in the Permissions array represents a permission level.
In turn, permission levels contain one or more PermissionsSets arrays.
The Permissions array is required when you select the Same users and groups as in your content system source content security option.
It contains the permission levels and permission sets to apply to the items indexed through this service.
The members listed in these levels and sets are either allowed or denied access to the indexed items.
For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.
This property is inheritable.
Example:
"Permissions": [
{
"Name": "Permission Level 1",
"PermissionsSets": [
{
"Name": "Permission Set 1",
"AllowedMembers": [
{
"Name": "jsmith@example.com",
"Type": "User",
"AdditionalInfo": {
"title": "%[customMetadata]"
}
}
],
"DeniedMembers": null,
"IsAnonymousAllowed": false
}
]
}
]The Permissions array supports the following properties:
PermissionsSets (array)
Services > Permissions > PermissionsSets
The PermissionsSets array specifies who can or can’t access the indexed items.
Each object in the array describes a single permission set, which can contain lists of allowed and denied members.
For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.
Objects in the PermissionsSets array support the following properties:
Permission set Name (string)
Services > Permissions > PermissionsSets > Name
The Name property indicates the name of the permission set.
AllowedMembers and DeniedMembers arrays
Services > Permissions > PermissionsSets > AllowedMembers/DeniedMembers
Each object in the AllowedMembers and DeniedMembers arrays represents a member security identity that should be allowed or denied to access the indexed content.
|
|
Note
If a user is listed under both |
For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.
The following is an example of a simple situation where, in your permission system, security identities are identified with email addresses. Your source will work with the Email security identity provider to ensure that your search interface users only see the content they’re allowed to.
If your permission system is more complex, requires a security provider other than Email, or is frequently updated, you should not only index allowed and denied security identities, but also provide a permission configuration instructing Coveo on how to index your permission system.
You index text items written by different people.
The metadata of each item contains the key author, and, in this system, authors are identified by their email address.
These addresses are therefore expected as values of the Name field.
With the following configuration, each user in your organization can access the items they have created through your Coveo-powered search interfaces.
"PermissionsSets": [
{
"Name": "Permission Set 1",
"AllowedMembers": [
{
"Name": "%[author]",
"Type": "User",
"Condition": "%[published] == true",
}
],
"DeniedMembers": null,
"IsAnonymousAllowed": false
}
]AllowedMembers and DeniedMembers arrays support the following properties:
-
Name(required)
Member Name (string, required)
Services > Permissions > PermissionsSets > AllowedMembers/DeniedMembers > Name
The name of the member security identity.
In simple use cases similar to that described under AllowedMembers and DeniedMembers arrays, this should be an email address.
Dynamic values are supported.
Member Type (string enum, required)
Services > Permissions > PermissionsSets > AllowedMembers/DeniedMembers > Type
The type of member.
Allowed values are User, Group, and VirtualGroup.
For more information on these types of security identities, see Coveo management of security identities and item permissions.
Permission Condition (string)
Services > Permissions > PermissionsSets > AllowedMembers/DeniedMembers > Condition
The Condition value is a condition that must resolve to true for the security identity specified in Name to be resolved, that is, extracted by the security identity provider.
As a result, the permission applies.
When the condition resolves to false, the security identity is not resolved and the corresponding permission doesn’t apply.
Adding a condition to your AllowedMembers or DeniedMembers object is optional.
To write your condition, use the same syntax as for indexing pipeline extension conditions.
Conditions can be assembled using the following operators: AND, OR, Exists, NOT, >, and <.
Parentheses are also supported to specify operation order.
The metadata fields in your condition must either be defined in the Metadata object of the source configuration or referenced with raw.
You can specify an array if the metadata refers to a multi-value field in your repository.
Dynamic values are also supported.
You index a list of available and discontinued products, and your public search interface is used by both your employees and your customers. On this search interface, your employees can log in to see additional content, while customers can’t log in.
You want to hide your discontinued products from your customers only, so you use the Condition property to apply a permission to these items.
As a result, all products with Discontinued: yes in their metadata are visible to members of the *@mycompany.com group only, that is, your employees, while non-discontinued products are available to all search interface users.
"PermissionsSets": [
{
"Name": "Permission Set 1",
"AllowedMembers": [
{
"Name": "*@mycompany.com",
"Type": "Group",
"Condition": "%[discontinued] == yes"
}
],
"DeniedMembers": null,
"IsAnonymousAllowed": true
}
]For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.
PermissionType (string)
Services > Permissions > PermissionsSets > AllowedMembers/DeniedMembers > PermissionType
This property is relevant only when you provide a permission configuration.
The PermissionType is a key representing the configuration on how to extract all relationships of the security identity specified in Name.
The security identity provider will use this configuration when processing the security identity.
In the Content Security tab of the Add/Edit a REST API Source panel, you’ll need to select the Same users and groups as in your content system, and then to provide a JSON permission configuration detailing how to retrieve the relationships of each security identity and how to index this data.
For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.
AdditionalInfo (object)
Services > Permissions > PermissionsSets > AllowedMembers/DeniedMembers > AdditionalInfo
The AdditionalInfo object is relevant only when you provide a permission configuration.
It is a key-value store of additional information required by some security providers. Each key represents the metadata name of the repository item, while its value is the value path (simple path or JSONPath) in the JSON response.
When writing your permission configuration, you can use coveo_parent to refer to the information retrieved with the AdditionalInfo object of a permission subquery.
Dynamic values are supported in the values of this object.
IsAnonymousAllowed (Boolean)
Permissions > PermissionsSets > IsAnonymousAllowed
The IsAnonymousAllowed property indicates whether anonymous, that is, unauthenticated users are allowed to access the retrieved items.
Default value is false.
For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.
PermissionsFromMetadata array
Services > Permissions > PermissionsSets > PermissionsFromMetadata
This array is relevant only when you provide a permission configuration.
The PermissionsFromMetadata array is an alternative to the PermissionSubQueries array, as it fetches permission data, typically the security identities that are allowed or denied access to the item.
This data is crucial to retrieve if you want to replicate the application’s permission system in Coveo.
Include a PermissionsFromMetadata array in your source configuration when the permission data appears in the metadata of an item.
Conversely, a PermissionSubQueries array should be used when an item and its permission data must be retrieved with two different queries.
For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.
Objects of the PermissionsFromMetadata array support:
PermissionSubQueries array
Services > Permissions > PermissionsSets > PermissionSubQueries
This array is relevant only when you provide a permission configuration.
Each object in the PermissionSubQueries array represents a subquery to execute on every item to fetch permission data, typically the security identities that are allowed or denied access to the item.
This data is crucial to retrieve if you want to replicate the application’s permission system in Coveo.
Each object in the PermissionSubQueries array inherits the paging properties specified at the parent endpoint or service level.
The PermissionSubQueries array is an alternative to the PermissionsFromMetadata array.
Include a PermissionSubQueries array in your source configuration when an item and its permission data must be retrieved with two different queries.
Conversely, the PermissionsFromMetadata array should be used when the permission data appears in the metadata of an item.
For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.
Objects of the PermissionSubQueries array support:
-
The following properties from the
Endpointsobject configuration: -
The following properties from objects of the
AllowedMembersandDeniedMembersarrays: -
The following properties:
Example:
"permissionSubQueries": [
{
"ItemPath": "permissions",
"Path": "%[productid]/permissions",
"Method": "GET",
"QueryParameters": {},
"Name": "%[name]",
"Type": "group",
"PermissionType": "config1",
"Condition": "%[field] == value",
"IsAllowedMember": true,
"AdditionalInfo":
{
"PermissionID": "%[id]",
"PermissionName": "%[name]"
}
}
]
IsAllowedMember (Boolean)
Services > Permissions > PermissionsSets > PermissionSubQueries > IsAllowedMember
The IsAllowedMember property indicates whether the specified member is allowed to access the item.
Default value is false.
For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.
StringItemPath (string)
Services > Permissions > PermissionsSets > StringItemPath
The path to the item in the JSON response returned by your API.
When indexing permissions with PermissionSubQueries, use StringItemPath in place of ItemPath if your API returns the list of identities in an array rather than as a series of objects.
Then, use value %[item] to tell Coveo where you want to store each of the identities extracted from the permission array.
The Name property is an appropriate place.
Read more on StringItemPath vs. ItemPath
Most APIs return permissions as a series of objects such as:
{
"allowed": [
{
"email": "jsmith@example.com"
},
{
"email": "ballen@example.com"
}
]
}In such a case, you would use ItemPath to extract permissions.
However, if your API rather returns permissions as follows, you should use StringItemPath instead.
{
"allowed": ["jsmith@example.com", "ballen@example.com"]
}If you expect your API to return permissions in an array such as:
{
"allowedusers": ["jsmith@example.com", "ballen@example.com"]
}Include the following in your permission subquery: "StringItemPath": "allowedusers".
In addition, to index the extracted user identities under Name, add: "Name": %[item].
Your permission subquery therefore looks as follows:
"permissionSubQueries": [
{
"StringItemPath": "allowedusers",
"Path": "item/location",
"Method": "GET",
"Name": "%[item]",
"Type": "User",
"IsAllowedMember": true
}
]StringItemPath can appear in PermissionSubQueries objects only.
|
|
Note
|
SkippableErrorCodes (string)
Services > SkippableErrorCodes
When crawling your content, Coveo may encounter an HTTP error and stop.
If you expect the web application API to return an error and want it to be ignored so that the crawling process continues, specify the error code as the SkippableErrorCodes property value.
This property is inheritable.
When you set it at the service level, it will apply to all endpoints in this service, and subqueries in these endpoints.
You can override the service-level configuration by setting the SkippableErrorCodes property again at the endpoint or subquery level.
Sub-items don’t inherit this property.
Use a semicolon (;) to separate error codes.
Example: "SkippableErrorCodes": "500;404;403"
Very useful
Not really
Very useful
Not really