Generic REST API Source Reference

When creating a Generic REST API source in the Coveo Cloud Administration Console, you must provide a JSON configuration detailing which services and endpoints to crawl to retrieve the desired content, and how to retrieve items of each type. This configuration is a JSON object with single property, Services, whose value must be an array of Service Object.

{
  "Services": [
    {
      "Url": "http://example.com/api/v1",
      "Endpoints": [
        {
          "Path": "/users",
          "Method": "GET",
          "ItemType": "User",
          "Uri": "%[coveo_url]/users/%[id]",
          "ClickableUri": "%[coveo_url]/users/%[id]",
          "Title": "%[name]",
          "ModifiedDate": "%[updated]",
          "Metadata": {
            "id": "%[id]",
            "username": "%[name]"
          }
        }
      ]
    }
  ]
}

This reference article defines the object and array parameters to include in your JSON configuration.

When building your JSON configuration, you may also want to refer to the Generic REST API Source JSON Configuration Examples for partial and complete configuration examples, as well as to the Concepts article and the Generic REST API Source Tutorial.

Service Object

Each object in the Services array specifies a REST API service to retrieve content through, along with the specific endpoints to call.

The properties expected in an object of the Services array are the following.

Endpoints (Array of Endpoint Object, Required)

Each array in the Endpoints object details how you want Coveo Cloud to crawl an endpoint of your web application. See the Endpoints section for details on the items found in an array.

Url (String, Required)

The Url value is the REST service URL of the web application of which you want to make the content searchable.

Example: "Url": "http://example.com/api/v1"

Authentication (Authentication Object)

If your web application service requires authentication, this object contains the REST service authentication configuration. See the Authentication section for details on the properties in this object.

You must also enter authentication information in the Authentication section of the Add/Edit a Generic REST API Source panel.

Then:

  • If your source uses an HTTP, Basic, Kerberos, or NTLM authentication protocol, the username and password you provide under Authentication replace any value you entered for the Username and Password properties.

  • If your source uses an API key authentication protocol, include the @APIkey placeholder in the HTTP headers, query parameters, or payload parameters.

    Example:

    API key placeholder in JSON configuration

    For more API key authentication examples, see API Key Authentication.

  • If your source uses the OAuth 2.0 authentication protocol, include the @ClientID, @ClientSecret, and @RefreshToken placeholders in the OAuth 2.0 object.

Paging (Paging Object)

In the Paging object, you can specify how you want the Services content to be paged.

See the Paging section for details on the properties in this object.

Permissions (Array of Permission Level Object)

The Permissions array is required when you select the Determined by source permissions source content security option. It contains the permission levels and sets to apply to the items indexed through this service. The members listed in these levels and sets are either allowed or denied access to the indexed items.

This property is inheritable.

See the Permission Level Object section for details on the properties in this object.

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo Cloud Management of Security Identities and Item Permissions.

SkippableErrorCodes (String)

When crawling your content, Coveo Cloud may encounter an HTTP error and stop. If you expect the web application API to return an error and want it to be ignored so that the crawling process continues, specify the error code as the SkippableErrorCodes property value.

This property applies at the Services level and is inheritable.

Use a semicolon (;) to separate two error codes.

Example: "SkippableErrorCodes": "500;404;403"

Endpoint Object

Each object in the Endpoints array details how to retrieve the content of the specified endpoint.

Moreover, in each endpoint object, the Metadata objects lists the content metadata to retrieve.

ClickableUri (String, Required)

The format of the item clickable URI.

Dynamic values are supported.

Example: "ClickableUri": "%[coveo_url]/posts/%[id]"

ItemType (String, Required)

The type of item that the resource contains. Coveo Cloud uses this data to organize the indexed content.

Space characters and dashes aren’t supported.

Examples:

  • "ItemType": "Post" for a blog post

  • "ItemType": "PostComment" for a comment on a post

Method (String Enum, Required)

The HTTP method to use to fetch the resource. Possible values are GET and POST. See the API documentation of the application to make searchable to determine which one you should use.

Example: "Method": "GET"

Path (String, Required)

The relative path to the desired resource endpoint.

Dynamic values are supported when specified within the SubItems, SubQueries, and PermissionSubQueries objects only.

Example:

To retrieve your GitHub repositories: "Path": "orgs/myorg/repos"

Uri (String, Required)

The format of the item unique identifier.

Dynamic values are supported.

Example: "Uri": "%[coveo_url]/repository/posts/%[id]"

Body (String)

The path (simple path or JSONPath) to the Body content of the item in the JSON response. The item body is displayed in the Administration Console Content Browser.

Typically, the Body property contains a dynamic value, since a static, hardcoded value would make all item bodies identical. Dynamic values and hardcoded text can also be mixed.

Example: "Body": "%[html.content]"

Headers (Object)

A key-value list of HTTP headers to add to the query. Each value can be either a number, string, or Boolean.

Example:

"Headers": {
  "Custom-Header-Name": "myValue",
  "X-Api-Key": "1234567890"
}

IndexingAction (Action Object)

Whether to ignore or to retrieve an item if the specified Condition resolves to true. See the Action Object section for details on the properties expected in the IndexingAction object.

When the specified Condition resolves to true, the Coveo Cloud crawler applies the specified action to the corresponding item. Possible actions are Retrieve and Ignore, i.e., the crawler can either index the item to make it searchable or ignore it.

An item ignored with IndexingAction isn’t indexed and therefore not visible in the Administration Console Content Browser or a search interface. However, its sub-items are sent to the index, unless otherwise specified in the ProcessingAction object.

If the condition resolves to false, the Coveo Cloud crawler falls back to the default action, which is Retrieve, and the specified item is indexed.

Example:

"IndexingAction": {
    "ActionOnItem": "Ignore",
    "Condition": "%[id] > 1"
}

IsBinaryBody (Boolean)

Whether the subquery retrieves binary content as the body of the item. Default value is false.

This property is available within SubQueries objects only. It allows you to download a file (e.g., image or PDF document) to use as the item body in a search result quick view. See Supported File Formats for an exhaustive list of the item types that Coveo Cloud can index.

Example:

{
  "Path":"%[coveo_parent.url]",
  "Method":"GET",
  "IsBinaryBody":true
}

ItemPath (String)

The path to the item in the JSON response returned by the application.

When ItemPath is omitted, Coveo Cloud assumes that the item is at the root level of the JSON response.

See the Collection of Items example.

Metadata (Object)

A key-value list of the metadata to parse. Each key represents the metadata name of the repository item, while its value is the value path (simple path or JSONPath) in the JSON response.

Typically, the value path consists of one or more dynamic values, since a static, hardcoded value would result in an identical piece of metadata for all items. However, you could choose to use a hardcoded value so that the corresponding Coveo Cloud field is filled even though the API doesn’t provide this information.

Example:

"Metadata": {
  "author": "hardcoded value",
  "id": "%[id]",
  "expiration": "%[info.expiration]",
  "createdby": "Mr. %[author]",
  "authorUrl": "%[coveo_url]/authors/%[id]",
  "description": "%[author] is an amazing author!"
}

ModifiedDate (String)

The date on which the item was last modified. Although this property isn’t required in the JSON configuration, we recommend that you include it, as the Coveo-powered search pages use it for their default Sort by Date option.

Typically, ModifiedDate has a dynamic value, since a static, hardcoded value would make dates identical for all items.

Example: "ModifiedDate": "%[updated]"

Paging (Paging Object)

The paging configuration at the Services level.

See the Paging section for details on the properties in this object.

PayloadJsonContent (String)

The JSON content to send as a POST request body. The Content-Type header is automatically set to application/json.

Dynamic values are supported when specified within the SubItems and SubQueries objects only.

  • Since the JSON content is provided as a string value in a JSON configuration, it must be escaped accordingly.

  • PayloadJsonContent and PayloadParameters are mutually exclusive. In other words, if you provide a PayloadJsonContent string, you cannot provide payload parameters, and vice versa.

You want to provide the following JSON content with your request:

{
  "key1": "value1",
  "key2": "@field=(\"value2\")",
  "key3":
    {
      "subkey1": "subvalue1"
    }
}

Your source JSON configuration therefore contains the following:

"Endpoints": [
  {
    "Method": "POST",
    "Path": "/rest/myendpoint/",
    "Headers": {
      "Authorization": "Bearer @ApiKey"
    },
    "PayloadJsonContent": "{\"key1\": \"value1\", \"key2\": \"@field=(\\\"value2\\\")\", \"key3\": {\"subkey1\": \"subvalue1\"}}"

PayloadParameters (Object)

A key-value list of HTTP parameters to add to the payload. Each value can be either a number, string, or Boolean.

This property can only be used in a POST request. The parameters are sent as URL-encoded data in the request body. The Content-Type header is automatically set to application/x-www-form-urlencoded.

Dynamic values are supported when specified within the SubItems and SubQueries objects only.

Example:

"PayloadParameters": {
  "type": "post",
  "expand": "true",
  "id": 120
}

PayloadParameters and PayloadJsonContent are mutually exclusive. In other words, if you provide payload parameters, you cannot provide a JSON object, and vice versa.

PermanentId (String)

The PermanentId is an alternative, permanent unique item identifier.

When you expect the item Uri to change, this ensures that Coveo Machine Learning (Coveo ML) recognizes the items despite their new URI and associates them to the old data. So, in the future, if you modify or delete this source and reindex the same items, specifying the same item PermanentId format in your next configuration will ensure that Machine Learning doesn’t consider your items as new and rather picks up where it left off.

Dynamic values are supported.

Example:

{
  "Uri": "%[coveo_url]/items/%[sys_id]",
  "PermanentId": "%[sys_id]"
}

Permissions (Array of Permission Level Object)

The Permissions object is required when you select the Determined by source permissions source content security option. It contains the permission levels and sets to apply to the items indexed in this endpoint. The members listed in these levels and sets are either allowed or denied access to the indexed items.

This property is inheritable.

See the Permission Level Object section for details on the properties in this object.

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo Cloud Management of Security Identities and Item Permissions.

ProcessingAction (Action Object)

Whether to ignore or to retrieve the child items of an item if the specified Condition resolves to true. See the Action Object section for details on the properties expected in the ProcessingAction object.

When the specified Condition resolves to true, the Coveo Cloud crawler applies the specified action on the corresponding child items of an item. Possible actions are Process and Ignore, i.e., the crawler can either retrieve the subitems to make them searchable or ignore them.

A child item ignored with ProcessingAction isn’t indexed and therefore not visible in the Administration Console Content Browser or a search interface.

If the condition resolves to false, it fallbacks to the default action, i.e., Process.

Example:

"ProcessingAction": {
    "ActionOnItem": "Ignore",
    "Condition": "%[id]==1"
}

QueryParameters (Object)

A key-value list of HTTP parameters to add to the query. Each value can be either a number, string, or Boolean.

Dynamic values are supported when specified within the SubItems, SubQueries, and PermissionSubQueries objects only.

Example:

"QueryParameters": {
  "type": "post",
  "expand": "true",
  "id": 120
}

RefreshEndpoints (Array of Refresh Endpoint Object)

A list of refresh endpoint queries to execute to retrieve added, updated, or deleted items when refresh is supported and performed. See the Refresh Endpoints section for details on the properties in this object.

SkippableErrorCodes (String)

When crawling your content, Coveo Cloud may encounter an HTTP error and stop. If you expect the web application API to return an error and want it to be ignored so that the crawling process continues, specify the error code as the SkippableErrorCodes property value.

This property applies at the Endpoints level and is inheritable.

Use a semicolon (;) to separate two error codes.

Example: "500;404;403"

SubItems (Array of Endpoint Object)

A list of sub-items to retrieve. Each sub-item is represented by a configuration object under Endpoints. An item can have many different sub-items, e.g., a blog post containing comments and attachments as child items. The SubItems object establishes a parent-child relationship between items. There’s no restriction on the number of sub-items an item can have.

To refer to the metadata of a parent item, prefix the metadata name with coveo_parent.. You may also need to use raw.

Example:

"SubItems": [
  {
    "Path": "/posts/%[coveo_parent.id]/comments",
    "Method": "GET",
    "ItemType": "Comment",
    "Uri": "%[coveo_url]/repository/posts/%[coveo_parent.id]/comments/%[id]",
    "ClickableUri": "%[coveo_url]/posts/%[coveo_parent.id]/comments/%[id]",
    "Title": "%[title]",
    "ModifiedDate": "%[updated]",
    "Body": "%[content]",
    "Metadata": {
      "author": "%[author]",
      "createdby" :"Mr. %[coveo_parent.author]",
      "id": "%[id]"
    }
  }
]

SubQueries (Array of Subquery Object)

A list of subqueries to execute on every item in order to fetch additional metadata and/or the item body, especially if this information is located under a different path than that of the item. See the Subquery Object section for details.

To refer to the metadata of a parent item, prefix the metadata name with coveo_parent.. You may also need to use raw.

Title (String)

The title of the item. Although this property isn’t required in the JSON configuration, we recommend that you include it, as the Administration Console Content Browser displays its value.

Typically, Title contains a dynamic value, since a static, hardcoded value would make all item titles identical.

Example: "Title": "%[title]"

Action Object

When the specified Condition resolves to true, the Coveo Cloud crawler applies the specified action to the corresponding items.

See IndexingAction and ProcessingAction for more details.

ActionOnItem (String, Required)

Action to perform if the specified Condition resolves to true.

In the IndexingAction object, this applies to retrieved items. Possible actions are Retrieve and Ignore, i.e., the crawler can either index the item to make it searchable or ignore it.

In the ProcessingAction object, this applies to child items of an item. Possible actions are Process and Ignore, i.e., the crawler can either retrieve the subitems to make them searchable or ignore them.

Condition (String)

The Condition value is a condition that must resolve to true for the specified action to apply to the child items of an item.

To write your condition, use the same syntax as for indexing pipeline extension conditions. Conditions can be assembled together using the following operators: AND, OR, Exists, NOT, >, and <. Parentheses are also supported to specify operation order.

The metadata fields in your condition must either be defined in the Metadata object or referenced with raw. You can specify an array if the metadata refers to a multi-value field in your repository.

Examples:

  • "%[author_id]" is true if the item has an author_id.

  • "NOT %[author_id]" is true if the item doesn’t have an author_id.

  • "%[author_id] == 1234" is true if the item author_id is 1234.

  • "%[author_ids] == [1,2,3,4]" is true if item author_ids are 1, 2, 3, and 4.

  • "%[author_id] OR %[author_name]" is true if the item has an author_id or an author_name.

  • "%[author_id] AND %[author_name]" is true if the item has an author_id and an author_name.

  • "%[author_id] &gt; 123" is true if the item author_id is greater than 123.

  • "(%[author_id] OR %[author_sys_id]) AND %[author_name]" is true if the item has an author_id or an author_name, as well as an author_name.

Member Object

A Member object describes a security identity that can be either allowed or denied in a given permission set.

See AllowedMembers and DeniedMembers for more details.

Dynamic values are supported.

You index text items written by different people. The metadata of each item contains the text author, and, in this system, author names are email addresses. With the following configuration, each user in your organization can access the items they have created through your Coveo-powered search interfaces.

"PermissionsSets": [
  {
    "Name": "Permission Set 1",
    "AllowedMembers": [
      {
        "Name": "%[author]",
        "Type": "User",
        "PermissionType": "config1",
        "Condition": "%[field] == value"
      }
    ],
    "DeniedMembers": null,
    "IsAnonymousAllowed": false
  }
]

By default, the source uses an Email Security Provider and therefore expects an email address under Name. If you want to use a different security provider, see Change the Default Security Provider of a Generic REST API Source.

Name (String, Required)

The name of the member security identity. Email addresses and Active Directory usernames are supported.

By default, the source uses an Email Security Provider and therefore expects an email address under Name. If you want to use an Active Directory security provider and provide names in the format <domain>\<username>, see Change the Default Security Provider of a Generic REST API Source.

Type (String Enum, Required)

The type of member. Allowed values are User, Group, and VirtualGroup.

For more information on these types of security identities, see Coveo Cloud Management of Security Identities and Item Permissions.

PermissionType (String, Required)

A key representing the configuration on how to extract all relationships of the security identity specified in Name. The security provider will use this configuration when processing the security identity.

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo Cloud Management of Security Identities and Item Permissions.

Permission Condition (String)

The Condition value is a condition that must resolve to true for the security identity specified in Name to be resolved, i.e., extracted by the security identity provider. As a result, the permission applies.

When the condition resolves to false, the security identity is not resolved and the corresponding permission doesn’t apply.

To write your condition, use the same syntax as for indexing pipeline extension conditions. Conditions can be assembled together using the following operators: AND, OR, Exists, NOT, >, and <. Parentheses are also supported to specify operation order.

The metadata fields in your condition must either be defined in the Metadata object or referenced with raw. You can specify an array if the metadata refers to a multi-value field in your repository.

You index a list of available and discontinued products, and your search interface is used by both your customers and your employees. You want to hide your discontinued products from your customers only, so you use the Condition parameter to apply a permission to these items. As a result, all products with Discontinued: yes in their metadata are visible to members of the *@mycompany.com* group only, i.e. your employees, while non-discontinued products are available to all search interface users.

"PermissionsSets": [
  {
    "Name": "Permission Set 1",
    "AllowedMembers": [
      {
        "Name": "*@mycompany.com",
        "Type": "Group",
        "PermissionType": "config1",
        "Condition": "%[discontinued] == yes"
      }
    ],
    "DeniedMembers": null,
    "IsAnonymousAllowed": false
  }
]

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo Cloud Management of Security Identities and Item Permissions.

Paging Object

In the Paging object, specify how you want the content to be paged.

This property is inheritable.

Example:

"Paging": {
  "PageSize": 20,
  "OffsetStart": 0,
  "OffsetType": "page",
  "Parameters": {
    "Limit":"limit",
    "Offset": "page"
  }
}

See Generic REST API Source JSON Configuration Examples for additional paging examples.

OffsetType (String Enum, Required)

The type of information on which the offset is based. Possible values are page, item, url, and cursor.

  • page: An offset per page is based on the page number, i.e., each paginated call increments the offset by 1.

    The offset of the following calls is based on the page number:

    • http://example.com/api/item?limit=10&page=0

    • http://example.com/api/item?limit=10&page=1

  • item: An offset based on the item number increments the offset by the number of items in each page.

    The offset of the following calls is based on the item number:

    • http://example.com/api/item?limit=10&start=0

    • http://example.com/api/item?limit=10&start=10

  • url: A URL-based offset retrieves the next page (offset) URL with the NextPageKey in the JSON response, and the next call is a request to this URL.

    The following links could be returned in the web application response as values of NextPageKey. When you choose to use a URL-based object, Coveo Cloud sends a request to this URL to retrieve your content.

    • http://example.com/api/item?limit=5

    • http://example.com/api/item?page=3d170d80d8n3n2342c328s

  • cursor: An offset based on the cursor retrieves the next page cursor using the NextPageKey in the JSON response and builds the next page URL.

    The offset of the following calls is based on the page cursor:

    • http://example.com/api/item?limit=5

    • http://example.com/api/item?limit=5&pageToken=3d170d80-7b3b-4371-9499-2ec6a78c507e

PageSize (Number, Required)

The number of items to fetch per page.

Example: With paging URL https://example.com/api/item?pposition=0&quantity=50, your JSON configuration should include "PageSize": 50.

NextPageKey (String)

When the OffsetType value is "url" or "cursor", provide the path (simple path or JSONPath) to the key representing the value of the next page reference, i.e., either the next page URL or the next page cursor.

The NextPageKey property isn’t required when the OffsetType value is page or item.

Example: "NextPageKey": "NextPageLink"

OffsetStart (Number)

Offset of the first page to fetch. This property is required when the OffsetType value is "page" or "item".

Example: Your first paging URL is https://example.com/api/item?position=0&quantity=50. If you want to retrieve all items except items 1 to 50, your paging configuration must include "OffsetStart": 1.

Parameters (Parameter Object)

In the Parameters object, you must provide the names of the query string fields in your web application pagination URL that are required to support pagination. The parameters expected in this object are Limit and Offset.

However, the Offset parameter isn’t required when the OffsetType value is "url" since the complete URL is already retrieved in the JSON response.

Example:

"Parameters": {
  "Limit": "limit",
  "Offset": "page"
}

TotalCountHeaderKey (String)

The name of an HTTP header whose value represents the total number of items retrieved through the API call. This property is useful when Coveo Cloud makes an API call to a page that doesn’t exist, causing an error to be returned.

Example: "TotalCountHeaderKey": "x-total-count"

TotalCountKey (String)

The path (simple path or JSONPath) to a response body property representing the total number of items retrieved through the API call. This property is useful when Coveo Cloud makes an API call to a page that doesn’t exist, causing an error to be returned.

Example: "TotalCountKey": "retrievedContent.totalCount"

Parameter Object

The parameters in the Parameter object indicate the name of the paging query string fields.

Limit (String, Required)

In the query string, the field whose value is retrieved with the PageSize parameter.

Example: With paging URL https://example.com/api/item?position=0&quantity=50, your JSON configuration should include "Limit": "quantity".

Offset (String)

In the query string, the field whose value is retrieved with the OffsetStart parameter.

Example: Your first paging URL is https://example.com/api/item?position=0&quantity=50. If you want to ignore items 1 to 50 and retrieve items 51 and over, your paging configuration must include "Offset": "position" and "OffsetStart": 1.

Permission Level Object

A Permission Level object describes a single permission level, which can contain one or more permission sets.

This property is inheritable.

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo Cloud Management of Security Identities and Item Permissions.

Example:

"Permissions": [
  {
    "Name": "Permission Level 1",
    "PermissionsSets": [
      {
        "Name": "Permission Set 1",
        "AllowedMembers": [
          {
            "Name": "jsmith@example.com",
            "Type": "User"
          }
        ],
        "DeniedMembers": null,
        "IsAnonymousAllowed": false
      }
    ]
  }
]

Name (String)

The name of the permission level.

PermissionsSets (Array of Permission Set Object)

The PermissionsSets array specifies who can or can’t access the indexed items.

See the Permission Set Object section for details on the properties expected for each element in this array.

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo Cloud Management of Security Identities and Item Permissions.

Permission Set Object

A Permission Set object describes a single permission set which can contain lists of allowed and denied members.

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo Cloud Management of Security Identities and Item Permissions.

Name (String)

The name of the permission set.

AllowedMembers (Array of Member Object)

Each object in the AllowedMembers array represents a member that should be allowed to access the indexed content.

If a user is listed under both AllowedMembers and DeniedMembers for an item, the denial prevails so that security holes are avoided.

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo Cloud Management of Security Identities and Item Permissions.

See the Member Object section for details on the properties expected in the AllowedMembers object.

DeniedMembers (Array of Member Object)

Each object in the DeniedMembers array represents a member to whom access to the indexed item should be denied.

If a user is both listed under AllowedMembers and DeniedMembers at once for an item, the denial prevails so that security holes are avoided.

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo Cloud Management of Security Identities and Item Permissions.

See the Member Object section for details on the properties expected in the DeniedMembers object.

IsAnonymousAllowed (Boolean)

The IsAnonymousAllowed property indicates whether anonymous, i.e., non authenticated users are allowed to access the retrieved items. Default value is false.

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo Cloud Management of Security Identities and Item Permissions.

PermissionSubQueries (Array of Permission Subquery Object)

The PermissionSubQueries array contains a list of subqueries to execute on every item to fetch permission data, especially if this information is located under a different path than that of the item. See the Permission Subquery Object section for details.

Permission Subquery Object

A list of subqueries to execute on every item in order to fetch permission data, especially if this information is located under a different path than that of the item.

The subquery configuration contains:

Example:

"permissionSubQueries": [
  {
    "ItemPath": "permissions",
    "Path": "%[productid]/permissions",
    "Method": "GET",
    "QueryParameters": {},
    "Name": "%[name]",
    "Type": "group",
    "PermissionType": "config1",
    "Condition": "%[field] == value",
    "IsAllowedMember": true,
    "AdditionalInfo":
      {
        "PermissionID": "%[id]",
        "PermissionName": "%[name]"
      }
  }
]

IsAllowedMember (Boolean)

The IsAllowedMember property indicates whether the specified member is allowed to access the item. Default value is false.

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo Cloud Management of Security Identities and Item Permissions.

AdditionalInfo (Object)

A key-value list of the metadata to parse. Each key represents the metadata name of the repository item, while its value is the value path (simple path or JSONPath) in the JSON response.

Typically, the value path consists of one or more dynamic values, since a static, hardcoded value would result in an identical piece of metadata for all items.

Refresh Endpoint Object

A refresh endpoint is an endpoint that inherits from the parent endpoint definition. The refresh endpoint allows you to redefine the Endpoints configuration properties. As a result, when you refresh your source, the properties defined under RefreshEndpoints override the properties defined under Endpoints. When you rescan or rebuild the source, however, the Endpoints properties apply.

Example:

"RefreshEndpoints":[
  {
      "DateFormat":"\\'yyyy-MM-dd\\',\\'hh:mm:ss\\'",
      "QueryParameters":{
        "workflow_state":"published",
        "modified":"%[modified_date] >= @RefreshDate"
      }
  },
  {
      "IsDeletionQuery":true,
      "DateFormat":"\\'yyyy-MM-dd\\',\\'hh:mm:ss\\'",
      "QueryParameters":{
        "workflow_state":"%[workflow_state] !=published",
        "modified":"%[modified_date] >= @RefreshDate"
      }
  },
  {
      "IsDeletionQuery":true,
      "DateFormat":"\\'yyyy-MM-dd\\',\\'hh:mm:ss\\'",
      "Path":"/api/table/audit_delete",
      "QueryParameters":{
        "tablename":"kb_knowledge",
        "modified":"%[modified_date] >= @RefreshDate"
      }
  },
  {
      "IsDeletedItem": "%[is_deleted]",
      "DateFormat":"\\'yyyy-MM-dd\\',\\'hh:mm:ss\\'",
      "QueryParameters":{
      "modified":"%[modified_date] >= @RefreshDate"
      }
  }
]

DateFormat (String)

The format to use for the refresh date and time when injecting them in the refresh query defined under the RefreshEndpoints. Default is ISO 8601 (yyyy-MM-ddTHH:mm:ssZ).

The refresh date is injected using the token @RefreshDate.

Examples:

  • "DateFormat": "yyyy-MM-ddTHH:mm:ssZ"

  • "DateFormat": "\\'yyyy-MM-dd\\',\\'hh:mm:ss\\'"

IsDeletionQuery (Boolean)

Whether the defined query returns items to delete from the index. Default value is false, which means that all retrieved items are added to the index or updated. If the query returns items to add, update, and delete all at once, use the IsDeletedItem property instead.

IsDeletedItem (Boolean)

When a refresh query returns items to add, update, and delete all at once, this property allows you to specify which of these items should be deleted. Default value is false, but it’s considered to be true if IsDeletionQuery is set to true.

You can provide a dynamic value (e.g., %[has_been_deleted_flag]) to be evaluated for each item or a static value.

Example:

"%[deleted]" deletes all items that have a "deleted": true JSON property flagging them as deleted.

"IsDeletedItem": %[is_deleted],
  "DateFormat":"\\'yyyy-MM-dd\\',\\'hh:mm:ss\\'",
  "QueryParameters":{
    "modified":"%[modified_date] >= @RefreshDate"
  }

DeleteChildren (Boolean)

Whether to delete children when the retrieved parent item is deleted. Default value is false. If the value is set to true, child items are deleted based on their Uri.

Subquery Object

A list of subqueries to execute on every item in order to fetch additional metadata and/or the item body, especially if this information is located under a different path than that of the item.

The subquery configuration contains a subset of the fields from the Endpoints object configuration:

Example:

"SubQueries": [
  {
    "Path": "/posts/%[coveo_parent.id]/additionalinfo",
    "Method": "GET",
    "Body": "%[data.body]",
    "Metadata": {
      "created": "%[created]"
    }
  }
]

Authentication Object

If one of the Services to crawl requires authentication, your JSON configuration must include the following properties.

Username (String)

Username used to log in to the REST service. You can use @Username to retrieve the value specified in the Add a Generic REST API Source panel.

Password (String)

Password used to log in to the REST service. This property can be empty even though a Username is specified. You can use @Password to retrieve the value specified in the Add a Generic REST API Source panel.

Domain (String)

Domain name to use when authenticating to a NTLM or Kerberos protected server.

ForceBasicAuthentication (Boolean)

Whether to force a basic HTTP header in the request. The default value is false.

OAuth (OAuth Object)

If the application to make searchable requires OAuth 2.0 authentication, the OAuth object must contain the required configuration. You can use the @ syntax to map the credentials or API key you specified in the Add a Generic REST API Source panel.

See the OAuth 2.0 section for details on the object properties.

Example:

"OAuth": {
  "Query": {
    "RefreshUrl": "http://example.com/token",
    "Method": "POST",
    "Parameters": {
      "grant_type": {
        "Type": "Payload",
        "Value": "refresh_token"
      },
      "refresh_token": {
        "Type": "Payload",
        "Value": "REFRESH_TOKEN",
        "IsRefreshToken": true
      },
      "client_id": {
        "Type": "Payload",
        "Value": "CLIENT_ID"
      },
      "client_secret": {
        "Type": "Payload",
        "Value": "CLIENT_SECRET"
      }
    }
  },
  "Response": {
    "AccessToken": "access_token",
    "SupportsRefreshToken" : "true",
    "RefreshToken": "refresh_token",
    "ExpiresIn": "expires_in",
    "TokenType": "token_type"
  }
}

OAuth Object

If the application to make searchable requires OAuth 2.0 authentication, include the following properties in your source JSON configuration.

Query (OAuth Query Object, Required)

The Query object represents the OAuth 2.0 refresh query configuration. See OAuth 2.0 Query for the properties expected in this object.

Enter your credentials in the Add a Generic REST API Source panel and use the @ClientId, @ClientSecret and @RefreshToken placeholders in your JSON configuration.

Response (OAuth Response Object)

The Response object represents the OAuth 2.0 refresh query configuration. See OAuth 2.0 Response for the properties expected in this object.

OAuth Query Object

If the application to make searchable requires OAuth 2.0 authentication, include the following properties in your JSON configuration.

Example:

"Query": {
  "RefreshUrl": "http://example.com/token",
  "Method": "POST",
  "Parameters": {
    "grant_type": {
      "Type": "Payload",
      "Value": "refresh_token"
    },
    "refresh_token": {
      "Type": "Payload",
      "Value": "@RefreshToken",
      "IsRefreshToken": true
    },
    "client_id": {
      "Type": "Payload",
      "Value": "@ClientId"
    },
    "client_secret": {
      "Type": "Payload",
      "Value": "@ClientSecret"
    }
  }
}

RefreshUrl (String, Required)

The OAuth 2.0 refresh URL.

Example: "RefreshUrl": "http://example.com/token"

Header (Object)

The additional HTTP headers to use to authenticate the client and refresh the token.

Coveo Cloud automatically sets the Content-Type header. Therefore, you don’t need to enter anything for this header.

Example:

"Header": {
  "X-Auth": "API_KEY"
}

Method (String)

The HTTP method to use to refresh the OAuth 2.0 token. Default is POST.

Parameters (Object)

The parameters to use to authenticate the client and refresh the token. Possible values are query or payload.

If the OAuth 2.0 flow to use doesn’t support refresh tokens, don’t specify it in the Query object parameters.

Enter your credentials in the Add a Generic REST API Source panel and use the @ClientId, @ClientSecret and @RefreshToken placeholders in your JSON configuration.

Example:

"Parameters": {
  "grant_type": {
    "Type": "Payload",
    "Value": "refresh_token"
  },
  "refresh_token": {
    "Type": "Payload",
    "Value": "@RefreshToken",
    "IsRefreshToken": true
  },
  "client_id": {
    "Type": "Payload",
    "Value": "@ClientId"
  },
  "client_secret": {
    "Type": "Payload",
    "Value": "@ClientSecret"
  }
}

OAuth Response Object

If the application to make searchable requires OAuth 2.0 authentication, include the following properties in your JSON configuration.

Example:

"Response": {
  "AccessToken": "access_token",
  "SupportsRefreshToken" : "true",
  "RefreshToken": "refresh_token",
  "ExpiresIn": "expires_in",
  "TokenType": "token_type"
}

AccessToken (String)

The path to the access token in the response. Default is access_token.

SupportsRefreshToken (Boolean)

Whether the OAuth 2.0 flow supports refresh tokens. Default is true.

RefreshToken (String)

The path to the refresh token in the response. Default is refresh_token.

ExpiresIn (String)

The path to the expiration time in the response. Default is expires_in. If there’s no expiration time in the response, use the ExpiresInDefaultValue property.

ExpiresInDefaultValue (Number)

Specify an expiration value in seconds for the refresh token as an alternative to the ExpiresIn property. Default is 3600.

TokenType (String)

The path to the token type in the response. Default is token_type.

If the TokenType isn’t specified in the response or if it’s invalid or inappropriate, it defaults to Bearer. If you want a different default value, use the TokenTypeDefaultValue property.

TokenTypeDefaultValue (String)

If the TokenType isn’t specified in the response or if it’s invalid or inappropriate, it defaults to Bearer. Should you not want to use Bearer as a replacement for the inadequate token_type value that the response returns, specify the desired value.

Recommended Articles