SAP source reference

When creating an SAP source in the Coveo Administration Console, you must provide a JSON configuration detailing which services and endpoints to crawl to retrieve the desired content, and how to retrieve items of each type. This configuration has a single property, Services, whose value must be an array of Service Object.

Example
{
  "services": [
    {
      "url": "https://api.acmestore.com/odata2webservices",
      "headers": {
        "Accept": "application/json",
        "Application-Interface-Key": "xxxxxxxx"
      },
      "paging": {
        "pageSize": 10,
        "offsetStart": 0,
        "offsetType": "item",
        "nextPageKey": "__next",
        "parameters": {
          "limit": "$top",
          "offset": "$skip"
        }
      },
      "endpoints": [
        {
          "path": "/Product/Products",
          "method": "GET",
          "queryParameters": {
            "$filter": "catalogVersion/integrationKey eq 'Online%7CpowertoolsProductCatalog' and approvalStatus/code eq 'approved'",
            "$expand": "supercategories,thumbnails,picture,thumbnail,unit,catalogVersion/catalog,europe1Prices,europe1Prices/currency,bundleTemplates,localizedAttributes,supercategories/localizedAttributes,approvalStatus"
          },
          "itemPath": "d.results",
          "itemType": "product",
          "uri": "https://accstorefront.commerce.ondemand.com/coveob2bstorefront/powertools/en/USD/%[url]",
          "clickableUri": "https://accstorefront.commerce.ondemand.com/coveob2bstorefront/powertools/en/USD/%[url]",
          "metadata": {
            "objecttype": "Product",
            "ec_brand": "%[brand]",
            "ec_name": "%[name]",
            "name": "%[name]",
            "ec_item_group_id": "%[code]",
            "ec_shortdesc": "%[summary]",
            "ec_description": "%[description]",
            "ec_price": "%[europe1Prices.results[0].price]",
            "ec_thumbnails": "https://accstorefront.commerce.ondemand.com%[thumbnail.URL]",
            "ec_in_stock": "%[stock]",
            "ec_images": "https://accstorefront.commerce.ondemand.com%[picture.URL]",
            "ec_category": "%[supercategories.results[:].name]",
            "ec_pricerange": "%[priceRange]",
            "ec_configurable": "%[configurable]",
            "ec_code": "%[code]",
            "ec_url": "https://accstorefront.commerce.ondemand.com/coveob2bstorefront/powertools/en/USD/%[url]"
          }
        }
      ]
    }
  ]
}

This reference article defines parameters to include in your JSON configuration.

When building your JSON configuration, you may also want to refer to the SAP Source Configuration Reference and the Concepts article.

Service Object

Each object in the Services array specifies a REST API service to retrieve content through, along with the specific endpoints to call.

The properties expected in an object of the Services array are the following.

Endpoints (Array of Endpoint Object, Required)

Each object in the Endpoints array details how you want Coveo to crawl an endpoint of your SAP Commerce instance. You can specify as many endpoints as you need. See the Endpoints section for details on the Endpoint object properties.

By default, the SAP source JSON configuration contains three endpoints, one for:

  • product (for example, "path": "/Product/Products")

  • variant (for example, "path": "FootWearVariant/GenericVariantProducts")

  • availability (for example, "path": "/Availability/Warehouses")

You can remove or add endpoints depending on your use case.

Url (String, Required)

The Url value is the URL of your SAP Commerce Cloud OData2 GET API.

Example: "Url": "https://api.acmestore.com/odata2webservices"

Headers (Object)

A key-value list of HTTP headers to add to the query. Each value can be either a number, string, or Boolean.

AIR is required

You have to specify a unique Application Interface Key (Application Interface Register, AIR).

Refer to the SAP documentation for details.

This property is inheritable.

Example:

"headers": {
  "Custom-Header-Name": "myValue",
  "Application-Interface-Key": "xxxxxxxx"
}

Paging (Paging Object)

In the Paging object, you can specify how you want the Services content to be paged.

See the Paging section for details on the properties in this object.

Permissions (Array of Permission Level Object)

The Permissions array is required when you select the Same users and groups as in your current permission system source content security option. It contains the permission levels and permission sets to apply to the items indexed through this service. The members listed in these levels and sets are either allowed or denied access to the indexed items.

This property is inheritable.

See the Permission Level Object section for details on the properties in this object.

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.

SkippableErrorCodes (String)

When crawling your content, Coveo may encounter an HTTP error and stop. If you expect the web application API to return an error and want it to be ignored so that the crawling process continues, specify the error code as the SkippableErrorCodes property value.

This property applies at the Services level and is inheritable.

Use a semicolon (;) to separate error codes.

Example: "SkippableErrorCodes": "500;404;403"

Endpoint Object

Each object in the Endpoints array details how to retrieve the content of the specified endpoint.

Moreover, in each endpoint object, the Metadata objects lists the content metadata to retrieve.

ClickableUri (String, Required)

The format of the item clickable URI.

Dynamic values are supported.

Example: "ClickableUri": "%[coveo_url]/posts/%[id]"

ItemType (String, Required)

The type of item that the resource contains. Coveo uses this data to organize the indexed content.

Space characters and dashes aren’t supported.

Example: "itemType": "product"

Method (String Enum, Required)

The HTTP method to use to fetch the resource. Possible values are GET and POST. See the API documentation of the application to make searchable to determine which one you should use.

Example: "Method": "GET"

Path (String, Required)

The relative path to the desired resource endpoint.

Is’s made in the format of /name of the integration object created in the Backoffice/the type of the object+s

Dynamic values are supported when specified within the SubItems, SubQueries, and PermissionSubQueries objects only.

Dynamic time expressions are also supported.

In sub-items and subqueries, should a dynamic value be invalid or unretrievable, the source skips the item, that is, the indexing process is dropped and Coveo moves on to the next item. In a permission subquery, however, an invalid or unretrievable dynamic value causes the source to stop crawling and to display an error in the Administration Console.

Example: "path": "/Product/Products"

Uri (String, Required)

The URI of the product’s PDP page.

Dynamic values are supported.

Example: "Uri": "https://accstorefront.commerce.ondemand.com/coveob2bstorefront/powertools/en/USD/%[url]"

Should a dynamic value be invalid or unretrievable, the source stops crawling and displays an error in the Administration Console.

Body (String)

The path (simple path or JSONPath) to the Body content of the item in the JSON response. The item body is displayed in the Administration Console Content Browser.

Typically, the Body property contains a dynamic value, since a static, hardcoded value would make all item bodies identical. Dynamic values and hardcoded text can also be mixed.

Example: "Body": "%[html.content]"

DateFormat (String)

The format of a date injected thanks to the tokens @Now and @RefreshDate.

All .NET formats are supported, and default is yyyy-MM-ddTHH:mm:ssZ (ISO 8601).

Alternatively, to handle Unix timestamps, enter UnixEpoch.

Coveo also uses this format when parsing dates coming from your API. If these dates aren’t in a .NET format, use DateFormat to specify the format to expect.

Examples:

  • "DateFormat": "yyyy-MM-ddTHH:mm:ssZ"

  • "DateFormat": "\\'yyyy-MM-dd\\',\\'hh:mm:ss\\'"

IndexingAction (Action Object)

Whether to ignore or to retrieve an item if the specified Condition resolves to true. See the Action Object section for details on the properties expected in the IndexingAction object.

When the specified Condition resolves to true, the Coveo crawler applies the specified action to the corresponding item. Possible actions are Retrieve and Ignore, that is, the crawler can either index the item to make it searchable or ignore it.

An item ignored with IndexingAction isn’t indexed and therefore not visible in the Administration Console Content Browser or a search interface. However, its sub-items are sent to the index, unless otherwise specified in the ProcessingAction object.

If the condition resolves to false, the Coveo crawler falls back to the default action, which is Retrieve, and the specified item is indexed.

Example:

"IndexingAction": {
    "ActionOnItem": "Ignore",
    "Condition": "%[id] > 1"
}

IsBinaryBody (Boolean)

Whether the subquery retrieves binary content as the body of the item. Default value is false.

This property is available within SubQueries objects only. It allows you to download a file (for example, XLS or PDF document) to use as the item body in a search result quick view. See Supported File Formats for an exhaustive list of the item types that Coveo can index.

Example:

"SubQueries": [
{
  "Headers": {
    "x-api-key": "@ApiKey"
  },
  "Path":"%[coveo_parent.url]",
  "Method":"GET",
  "IsBinaryBody":true,
  "Body": "%[data.body]"
}
]
Notes
  • IsBinaryBody and IsThumbnail are mutually exclusive. In other words, if you set IsBinaryBody to true, you must set IsThumbnail to false, and vice versa.

  • If Coveo fails to index the item’s body, ensure that your API key is valid for subqueries, and try adding a Headers object to your subquery.

ItemPath (String, Required)

The path to the item in the JSON response returned by the application.

Its value should be d.results (see response structure.

Metadata (Object)

A key-value list of the metadata to parse.

This object maps attributes from the integration object to the Coveo fields. Some ecommerce fields are created and mapped by default. You can add customs fields in the Coveo Administration Console (platform-ca | platform-eu | platform-au).

Each key represents the metadata name of the SAP item, while its value is the value path (simple path or JSONPath) in the JSON response.

Typically, the value path consists of one or more dynamic values, since a static, hardcoded value would result in an identical piece of metadata for all items. However, you could choose to use a hardcoded value so that the corresponding Coveo field is filled even if the API doesn’t provide this information.

Example:

"metadata": {
  "objecttype": "Product",
  "ec_brand": "%[brand]",
  "ec_name": "%[name]",
  "name": "%[name]",
  "ec_item_group_id": "%[code]",
  "ec_shortdesc": "%[summary]",
  "ec_description": "%[description]",
  "ec_price": "%[europe1Prices.results[0].price]",
  "ec_thumbnails": "https://accstorefront.commerce.ondemand.com%[thumbnail.URL]",
  "ec_in_stock": "%[stock]",
  "ec_images": "https://accstorefront.commerce.ondemand.com%[picture.URL]",
  "ec_category": "%[supercategories.results[:].name]",
  "ec_pricerange": "%[priceRange]",
  "ec_configurable": "%[configurable]",
  "ec_code": "%[code]",
  "ec_url": "https://accstorefront.commerce.ondemand.com/coveob2bstorefront/powertools/en/USD/%[url]"
}

ModifiedDate (String)

The date on which the item was last modified. Although this property isn’t required in the JSON configuration, we recommend that you include it, as the Coveo-powered search pages use it for their default Sort by Date option.

Typically, ModifiedDate has a dynamic value, since a static, hardcoded value would make dates identical for all items.

Example: "ModifiedDate": "%[updated]"

Paging (Paging Object)

In the Paging object, you can specify how you want the Endpoint content to be paged.

See the Paging section for details on the properties in this object.

PayloadJsonContent (String)

The JSON content to send as a POST request body. The Content-Type header is automatically set to application/json.

Dynamic values are supported when specified within the SubItems and SubQueries objects only.

In sub-items and subqueries, should a dynamic value be invalid or unretrievable, the source skips the item, that is, the indexing process is dropped and Coveo moves on to the next item. In a permission subquery, however, an invalid or unretrievable dynamic value causes the source to stop crawling and to display an error in the Administration Console.

Notes
  • Since the JSON content is provided as a string value in a JSON configuration, it must be escaped accordingly.

  • PayloadJsonContent and PayloadParameters are mutually exclusive. In other words, if you provide a PayloadJsonContent string, you cannot provide payload parameters, and vice versa.

Example

You want to provide the following JSON content with your request:

{
  "key1": "value1",
  "key2": "@field=(\"value2\")",
  "key3":
    {
      "subkey1": "subvalue1"
    }
}

Your source JSON configuration therefore contains the following:

"Endpoints": [
  {
    "Method": "POST",
    "Path": "/rest/myendpoint/",
    "PayloadJsonContent": "{\"key1\": \"value1\", \"key2\": \"@field=(\\\"value2\\\")\", \"key3\": {\"subkey1\": \"subvalue1\"}}"
  }
]

PayloadParameters (Object)

A key-value list of HTTP parameters to add to the payload. Each value can be either a number, string, or Boolean.

This property can only be used in a POST request. The parameters are sent as URL-encoded data in the request body. The Content-Type header is automatically set to application/x-www-form-urlencoded.

Dynamic values are supported when specified within the SubItems and SubQueries objects only.

Dynamic time expressions are also supported.

Example:

"PayloadParameters": {
  "type": "post",
  "expand": "true",
  "id": 120
}

In sub-items and subqueries, should a dynamic value be invalid or unretrievable, the source skips the item, that is, the indexing process is dropped and Coveo moves on to the next item. In a permission subquery, however, an invalid or unretrievable dynamic value causes the source to stop crawling and to display an error in the Administration Console.

Note

PayloadParameters and PayloadJsonContent are mutually exclusive. In other words, if you provide payload parameters, you cannot provide a JSON object, and vice versa.

PermanentId (String)

The PermanentId is an alternative, permanent unique item identifier.

When you expect the item Uri to change, this ensures that Coveo Machine Learning (Coveo ML) recognizes the items despite their new URI and associates them to the old data. So, in the future, if you modify or delete this source and reindex the same items, specifying the same item PermanentId format in your next configuration will ensure that Machine Learning doesn’t consider your items as new and rather picks up where it left off.

Dynamic values are supported.

Example:

{
  "Uri": "%[coveo_url]/items/%[sys_id]",
  "PermanentId": "%[sys_id]"
}

Permissions (Array of Permission Level Object)

The Permissions object is required when you select the Same users and groups as in your current permission system source content security option. It contains the permission levels and sets to apply to the items indexed in this endpoint. The members listed in these levels and sets are either allowed or denied access to the indexed items.

This property is inheritable.

See the Permission Level Object section for details on the properties in this object.

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.

ProcessingAction (Action Object)

Whether to ignore or to retrieve the child items of an item if the specified Condition resolves to true. See the Action Object section for details on the properties expected in the ProcessingAction object.

When the specified Condition resolves to true, the Coveo crawler applies the specified action on the corresponding child items of an item. Possible actions are Process and Ignore, that is, the crawler can either retrieve the subitems to make them searchable or ignore them.

A child item ignored with ProcessingAction isn’t indexed and therefore not visible in the Administration Console Content Browser or a search interface.

If the condition resolves to false, it fallbacks to the default action, that is, Process.

Example:

"ProcessingAction": {
    "ActionOnItem": "Ignore",
    "Condition": "%[id]==1"
}

QueryParameters (Object)

A key-value list of HTTP parameters to add to the query. Each value can be either a number, string, or Boolean.

Dynamic values are supported when specified within the SubItems, SubQueries, and PermissionSubQueries objects only. They aren’t supported in QueryParameters objects within a RefreshEndpoints object.

Dynamic time expressions are also supported.

Example:

"QueryParameters": {
  "type": "post",
  "expand": "true",
  "id": 120
}

In sub-items and subqueries, should a dynamic value be invalid or unretrievable, the source skips the item, that is, the indexing process is dropped and Coveo moves on to the next item. In a permission subquery, however, an invalid or unretrievable dynamic value causes the source to stop crawling and to display an error in the Administration Console.

Filter (String)

The expression that specifies the products to be indexed. Must be URL-encoded.

Example: "$filter": "catalogVersion/integrationKey eq 'Online%7CpowertoolsProductCatalog' and approvalStatus/code eq 'approved'"

Expand (String)

The list of entries associated with a single integration object or a collection of integration objects.

Example: "$expand": "supercategories,thumbnails,picture,thumbnail,unit,catalogVersion/catalog,europe1Prices,europe1Prices/currency,bundleTemplates,localizedAttributes,supercategories/localizedAttributes,approvalStatus"

RefreshEndpoints (Array of Refresh Endpoint Object)

A list of refresh endpoint queries to execute to retrieve added, updated, or deleted items when refresh is supported and performed. See the Refresh Endpoints section for details on the properties in this object.

SkippableErrorCodes (String)

When crawling your content, Coveo may encounter an HTTP error and stop. If you expect the web application API to return an error and want it to be ignored so that the crawling process continues, specify the error code as the SkippableErrorCodes property value.

This property applies at the Endpoints level and is inheritable.

Use a semicolon (;) to separate error codes.

Example: "500;404;403"

SubItems (Array of Endpoint Object)

A list of sub-items to retrieve. Each sub-item is represented by a configuration object under Endpoints.

An item can have many different sub-items, for example, a blog post containing comments and attachments as child items. The SubItems object establishes a parent-child relationship between items. There’s no restriction on the number of sub-items an item can have.

In addition, sub-items can also have sub-items, for example if blog post comments also have attachments. In the application JSON response, these sub-items are nested within each other, and so should be your SubItems arrays in your source configuration.

To refer to the metadata of a parent item, prefix the metadata name with coveo_parent. You may also need to use raw.

Examples:

When the sub-items to index are located under a different endpoint, you provide the Path to their location.

"SubItems": [
  {
    "Path": "/posts/%[coveo_parent.id]/comments",
    "Method": "GET",
    "ItemType": "Comment",
    "Uri": "%[coveo_url]/repository/posts/%[coveo_parent.id]/comments/%[id]",
    "ClickableUri": "%[coveo_url]/posts/%[coveo_parent.id]/comments/%[id]",
    "Title": "%[title]",
    "ModifiedDate": "%[updated]",
    "Body": "%[content]",
    "Metadata": {
      "author": "%[author]",
      "createdby" :"%[coveo_parent.author]",
      "id": "%[id]"
    }
  }
]

Alternatively, when the desired sub-items are located under the same endpoint as their parent items (for example, the child item data is nested within the parent item data), you omit the Path property and provide the sub-items' location using the ItemPath property.

"SubItems": [
  {
    "ItemPath": "/comments",
    "ItemType": "Comment",
    "Uri": "%[coveo_url]/repository/posts/%[coveo_parent.id]/comments/%[id]",
    "ClickableUri": "%[coveo_url]/posts/%[coveo_parent.id]/comments/%[id]",
    "Title": "%[title]",
    "ModifiedDate": "%[updated]",
    "Body": "%[content]",
    "Metadata": {
      "author": "%[author]",
      "createdby" :"%[coveo_parent.author]",
      "id": "%[id]"
    }
  }
]

SubQueries (Array of Subquery Object)

A list of subqueries to execute on every item in order to fetch additional metadata and/or the item body, especially if this information is located under a different path than that of the item. See the Subquery Object section for details.

To refer to the metadata of a parent item, prefix the metadata name with coveo_parent. You may also need to use raw.

Title (String)

The title of the item. Although this property isn’t required in the JSON configuration, we recommend that you include it, as the Administration Console Content Browser displays its value.

Typically, Title contains a dynamic value, since a static, hardcoded value would make all item titles identical.

Example: "Title": "%[title]"

Action Object

When the specified Condition resolves to true, the Coveo crawler applies the specified action to the corresponding items.

See IndexingAction and ProcessingAction for more details.

ActionOnItem (String, Required)

Action to perform if the specified Condition resolves to true.

In the IndexingAction object, this applies to retrieved items. Possible actions are Retrieve and Ignore, that is, the crawler can either index the item to make it searchable or ignore it.

In the ProcessingAction object, this applies to child items of an item. Possible actions are Process and Ignore, that is, the crawler can either retrieve the subitems to make them searchable or ignore them.

Condition (String)

The Condition value is a condition that must resolve to true for the specified action to apply to the child items of an item.

To write your condition, use the same syntax as for indexing pipeline extension conditions. Conditions can be assembled using the following operators: AND, OR, Exists, NOT, >, and <. Parentheses are also supported to specify operation order.

The metadata fields in your condition must either be defined in the Metadata object or referenced with raw. You can specify an array if the metadata refers to a multi-value field in your repository.

Examples:

  • "%[author_id]" is true if the item has an author_id.

  • "NOT %[author_id]" is true if the item doesn’t have an author_id.

  • "%[author_id] == 1234" is true if the item author_id is 1234.

  • "%[author_ids] == [1,2,3,4]" is true if item author_ids are 1, 2, 3, and 4.

  • "%[author_id] OR %[author_name]" is true if the item has an author_id or an author_name.

  • "%[author_id] AND %[author_name]" is true if the item has an author_id and an author_name.

  • "%[author_id] &gt; 123" is true if the item author_id is greater than 123.

  • "(%[author_id] OR %[author_sys_id]) AND %[author_name]" is true if the item has an author_id or an author_name, as well as an author_name.

Member Object

A Member object describes a security identity that can be either allowed or denied in a given permission set.

See AllowedMembers and DeniedMembers for more details.

Dynamic values are supported.

Example

You index text items written by different people. The metadata of each item contains the text author, and, in this system, author names are email addresses. With the following configuration, each user in your organization can access the items they have created through your Coveo-powered search interfaces.

"PermissionsSets": [
  {
    "Name": "Permission Set 1",
    "AllowedMembers": [
      {
        "Name": "%[author]",
        "Type": "User",
        "PermissionType": "config1",
        "Condition": "%[field] == value",
        "AdditionalInfo": {
          "title": "%[customMetadata]"
        }
      }
    ],
    "DeniedMembers": null,
    "IsAnonymousAllowed": false
  }
]

Member Name (String, Required)

The name of the member security identity Email addresses and Active Directory usernames are supported.

Type (String Enum, Required)

The type of member. Allowed values are User, Group, VirtualGroup.

For more information on these types of security identities, see Coveo management of security identities and item permissions.

PermissionType (String, Required)

A key representing the configuration on how to extract all relationships of the security identity specified in Name. The security identity provider will use this configuration when processing the security identity.

In the Content Security tab of the Add/Edit an SAP Source panel, you’ll need to select the Same users and groups as in your current permission system, and then to provide a JSON permission configuration detailing how to retrieve the relationships of each security identity and how to index this data.

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.

AdditionalInfo (Object)

A key-value store of additional information required by some security providers. Each key represents the metadata name of the repository item, while its value is the value path (simple path or JSONPath) in the JSON response.

When writing a permission configuration, you can use coveo_parent to refer to the information retrieved with the AdditionalInfo parameter of a permission subquery.

Permission Condition (String)

The Condition value is a condition that must resolve to true for the security identity specified in Name to be resolved, that is, extracted by the security identity. As a result, the permission applies.

When the condition resolves to false, the security identity is not resolved and the corresponding permission doesn’t apply.

To write your condition, use the same syntax as for indexing pipeline extension conditions. Conditions can be assembled using the following operators: AND, OR, Exists, NOT, >, and <. Parentheses are also supported to specify operation order.

The metadata fields in your condition must either be defined in the Metadata object or referenced with raw. You can specify an array if the metadata refers to a multi-value field in your repository. Dynamic values are also supported.

Example

You index a list of available and discontinued products, and your search interface is used by both your customers and your employees. You want to hide your discontinued products from your customers only, so you use the Condition parameter to apply a permission to these items. As a result, all products with Discontinued: yes in their metadata are visible to members of the *@mycompany.com group only, that is, your employees, while non-discontinued products are available to all search interface users.

"PermissionsSets": [
  {
    "Name": "Permission Set 1",
    "AllowedMembers": [
      {
        "Name": "*@mycompany.com",
        "Type": "Group",
        "PermissionType": "config1",
        "Condition": "%[discontinued] == yes"
      }
    ],
    "DeniedMembers": null,
    "IsAnonymousAllowed": false
  }
]

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.

Paging Object

In the Paging object, specify how you want the content to be paged.

This property is inheritable.

Example:

"Paging": {
  "PageSize": 20,
  "OffsetStart": 0,
  "OffsetType": "page",
  "Parameters": {
    "Limit":"limit",
    "Offset": "page"
  }
}

OffsetType (String Enum, Required)

The type of information on which the offset is based. Possible values are page, item, url, and cursor.

  • page: An offset per page is based on the page number, that is, each paginated call increments the offset by 1.

Example

The offset of the following calls is based on the page number:

  • http://example.com/api/item?limit=10&page=0

  • http://example.com/api/item?limit=10&page=1

Example

The offset of the following calls is based on the item number:

  • http://example.com/api/item?limit=10&start=0

  • http://example.com/api/item?limit=10&start=10

  • url: A URL-based offset retrieves the next page (offset) URL with the NextPageKey in the JSON response, and the next call is a request to this URL.

Example

The following links could be returned in the web application response as values of NextPageKey. When you choose to use a URL-based object, Coveo sends a request to this URL to retrieve your content.

  • http://example.com/api/item?limit=5

  • http://example.com/api/item?page=3d170d80d8n3n2342c328s

  • cursor: An offset based on the cursor retrieves the next page cursor using the NextPageKey in the JSON response and builds the next page URL.

Example

The offset of the following calls is based on the page cursor:

  • http://example.com/api/item?limit=5

  • http://example.com/api/item?limit=5&pageToken=3d170d80-7b3b-4371-9499-2ec6a78c507e

PageSize (Number, Required)

The number of items to fetch per page.

Example: With paging URL https://example.com/api/item?position=0&quantity=50, your JSON configuration should include "PageSize": 50.

NextPageKey (String)

When the OffsetType value is "url" or "cursor", provide the path (simple path or JSONPath) to the key representing the value of the next page reference, that is, either the next page URL or the next page cursor.

The NextPageKey property isn’t required when the OffsetType value is page or item.

Example: "NextPageKey": "NextPageLink"

OffsetStart (Number)

Offset of the first page to fetch. This property is required when the OffsetType value is "page" or "item".

Example: Your first paging URL is https://example.com/api/item?position=0&quantity=50. If you want to retrieve all items except items 1 to 50, your paging configuration must include "OffsetStart": 1.

Parameters (Parameter Object)

In the Parameters object, you must provide the names of the query string fields in your web application pagination URL that are required to support pagination. The parameters expected in this object are Limit and Offset.

However, the Offset parameter isn’t required when the OffsetType value is "url" since the complete URL is already retrieved in the JSON response.

Example:

"Parameters": {
  "Limit": "limit",
  "Offset": "page"
}

TotalCountHeaderKey (String)

The name of an HTTP header whose value represents the total number of items retrieved through the API call. This property is useful when Coveo makes an API call to a page that doesn’t exist, causing an error to be returned.

Example: "TotalCountHeaderKey": "x-total-count"

TotalCountKey (String)

The path (simple path or JSONPath) to a response body property representing the total number of items retrieved through the API call. This property is useful when Coveo makes an API call to a page that doesn’t exist, causing an error to be returned.

Example: "TotalCountKey": "retrievedContent.totalCount"

Parameter Object

The parameters in the Parameter object indicate the name of the paging query string fields.

Limit (String, Required)

In the query string, the field whose value is retrieved with the PageSize parameter.

Example: With paging URL https://example.com/api/item?position=0&quantity=50, your JSON configuration should include "Limit": "quantity".

FetchNextPageUntilNoResult (Boolean)

The FetchNextPageUntilNoResult property indicates whether Coveo will try to fetch pages until no result is returned, or until the application API returns an error. Default value is false.

Note

When setting this parameter to true, we recommend also using the SkippableErrorCodes parameter at the service and/or endpoint level. This ensures that the crawling process continues when the API returns an error.

Example:

...
"Paging": {
  "OffsetType": "page",
  "PageSize": 25,
  "Parameters": {
    "FetchNextPageUntilNoResult": true
  }
},
...

Offset (String)

In the query string, the field whose value is retrieved with the OffsetStart parameter.

Example: Your first paging URL is https://example.com/api/item?position=0&quantity=50. If you want to ignore items 1 to 50 and retrieve items 51 and over, your paging configuration must include "Offset": "position" and "OffsetStart": 1.

Permission Level Object

A Permission Level object describes a single permission level, which can contain one or more permission sets.

This property is inheritable.

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.

Example:

"Permissions": [
  {
    "Name": "Permission Level 1",
    "PermissionsSets": [
      {
        "Name": "Permission Set 1",
        "AllowedMembers": [
          {
            "Name": "jsmith@example.com",
            "Type": "User",
            "AdditionalInfo": {
              "title": "%[customMetadata]"
            }
          }
        ],
        "DeniedMembers": null,
        "IsAnonymousAllowed": false
      }
    ]
  }
]

Name (String)

The name of the permission level.

PermissionsSets (Array of Permission Set Object)

The PermissionsSets array specifies who can or can’t access the indexed items.

See the Permission Set Object section for details on the properties expected for each element in this array.

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.

Permission Set Object

A Permission Set object describes a single permission set which can contain lists of allowed and denied members.

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.

Permission Set Name (String)

The Name property indicates the name of the permission set.

AllowedMembers (Array of Member Object)

Each object in the AllowedMembers array represents a member that should be allowed to access the indexed content.

Note

If a user is listed under both AllowedMembers and DeniedMembers for an item, the denial prevails so that security holes are avoided.

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.

See the Member Object section for details on the properties expected in the AllowedMembers object.

DeniedMembers (Array of Member Object)

Each object in the DeniedMembers array represents a member to whom access to the indexed item should be denied.

Note

If a user is both listed under AllowedMembers and DeniedMembers at once for an item, the denial prevails so that security holes are avoided.

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.

See the Member Object section for details on the properties expected in the DeniedMembers object.

IsAnonymousAllowed (Boolean)

The IsAnonymousAllowed property indicates whether anonymous, that is, unauthenticated users are allowed to access the retrieved items. Default value is false.

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.

PermissionsFromMetadata (Array of PermissionsFromMetadata Object)

The PermissionsFromMetadata object is an alternative to the PermissionSubqueries object, as it fetches security identity relationship data, typically the security identities that are children of a group.

See the PermissionsFromMetadata Object section for details on the properties expected for each element in this array.

PermissionSubQueries (Array of Permission Subquery Object)

The PermissionSubQueries array contains a list of subqueries to execute on every item to fetch permission data, typically the security identities that are allowed or denied access to the item. This data is crucial to retrieve if you want to replicate the application’s permission system in Coveo.

The PermissionSubQueries array si an alternative to the PermissionsFromMetadata object.

See the Permission Subquery Object section for details on the items found in an array.

PermissionsFromMetadata Object

The PermissionsFromMetadata object is an alternative to the PermissionSubQueries object, as it fetches permission data, typically the security identities that are allowed or denied access to the item. This data is crucial to retrieve if you want to replicate the application’s permission system in Coveo.

Include a PermissionsFromMetadata object in your SAP source configuration when the permission data appears in the metadata of an item. Conversely, a PermissionSubQueries object should be used when an item and its permission data must be retrieved with two different queries.

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.

The PermissionsFromMetadata object contains:

Permission Subquery Object

The PermissionSubQueries object is a list of subqueries to execute on every item in order to fetch permission data, typically the security identities that are allowed or denied access to the item. This data is crucial to retrieve if you want to replicate the application’s permission system in Coveo. It is an alternative to the PermissionsFromMetadata object.

Include a PermissionSubQueries object in your SAP source configuration when an item and its permission data must be retrieved with two different queries. Conversely, the PermissionsFromMetadata object should be used when the permission data appears in the metadata of an item.

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.

The subquery configuration contains:

Example:

"permissionSubQueries": [
  {
    "ItemPath": "permissions",
    "Path": "%[productid]/permissions",
    "Method": "GET",
    "QueryParameters": {},
    "Name": "%[name]",
    "Type": "group",
    "PermissionType": "config1",
    "Condition": "%[field] == value",
    "IsAllowedMember": true,
    "AdditionalInfo":
      {
        "PermissionID": "%[id]",
        "PermissionName": "%[name]"
      }
  }
]

IsAllowedMember (Boolean)

The IsAllowedMember property indicates whether the specified member is allowed to access the item. Default value is false.

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.

Refresh Endpoint Object

A refresh endpoint is an endpoint that inherits from the parent endpoint definition. The refresh endpoint allows you to redefine the Endpoints configuration properties. As a result, when you refresh your source, the properties defined under RefreshEndpoints override the properties defined under Endpoints. When you rescan or rebuild the source, however, the Endpoints properties apply.

In addition to the Endpoints object configuration properties, the RefreshEndpoints object supports the following properties:

Note

Dynamic values aren’t supported in the query parameters of a RefreshEndpoints object.

Example:

"RefreshEndpoints":[
  {
    "DateFormat":"\\'yyyy-MM-dd\\',\\'hh:mm:ss\\'",
    "QueryParameters":{
      "workflow_state":"published",
      "since": "@RefreshDate"
    }
  },
  {
    "IsDeletionQuery":true,
    "DateFormat":"\\'yyyy-MM-dd\\',\\'hh:mm:ss\\'",
    "QueryParameters":{
      "lastModifiedDate": "@Now-7d",
      "status":"Deleted"
    }
  },
  {
    "IsDeletionQuery":true,
    "DateFormat":"\\'yyyy-MM-dd\\',\\'hh:mm:ss\\'",
    "Path":"/api/table/audit_delete",
    "QueryParameters":{
      "tablename":"kb_knowledge",
      "since": "@RefreshDate"
    }
  },
  {
    "IsDeletedItem": "%[is_deleted]",
    "DateFormat":"\\'yyyy-MM-dd\\',\\'hh:mm:ss\\'",
    "QueryParameters":{
      "since": "@RefreshDate"
    }
  }
]

IsDeletionQuery (Boolean)

Whether the defined query returns items to delete from the index. Default value is false, which means that all retrieved items are added to the index or updated. If the query returns items to add, update, and delete all at once, use the IsDeletedItem property instead.

IsDeletedItem (Boolean)

When a refresh query returns items to add, update, and delete all at once, this property allows you to specify which of these items should be deleted. Default value is false, but it’s considered to be true if IsDeletionQuery is set to true.

You can provide a dynamic value (for example, %[has_been_deleted_flag]) to be evaluated for each item or a static value.

Example:

"%[is_deleted]" deletes all items that have a "is_deleted": true JSON property flagging them as deleted.

"IsDeletedItem": %[is_deleted],
"DateFormat":"\\'yyyy-MM-dd\\',\\'hh:mm:ss\\'",
"QueryParameters":{
  "since": "@RefreshDate"
}

DeleteChildren (Boolean)

Whether to delete children when the retrieved parent item is deleted. Default value is false. If the value is set to true, child items are deleted based on their Uri.

Subquery Object

A list of subqueries to execute on every item in order to fetch additional metadata and/or the item body, especially if this information is located under a different path than that of the item.

The subquery configuration contains a subset of the fields from the Endpoints object configuration:

It also contains the following parameters:

Example:

"SubQueries": [
  {
    "Path": "/posts/%[coveo_parent.id]/additionalinfo",
    "Method": "GET",
    "Body": "%[data.body]",
    "Metadata": {
      "created": "%[created]"
    }
  }
]

IsThumbnail (Boolean)

If the SubQueries object path leads to an image, set IsThumbnail to true to show this image as the thumbnail of an item.

The default value is false.

Supported thumbnail file formats are the following: .bmp, .emf, .exif, .gif, .icon, .jpeg, .png, .tiff, .wmf .

Notes
  • An Endpoint object can contain at most one subquery where IsThumbnail is set to true.

  • IsThumbnail and IsBinaryBody are mutually exclusive. In other words, if you set IsThumbnail to true, you must set IsBinaryBody to false, and vice versa.

Authentication Object

If one of the Services to crawl requires authentication, your JSON configuration must include the following properties.

Username (String)

Username used to log in to the REST service. You can use @Username to retrieve the value specified in the Add an SAP Source panel.

Password (String)

Password used to log in to the REST service. This property can be empty even though a Username is specified. You can use @Password to retrieve the value specified in the Add an SAP Source panel.

Domain (String)

Domain name to use when authenticating to a NTLM or Kerberos protected server.

ForceBasicAuthentication (Boolean)

Whether to force a basic HTTP header in the request. The default value is false.

OAuth (OAuth Object)

If the application to make searchable requires OAuth 2.0 authentication, the OAuth object must contain the required configuration. You can use the @ syntax to map the credentials or the API key you specified in the Add an SAP Source panel.

See the OAuth 2.0 section for details on the object properties.

Example:

"OAuth": {
  "Query": {
    "RefreshUrl": "http://example.com/token",
    "Method": "POST",
    "Parameters": {
      "grant_type": {
        "Type": "Payload",
        "Value": "refresh_token"
      },
      "refresh_token": {
        "Type": "Payload",
        "Value": "REFRESH_TOKEN",
        "IsRefreshToken": true
      },
      "client_id": {
        "Type": "Payload",
        "Value": "CLIENT_ID"
      },
      "client_secret": {
        "Type": "Payload",
        "Value": "CLIENT_SECRET"
      }
    }
  },
  "Response": {
    "AccessToken": "access_token",
    "SupportsRefreshToken" : "true",
    "RefreshToken": "refresh_token",
    "ExpiresIn": "expires_in",
    "TokenType": "token_type"
  }
}

OAuth Object

If the application to make searchable requires OAuth 2.0 authentication, include the following properties in your source JSON configuration.

Query (OAuth Query Object, Required)

The Query object represents the OAuth 2.0 refresh query configuration. See OAuth 2.0 Query for the properties expected in this object.

Enter your credentials in the Add an SAP Source panel and use the @ClientId, @ClientSecret, and @RefreshToken placeholders in your JSON configuration.

Response (OAuth Response Object)

The Response object represents the OAuth 2.0 refresh query configuration. See OAuth 2.0 Response for the properties expected in this object.

OAuth Query Object

If the application to make searchable requires OAuth 2.0 authentication, include the following properties in your JSON configuration.

Example:

"Query": {
  "RefreshUrl": "http://example.com/token",
  "Method": "POST",
  "Parameters": {
    "grant_type": {
      "Type": "Payload",
      "Value": "refresh_token"
    },
    "refresh_token": {
      "Type": "Payload",
      "Value": "@RefreshToken",
      "IsRefreshToken": true
    },
    "client_id": {
      "Type": "Payload",
      "Value": "@ClientId"
    },
    "client_secret": {
      "Type": "Payload",
      "Value": "@ClientSecret"
    }
  }
}

RefreshUrl (String, Required)

The OAuth 2.0 refresh URL.

Example: "RefreshUrl": "http://example.com/token"

Authentication Headers (Object)

The additional HTTP headers to use to authenticate the client and refresh the token.

Note

Coveo automatically sets the Content-Type header. Therefore, you don’t need to enter anything for this header.

Example:

"Headers": {
  "X-Auth": "API_KEY"
}

Method (String Enum)

The HTTP method to use to refresh the OAuth 2.0 token. Default is POST.

Parameters (Object)

The parameters to use to authenticate the client and refresh the token. Possible values are query or payload.

If the OAuth 2.0 flow to use doesn’t support refresh tokens, don’t specify it in the Query object parameters.

Enter your credentials in the Add an SAP Source panel and use the @ClientId, @ClientSecret, and @RefreshToken placeholders in your JSON configuration.

Example:

"Parameters": {
  "grant_type": {
    "Type": "Payload",
    "Value": "refresh_token"
  },
  "refresh_token": {
    "Type": "Payload",
    "Value": "@RefreshToken",
    "IsRefreshToken": true
  },
  "client_id": {
    "Type": "Payload",
    "Value": "@ClientId"
  },
  "client_secret": {
    "Type": "Payload",
    "Value": "@ClientSecret"
  }
}

OAuth Response Object

If the application to make searchable requires OAuth 2.0 authentication, include the following properties in your JSON configuration.

Example:

"Response": {
  "AccessToken": "access_token",
  "SupportsRefreshToken" : "true",
  "RefreshToken": "refresh_token",
  "ExpiresIn": "expires_in",
  "TokenType": "token_type"
}

AccessToken (String)

The path to the access token in the response. Default is access_token.

SupportsRefreshToken (Boolean)

Whether the OAuth 2.0 flow supports refresh tokens. Default is true.

RefreshToken (String)

The path to the refresh token in the response. Default is refresh_token.

ExpiresIn (String)

The path to the expiration time in the response. Default is expires_in. If there’s no expiration time in the response, use the ExpiresInDefaultValue property.

ExpiresInDefaultValue (Number)

Specify an expiration value in seconds for the refresh token as an alternative to the ExpiresIn property. Default is 3600.

TokenType (String)

The path to the token type in the response. Default is token_type.

If the TokenType isn’t specified in the response or if it’s invalid or inappropriate, it defaults to Bearer. If you want a different default value, use the TokenTypeDefaultValue property.

TokenTypeDefaultValue (String)

If the TokenType isn’t specified in the response or if it’s invalid or inappropriate, it defaults to Bearer. Should you not want to use Bearer as a replacement for the inadequate token_type value that the response returns, specify the desired value.

AuthorizationHeader (String)

Specify the name of the authorization header where the access token should appear. Default is Authorization.

Example:

"AuthorizationHeader": "X-ACCESS-TOKEN"