Add an SAP source

This is for:

Developer System Administrator

An SAP source allows you to crawl content from SAP Commerce Cloud.

When creating your source, you must provide a JSON configuration instructing Coveo to retrieve items from the SAP services and their respective resource endpoints. This configuration indicates which API calls to execute to fetch the desired items, how to parse the responses to extract relevant metadata, and which content type these items represent.

When working on your SAP source, you may also want to refer to the following articles:

Tip
Leading practice

The number of items that a source processes per hour (crawling speed) depends on various factors, such as network bandwidth and source configuration. See About crawling speed for information on what can impact crawling speed, as well as possible solutions.

Source key characteristics

The following table presents the main characteristics of an SAP source.

Features Supported Additional information

Content update operations

refresh

check

For each endpoint you define in your source configuration, you can provide a refresh endpoint to override the initial endpoint. When you do so, the connector can add, update, or delete specific items in the index instead of refreshing the entire repository. However, the following limitations currently apply:

  • Only refresh queries using the date of the last refresh operation can be made. Tokens can’t be used.

  • A sub-item can’t be refreshed if its parent item isn’t detected as modified.

rescan

check

Takes place every day by default.

rebuild

check

Content security options

Same users and groups as in your content system

x
Note

If you’d like to enable this option, contact Coveo Support.

If your source configuration includes the PermissionType parameter, you must provide a JSON configuration detailing how to extract the relationships of the indexed permissions.

Specific users and groups

x

Everyone

check

Prerequisites

Before you create an SAP source, you must:

  1. Ensure that your license allows you to create this source and that your organization hasn’t reached its source quota. You can review your quotas on the License & Usage (platform-ca | platform-eu | platform-au) page, along with your usage.

  2. Ensure that you have the privileges to create and/or to edit your source. If you don’t have them, the Add source button and/or sources are grayed out. In such case, ask your Coveo administrator to grant you what’s needed. See the required privileges and the series of articles on privilege management for details.

  3. If you’re using the SAP source in a Coveo for Commerce implementation, meet the SAP Commerce requirements.

SAP Commerce requirements

SAP sources must be associated with a catalog entity to ensure a complete configuration. This allows Coveo to build a product vector space. Behind the scenes, Coveo uses the Stream API to update the position of products in the vector space.

For instructions on how to create a catalog entity, see Commerce catalog entity.

Add an SAP source

Tip
Leading practice

It’s best to create or edit your source in your sandbox organization first. Once you’ve confirmed that it indexes the desired content, you can copy your source configuration to your production organization, either with a snapshot or manually. Source re-authentication with a separate crawling account may be required in the production organization.

See About non-production organizations for more information and best practices regarding sandbox organizations.

Follow these steps to add an SAP source.

  1. Make sure that your organization meets the prerequisites before adding this source.

  2. On the Sources (platform-ca | platform-eu | platform-au) page, click Add source.

  3. In the Add a source of content panel, click the SAP source tile.

  4. Configure your source.

The Build the source and Index metadata steps are especially important when creating a source of this type.

"Configuration" tab

In the Add an SAP Source panel, the Configuration tab is selected by default. It contains your source’s general and authentication information, as well as the JSON configuration that lets you specify the content to index.

"Identification" subtab

The Identification subtab contains general information about the source.

Name

Enter a name for your source.

Tip
Leading practice

A source name can’t be modified once it’s saved, therefore be sure to use a short and descriptive name, using letters, numbers, hyphens (-), and underscores (_). Avoid spaces and other special characters.
Project

Use the Project selector to associate your source with one or more Coveo projects.

"Content to index" subtab

In the JSON configuration box, enter your source JSON configuration. This configuration instructs Coveo on how to retrieve items from SAP’s REST API services and their respective resource endpoints.

You can start from our basic example and adapt it to your needs. For example, you could configure your source to benefit from the authentication, paging and refresh capabilities.

When working on your SAP source, you may also want to refer to the following articles:

"Authentication" subtab

In the Authentication section, all parameters are optional. Fill the appropriate boxes depending on the authentication type used by your SAP instance.

  • If your SAP Commerce uses a HTTP, Basic, Kerberos, or NTLM authentication protocol, enter the Username and Password of the account with which you want to crawl the source. Then, use the @Username and @Password placeholders in the Authentication object of your source JSON configuration instead of exposing the credentials in clear text. The account of which you enter the credentials must have access to all the content that you want to make searchable. See Source credentials leading practices.

  • If your SAP instance uses the OAuth 2.0 authentication protocol, enter your content source Client ID, Client secret and Refresh token in the corresponding boxes. Then, use the @ClientID, @ClientSecret, and @RefreshToken placeholders in the Authentication object of your source JSON configuration instead of exposing this information in clear text.

  • If your SAP instance uses an API key to authenticate, enter it in the API key box. Then, use the @ApiKey placeholder in the Headers, QueryParameters, or PayloadParameter object of your source JSON configuration instead of exposing the API key in clear text.

  • If your SAP instance doesn’t require authentication, leave all boxes empty.

"Items" tab

On the Items tab, you can specify how the source handles items based on their file type or content type.

File types

File types let you define how the source handles items based on their file extension or content type. For each file type, you can specify whether to index the item content and metadata, only the item metadata, or neither.

You should fine-tune the file type configurations with the objective of indexing only the content that’s relevant to your users.

Example

Your repository contains .pdf files, but you don’t want them to appear in search results. You click Extensions and then, for the .pdf extension, you change the Action by default and Action on error values to Ignore item.

For more details about this feature, see File type handling.

Content and images

If you want Coveo to extract text from image files or PDF files containing images, enable the appropriate option. The extracted text is processed as item data, meaning that it’s fully searchable and will appear in the item Quick view.

Note

When OCR is enabled, ensure the source’s relevant file type configurations index the item content. Indexing the item’s metadata only or ignoring the item will prevent OCR from being applied.

See Enable optical character recognition for details on this feature.

"Content security" tab

Select who will be able to access the source items through a Coveo-powered search interface. For details on the content security options, see Content security.

Note

If, while writing your source JSON configuration, you chose to index content access permissions and used the PermissionType parameter, you must select the Same users and groups as in your content system option and provide a JSON permission configuration detailing how to retrieve the relationships of each security identity and how to index this data.

"Access" tab

On the Access tab, specify whether each group (and API key, if applicable) in your Coveo organization can view or edit the current source.

For example, when creating a new source, you could decide that members of Group A can edit its configuration, while Group B can only view it.

For more information, see Custom access level.

Build the source

  1. Finish adding or editing your source:

    • When you’re done editing the source and want to make your changes effective, click Add and build source/Save and rebuild source.

    • When you want to save your source configuration changes without starting a build/rebuild, such as when you know you want to make other changes soon, click Add source/Save. On the Sources (platform-ca | platform-eu | platform-au) page, click Launch build or Launch rebuild when you’re ready to make your changes effective and index your content.

  2. On the Sources (platform-ca | platform-eu | platform-au) page, follow the progress of your source addition or modification.

  3. Once the source is built or rebuilt, review its content in the Content Browser.

Index metadata

To use metadata values in search interface facets or result templates, the metadata must be mapped to fields. Coveo automatically maps only a subset of the metadata it extracts. You must map any additional metadata to fields manually.

Note

Not clear on the purpose of indexing metadata? Watch this video.

  1. On the Sources (platform-ca | platform-eu | platform-au) page, click your source, and then click More > View and map metadata in the Action bar.

  2. Review the default metadata that your source is extracting from your content.

  3. Map any currently not indexed metadata that you want to use in facets or result templates to fields.

    1. Click the metadata and then, at the top right, click Add to index.

    2. In the Apply a mapping on all item types of a source panel, select the field you want to map the metadata to, or add a new field if none of the existing fields are appropriate.

      Note

      For advanced mapping configurations, like applying a mapping to a specific item type, see Manage mappings.

    3. Click Apply mapping.

  4. Return to the Sources (platform-ca | platform-eu | platform-au) page.

  5. To reindex your source with your new mappings, click your source, and then click More > Rebuild in the Action bar.

  6. Once the source is rebuilt, review your item field values. They should now include the values of the metadata you selected to index.

    1. On the Sources (platform-ca | platform-eu | platform-au) page, click your source, and then click More > Open in Content Browser in the Action bar.

    2. Select the card of the item for which you want to inspect properties, and then click Properties in the Action bar.

    3. In the panel that appears, select the Fields tab.

Change the source request rate

Your source makes requests to an API to index the desired content. By default, these requests are made immediately one after the other. If your API throttles Coveo’s requests, you can increase the delay between each request.

To do so, in the source JSON configuration (not the crawling configuration) parameters object, add the following code.

"RequestsIntervalInMs": {
  "value": "<NUMBER_OF_MILLISECONDS>" 1
  }
1 Replace <NUMBER_OF_MILLISECONDS> with the number of milliseconds Coveo should wait between each request. The default value is 0.

Required privileges

You can assign privileges to allow access to specific tools in the Coveo Administration Console. The following table indicates the privileges required to view or edit elements of the Sources (platform-ca | platform-eu | platform-au) page and associated panels. See Manage privileges and Privilege reference for more information.

Actions Service Domain Required access level

View sources, view source update schedules, and subscribe to source notifications

Content

Fields

View

Sources

View

Organization

Organization

View

Edit sources, edit source update schedules, and edit source mappings

Content

Fields

Edit

Sources

Edit

Organization

Organization

View

View and map metadata

Content

Fields

View

Source metadata

View

Sources

Edit

Organization

Organization

View
Notes

  • The Edit all privilege isn’t required to create sources. When granting privileges for the Sources domain, you can grant a group or API key the View all or Custom access level, instead of Edit all, and then select the Can create checkbox to allow users to create sources. See Can create ability dependence for more information.

  • Granting the Edit access level on a domain also implicitly grants View on that domain.

Ignoring "no first page" errors in subitems

When indexing your content, Coveo may encounter an HTTP error. By default, Coveo stops the crawling process when it encounters such an error.

However, you can configure your source to ignore specific errors and continue indexing.

Similarly, when requesting subitems from your API, Coveo will stop the indexing process if your API returns a 404 error rather than the first page of results. A 404 error on the first page prevents Coveo from indexing any of your subitems, as the missing first page contains the information needed to request the second page, such as a cursor or the URL of the next page.

However, you can configure your source to ignore this error and continue indexing. It will therefore finish indexing the other items of the same endpoint, including their subitems if the API returns valid result pages. Then, your source will move on to any other endpoint you’ve defined.

To ignore "no first page" errors in subitems

  1. On the Sources (platform-ca | platform-eu | platform-au), select your source, and then, in the More menu, click Edit configuration with JSON.

  2. In your source JSON configuration, in the parameters object, add the following:

    "SkipNoFirstPageErrorsInSubItems": {
  "value": "true"
},

  3. Click Save and rebuild source.

This parameter applies to all subitem requests made by your source.

What’s next?