Add a REST API source

This is for:

Developer System Administrator

Coveo has dedicated connectors for many web and on-premises systems, therefore allowing you to quickly make application content searchable. See the Connector Directory for the full list.

However, there may be applications of which you want to index the content, but for which there’s no dedicated connector. In such a case, when you have the required privileges, you can use a REST API source to retrieve and make the desired content searchable with Coveo.

A REST API source allows you to crawl content from a remote repository exposing its data through a REST API. When creating your source, you must provide a JSON REST configuration instructing Coveo to retrieve items from the repository REST services and their respective resource endpoints. This configuration indicates which API calls to execute to fetch the desired items, how to parse the responses to extract relevant metadata, and which content type these items represent.

When working on your REST API source configuration, you may also want to refer to the following articles:

Example

Your company uses Vimeo to host videos. You want to make these videos, along with their comments, accessible through your intranet’s search page. So, you create a REST API source that will query Vimeo’s API to index the desired content.

Tip
Leading practice

The number of items that a source processes per hour (crawling speed) depends on various factors, such as network bandwidth and source configuration. See About crawling speed for information on what can impact crawling speed, as well as possible solutions.

Source key characteristics

The following table presents the main characteristics of a REST API source.

Features Supported Additional information

Content update operations

refresh

check

To enable the refresh capability, add a RefreshEndpoints array for each endpoint in your source configuration.

rescan

check

Takes place every day by default.

rebuild

check

Content security options

Same users and groups as in your content system

check

If your source configuration includes the PermissionType parameter, you must provide a JSON configuration detailing how to extract the relationships of the indexed permissions.

Specific users and groups

check

Everyone

check

metadata indexing for search

Automapping of metadata to a field with a matching name

Enabled by default.

Automatically indexed metadata

Sample of autopopulated fields (no user-defined metadata required):

  • date

  • language (autodetected from Body object content)

  • size
     

Sample of autopopulated fields (source configuration parameter of matching name required):

After a content update operation, inspect your item field values in the Content Browser.

Collected indexable metadata

Parameters specified in the source configuration Metadata object that don’t match an existing field name are collected, but not indexed.

After a rebuild, review the View and map metadata subpage for the list of indexed metadata and to index additional metadata from those available.

Commerce requirements

When using a REST API source to index commerce-specific content, such as products, variants, and availabilities, you have to undergo a catalog configuration process to benefit from all commerce-related capabilities.

More specifically, you must:

Enable Coveo Personalization-as-you-go

Coveo Machine Learning tools include Coveo Personalization-as-you-go (PAYG) capabilities for commerce use cases. This suite of advanced features learns from a user’s intent and reacts within a few clicks. PAYG models require the building of a product vector space to represent the products contained in your source. For REST API sources, Coveo PAYG needs to be enabled in order to produce the product vector space. Contact your Coveo representative to discuss your options.

Catalog configuration

Behind the scenes, the REST API source uses the Stream API to push content to the Coveo index. Therefore, REST API sources must be associated with a catalog entity to ensure a complete configuration. This allows the source to accurately build a product vector space.

For instructions on how to create a catalog entity, see Commerce catalog entity.

Add a REST API Source

Follow the instructions below to add a REST API source using the desired content retrieval method.

  1. On the Sources (platform-ca | platform-eu | platform-au) page, click Add source.

  2. In the Add a source of content panel, click the Cloud (cloud-icon) or Crawling Module (crawlingmodule) tab, depending on your content retrieval context. With the latter, you must install the Crawling Module to make your source operational.

  3. Click the REST API tile.

  4. Configure your source.

The completion steps are especially important when creating a source of this type.

Tip
Leading practice

It’s best to create or edit your source in your sandbox organization first. Once you’ve confirmed that it indexes the desired content, you can copy your source configuration to your production organization, either with a snapshot or manually.

See About non-production organizations for more information and best practices regarding sandbox organizations.

"Configuration" tab

In the Add a REST API Source panel, the Configuration tab is selected by default. It contains your source’s general and authentication information, as well as other parameters.

General information

Source name

Enter a name for your source.

Tip
Leading practice

A source name can’t be modified once it’s saved, therefore be sure to use a short and descriptive name, using letters, numbers, hyphens (-), and underscores (_). Avoid spaces and other special characters.
Paired Crawling Module

If your source is a Crawling Module source, and if you have more than one Crawling Module linked to this organization, select the one with which you want to pair your source. If you change the Crawling Module instance paired with your source, a successful rebuild is required for your change to apply.

Optical character recognition (OCR)

If you want Coveo to extract text from image files or PDF files containing images, enable the appropriate option.

The extracted text is processed as item data, meaning that it’s fully searchable and will appear in the item Quick view. See Enable optical character recognition for details on this feature.

Project

Use the Project selector to associate your source with one or more Coveo projects.

"Authentication" section

In the Authentication section, all parameters are optional. Fill the appropriate boxes depending on the authentication type used by the source you want to make searchable.

  • If your source uses a HTTP, Basic, Kerberos, or NTLM authentication protocol, enter the Username and Password of the account with which you want to crawl the source. Then, use the @Username and @Password placeholders in your source JSON configuration instead of exposing the credentials in clear text. The account of which you enter the credentials must have access to all the content that you want to make searchable. See Source Credentials Leading Practices.

  • If your source uses the OAuth 2.0 authentication protocol, enter your content source Client ID, Client secret and Refresh token in the corresponding boxes. Then, use the @ClientID, @ClientSecret, and @RefreshToken placeholders in your source JSON configuration instead of exposing this information in clear text.

  • If your source uses an API key to authenticate, enter it in the API key box. Then, use the @ApiKey placeholder in your source JSON configuration instead of exposing the API key in clear text.

  • If your source doesn’t require authentication, leave all boxes empty.

"Content to include" section

In the JSON configuration box, enter your source JSON configuration. This configuration instructs Coveo on how to retrieve items from your API’s REST services and their respective resource endpoints.

You can start from our Basic example and adapt them to your needs. For example, you could configure your source to benefit from the authentication, paging and refresh capabilities.

When working on your REST API source configuration, you may also want to refer to the following articles:

"Content security" tab

Select who will be able to access the source items through a Coveo-powered search interface. For details on this parameter, see Content security.

Note

If, while writing your source JSON configuration, you chose to index content access permissions and used the PermissionType parameter, you must select the Same users and groups as in your content system option and provide a JSON permission configuration detailing how to retrieve the relationships of each security identity and how to index this data.

"Access" tab

In the Access tab, specify whether each group (and API key, if applicable) in your Coveo organization can view or edit the current source.

For example, when creating a new source, you could decide that members of Group A can edit its configuration, while Group B can only view it.

For more information, see Custom access level.

Completion

  1. Click Add Source/Save to add/save your source configuration.

  2. While writing your JSON configuration, you may have decided to populate fields that aren’t default fields. If you have not already created these fields for another source, you must create them in the Fields (platform-ca | platform-eu | platform-au) page before building your source.

    Examples

    • You decided to retrieve picture URIs and to have Coveo populate the pictureuri field with this data. Your item metadata therefore contains:

        "pictureuri": "%[picture.uri]"

      However, since the pictureuri field isn’t a default field like author or date, you must create it.

    • You have another REST API source populating the custom field facebookaccountid. When creating your new source, you therefore don’t need to create this field, as it’s already in the Fields (platform-ca | platform-eu | platform-au) page.

  3. Ensure that your source correctly maps all the fields to populate. In sources created before January 10, 2024, if a field doesn’t have a mapping, you must create one. In sources created after that date, Coveo will automatically populate fields whose name exactly matches a metadata key.

    Example

    In your content, profile picture URIs are stored under profilepic. If a pictureuri fields exists in your organization, you’ll create the following mapping to populate the pictureuri field with picture URIs: %[profilepic].

    However, if a profilepic field already exists in your organization, Coveo will automatically populate it with the profilepic value extracted from your content.

  4. On the Sources (platform-ca | platform-eu | platform-au) page, you must click Launch build or Launch rebuild in the source Status column to add the source content or to make your changes effective, respectively.

Limitation

When indexing content with the Crawling Module, ensure not to change space character encoding in an item’s URI, as Coveo uses URIs to distinguish items.

For example, an item whose URI would change from example.com/my first item to example.com/my%20first%20item wouldn’t be recognized as the same by Coveo. As a result, it would be indexed twice, and the older version wouldn’t be deleted.

Item URIs are displayed in the Content Browser (platform-ca | platform-eu | platform-au). We recommend you check where these URIs come from before making changes that affect space character encoding. Depending on your source type, the URI may be an item’s URL, or it may be built out of pieces of metadata by your source mapping rules. For example, your item URIs may consist of the main site URL plus the item filename, due to a mapping rule such as example.com/%[filename]. In such a case, changing space encoding in the item filename could impact the URI.

Change the source request rate

Your source makes requests to an API to index the desired content. By default, these requests are made immediately one after the other. If Coveo’s requests reach the throttling limit of your API, you can increase the delay between each request.

To do so, in the source JSON configuration (not the crawling configuration) parameters object, add the following code.

"RequestsIntervalInMs": {
  "value": "<NUMBER_OF_MILLISECONDS>" 1
  }
1 Replace <NUMBER_OF_MILLISECONDS> with the number of milliseconds Coveo should wait between each request. The default value is 0, that is, there is no delay between requests.

Ignoring "no first page" errors in subitems

When indexing your content, Coveo may encounter an HTTP error. By default, Coveo stops the crawling process when it encounters such an error.

However, you can configure your source to ignore specific errors and continue indexing.

Similarly, when requesting subitems from your API, Coveo will stop the indexing process if your API returns a 404 error rather than the first page of results. A 404 error on the first page prevents Coveo from indexing any of your subitems, as the missing first page contains the information needed to request the second page, such as a cursor or the URL of the next page.

However, you can configure your source to ignore this error and continue indexing. It will therefore finish indexing the other items of the same endpoint, including their subitems if the API returns valid result pages. Then, your source will move on to any other endpoint you’ve defined.

To ignore "no first page" errors in subitems

  1. On the Sources (platform-ca | platform-eu | platform-au), select your source, and then, in the More menu, click Edit configuration with JSON.

  2. In your source JSON configuration, in the parameters object, add the following:

    "SkipNoFirstPageErrorsInSubItems": {
  "value": "true"
},

  3. Click Save and rebuild source.

This parameter applies to all subitem requests made by your source.

Required privileges

You can assign privileges to allow access to specific tools in the Coveo Administration Console. The following table indicates the privileges required to view or edit elements of the Sources (platform-ca | platform-eu | platform-au) page and associated panels. See Manage privileges and Privilege reference for more information.

Note

The Edit all privilege isn’t required to create sources. When granting privileges for the Sources domain, you can grant a group or API key the View all or Custom access level, instead of Edit all, and then select the Can Create checkbox to allow users to create sources. See Can Create ability dependence for more information.
Actions Service Domain Required access level

View sources, view source update schedules, and subscribe to source notifications

Content

Fields

View

Sources

Organization

Organization

Edit sources, edit source update schedules, and edit source mappings

Organization

Organization

View

Content

Fields

Edit

Sources

View and map metadata

Content

Source metadata

View

Fields

Organization

Organization

Content

Sources

Edit

What’s next?

Schedule source updates.