Add or Edit a Generic REST API Source

Index Content Coveo Relevance Cloud Developer System Administrator Product Documentation

Coveo has dedicated connectors for many web and on-premises systems, therefore allowing you to quickly make application content searchable. See the Connector Directory for the full list.

However, there may be applications of which you want to index the content, but for which there’s no dedicated connector. In such a case, when you have the required privileges, you can use a Generic REST API source to retrieve and make the desired content searchable with Coveo.

A Generic REST API source allows you to crawl content from a remote repository exposing its data through a REST API. When creating your source, you must provide a JSON REST configuration instructing Coveo to retrieve items from the repository REST services and their respective resource endpoints (see Concepts, Reference, JSON Configuration Examples, and Tutorial). This configuration indicates which API calls to execute to fetch the desired items, how to parse the responses to extract relevant metadata, and which content type these items represent.

Example

You have valuable content in a content management system (CMS) developed in-house. One of your developers can create a crawler to get this content and push it to your Coveo organization.

Tip
Leading practice

The number of items that a source processes per hour (crawling speed) depends on various factors, such as network bandwidth and source configuration. See About Crawling Speed for information on what can impact crawling speed, as well as possible solutions.

Source Key Characteristics

Features Supported Additional information

Content update operations

Refresh

check

For each endpoint you define in your source configuration, you can provide a refresh endpoint to override the initial endpoint. When you do so, the connector can add, update, or delete specific items in the index instead of refreshing the entire repository. However, the following limitations currently apply:

  • Only refresh queries using the date of the last refresh operation can be made. Tokens can’t be used.

  • A sub-item can’t be refreshed if its parent item isn’t detected as modified.

Rescan

check

Takes place every day by default.

Rebuild

check

Content security options

Same users and groups as in your content system

check

If your source configuration includes the PermissionType parameter, you must provide a JSON configuration detailing how to extract the relationships of the indexed permissions.

Specific users and groups

check

Everyone

check

Add or Edit a Generic REST API Source

When adding a source, in the Add a source of content panel, click the Cloud (Cloud icon) or the Crawling Module (Crawling Module icon) tab, depending on whether your content is located in the cloud or on a server you manage. With the latter, you must install the Coveo On-Premises Crawling Module to retrieve your content. See Content Retrieval Methods for details.

To edit a source, on the Sources (platform-eu | platform-au) page, click the desired source, and then click Edit in the Action bar.

The completion steps are especially important when creating or editing a source of this type.

"Configuration" Tab

On the Add/Edit a Generic REST API Source subpage, the Configuration tab is selected by default. It contains your source’s general and authentication information, as well as other parameters.

General Information

Source Name

Enter a name for your source.

Tip
Leading practice

A source name can’t be modified once it’s saved, therefore be sure to use a short and descriptive name, using letters, numbers, hyphens (-), and underscores (_). Avoid spaces and other special characters.
Paired Crawling Module

If your source is a Crawling Module source, and if you have more than one Crawling Module linked to this organization, select the one with which you want to pair your source. If you change the Crawling Module instance paired with your source, a successful rebuild is required for your change to apply.

Optical Character Recognition (OCR)

If you want Coveo to extract text from image files or PDF files containing images, check the appropriate box. OCR-extracted text is processed as item data, meaning that it’s fully searchable and will appear in the item Quick View. See Enable Optical Character Recognition for details on this feature.

Note

Contact Coveo Sales to add this feature to your organization license.

"Authentication" Section

In the Authentication section, all parameters are optional. Fill the appropriate boxes depending on the authentication type used by the source you want to make searchable.

  • If your source uses a HTTP, Basic, Kerberos, or NTLM authentication protocol, enter the Username and Password of the account with which you want to crawl the source. This fills the username and password fields in your source JSON configuration. The account of which you enter the credentials must have access to all the content that you want to make searchable. See Source Credentials Leading Practices.

  • If your source uses the OAuth 2.0 authentication protocol, enter your content source Client ID, Client secret and Refresh token in the corresponding boxes.

  • If your source uses an API key to authenticate, enter it in the API key box.

  • If your source doesn’t require authentication, leave all boxes empty.

"Content to Include" Section

In the JSON configuration box, enter your source JSON configuration.

For more information on the Generic REST API source JSON configuration, see:

"Content Security" Tab

Select who will be able to access the source items through a Coveo-powered search interface. For details on this parameter, see Content Security.

Note

If, while writing your source JSON configuration, you chose to index content access permissions and used the PermissionType parameter, you must select the Same users and groups as in your content system option and provide a JSON permission configuration detailing how to retrieve the relationships of each security identity and how to index this data.

"Access" Tab

In the Access tab, set whether each group and API key can view or edit the source configuration (see Resource Access):

  1. If available, in the left pane, click Groups or API Keys to select the appropriate list.

  2. In the Access Level column for groups or API keys with access to source content, select View or Edit.

Completion

  1. Click Add Source/Save to add/save your source configuration.

  2. While writing your JSON configuration, you may have decided to populate fields that aren’t default fields. If you have not already created these fields for another source, you must create them in the Fields (platform-eu | platform-au) page before building your source.

    Examples

    • You decided to retrieve picture URIs and to have Coveo populate the pictureuri field with this data. Your item metadata therefore contains:

        "pictureuri": "%[picture.uri]"

      However, since the pictureuri field isn’t a default field like author or date, you must create it.

    • You have another Generic REST API source populating the custom field facebookaccountid. When creating your new source, you therefore don’t need to create this field, as it’s already in the Fields (platform-eu | platform-au) page.

  3. Ensure that your source correctly maps all the fields to populate. If a field doesn’t have a mapping, you must create one.

    Example

    You map the pictureuri field with the following rule: %[pictureuri].

  4. On the Sources (platform-eu | platform-au) page, you must click Launch build or Launch rebuild in the source Status column to add the source content or to make your changes effective, respectively.

What’s Next?

Schedule source updates.