Add or Edit a Generic REST API Source

The Coveo Cloud Platform has dedicated connectors for many web and on-premises systems, thus allowing you to quickly make application content searchable (see Available Connectors). However, there may be applications of which you want to include the content in Coveo Cloud, but for which there’s no dedicated connector. In such a case, members of the Administrators and Content Managers built-in groups can use a Generic REST API source to retrieve and make the desired content searchable in Coveo Cloud.

A Generic REST API source allows you to crawl content from a remote repository exposing its data through a REST API. When creating your source, you must provide a JSON REST configuration instructing Coveo Cloud to retrieve items from the repository REST services and their respective resource endpoints (see Concepts, Reference, JSON Configuration Examples, and Tutorial). This configuration indicates which API calls to execute to fetch the desired items, how to parse the responses to extract relevant metadata, and which content type these items represent.

You have valuable content in an on-premises content management system (CMS) developed in-house. One of your developers can create a crawler to get this content and push it to your Coveo Cloud organization.

Source Feature Summary

Features Supported Additional information
Content update Refresh For each endpoint you define in your source configuration, you can provide a refresh endpoint to override the initial endpoint. When you do so, the connector can add, update, or delete specific items in the index instead of refreshing the entire repository. However, the following limitations currently apply:
  • Only refresh queries using the date of the last refresh operation can be made. Tokens can't be used.
  • A sub-item can't be refreshed if its parent item isn't detected as modified.
Rescan
Rebuild
Content security options Determined by source permissions You can define permissions in the source JSON configuration at the service or endpoint level. However, this source allows for only one security identity provider. All members specified in the PermissionsSets JSON object will be mapped to this provider.
You can also define permissions at the source level, but the same security identity provider restriction applies.
Source creator
Everyone

Add or Edit a Generic REST API Source

Follow the instructions below when adding or editing a Generic REST API source. The completion steps are especially important when creating a source of this type.

“Configuration” Tab

In the Add/Edit a Generic REST API Source subpage, the Configuration tab is selected by default. It contains your source general and authentication information, as well as other parameters.

General Information

Source Name

Enter a descriptive name for your source.

Use a short and descriptive name, using letters, numbers, - and _ characters, and avoid spaces and other special characters.

Character Optical Recognition (OCR)

Check this box if you want Coveo Cloud to extract text from image files or PDF files containing images. OCR-extracted text is processed as item data, meaning that it’s fully searchable and will appear in the item Quick View.

Since the OCR feature is available at an extra charge, you must first contact Coveo Sales to add this feature to your organization license. You can then enable it for your source.

Index

When adding a source, if you have more than one logical (non-Elasticsearch) index in your organization, select the index in which the retrieved content will be stored (see Leverage Many Coveo Indexes). If your organization only has one index, this drop-down menu isn’t visible and you have no decision to make.

  • To add a source storing content in an index different than default, you need the View access level on the Logical Index domain (see Privilege Management and Logical Indexes Domain).

  • Once the source is added, you can’t switch to a different index.

“Authentication” Section

In the Authentication section, all parameters are optional. Fill the appropriate boxes depending on the authentication type used by the source you want to make searchable.

  • If your source uses a HTTP, Basic, Kerberos, or NTLM authentication protocol, enter the Username and Password of the account with which you want to crawl the source. This fills the username and password fields in your source JSON configuration. The account of which you enter the credentials must have access to all the content that you want to make searchable.

    If the repository allows it, we recommend that you use a dedicated crawling account instead of your own account.

  • If your source uses the OAuth 2.0 authentication protocol, enter your content source Client ID, Client secret and Refresh token in the corresponding boxes.

  • If your source uses an API key to authenticate, enter it in the API key box.

  • If your source doesn’t require authentication, leave all boxes empty.

“Content to Include” Section

In the JSON configuration box, enter your source JSON configuration.

For more information on the Generic REST API source JSON configuration, see:

“Content Security” Tab

In the Content Security tab, select who will be able to access the source items through a Coveo-powered search interface. For details on this parameter, see Content Security.

“Access” Tab

In the Access tab, determine whether each group and API key can view or edit the source configuration (see Understanding Resource Access):

  1. In the Access Level column, select View or Edit for each available group.

  2. On the left-hand side of the tab, if available, click Groups or API Keys to switch lists.

If you remove the Edit access level from all the groups of which you’re a member, you won’t be able to edit the source again after saving. Only administrators and members of other groups that have Edit access on this resource will be able to do so. To keep your ability to edit this resource, you must grant the Edit access level to at least one of your groups.

Completion

  1. Click Add Source/Save to add/save your source configuration.

  2. While writing your JSON configuration, you may have decided to populate fields that aren’t Coveo Cloud default fields. If you have not already created these fields for another source, you must create them in the Fields page before building your source.

    • You decided to retrieve picture URIs and to have Coveo Cloud populate the pictureuri field with this data. Your item metadata therefore contains:

        "pictureuri": "%[picture.uri]"

      However, since the pictureuri field isn’t a default field like author or date, you must create it.

    • You have another Generic REST API source populating the custom field facebookaccountid. When creating your new source, you therefore don’t need to create this field, as it’s already in the Fields page.

  3. Ensure that your source maps correctly all the fields to populate. If a field doesn’t have a mapping, you must create one.

    You map the pictureuri field with the following rule: %[pictureuri].

  4. On the Sources page, you must click Start initial build or Start required rebuild in the source Status column to add the source content or make your changes effective, respectively.

What’s Next?

  • Once the source is built or rebuilt, you can review its content in the Content Browser .

  • Review the source update schedule. By default, your Generic REST API source is rescanned daily to make your source content changes searchable (additions, modifications, or deletions). If your content changes very frequently, consider decreasing the rescan time interval (such as every hour or every few hours) to optimize resource consumption.

Recommended Articles