Add or Edit a Generic REST API Source

The Coveo Cloud platform has dedicated connectors for many web and on-premises systems, thus allowing you to quickly make application content searchable (see Available Coveo Cloud V2 Connectors). However, there may be applications of which you want to include the content in Coveo Cloud, but for which there is no dedicated connector. In such a case, administrators and content managers can use a Generic REST API source to retrieve and make the desired content searchable in Coveo Cloud.

A Generic REST API source allows you to crawl content from a remote repository exposing its data through a REST API. When creating your source, you must provide a JSON REST configuration instructing Coveo Cloud to retrieve items from the repository REST services and their respective resource endpoints (see Generic REST API Source Reference, Generic REST API Source JSON Configuration Examples, and Generic REST API Source Tutorial). This configuration indicates which API calls to execute to fetch the desired items, how to parse the responses to extract relevant metadata, and which content type these items represent.

Source Feature Summary

Add or Edit a Generic REST API Source

1. If not already in the Add/Edit a Generic REST API Source panel, go to the panel:

   • To add a source, in the main menu, under Content, select Sources > Add Source button > Generic REST API.

     OR

   • To edit a source, in the main menu, under Content, select Sources > source row > Edit in the Action bar.

2. In the Configuration tab, enter appropriate values for the available parameters:

   • Source name

     A descriptive name for your source content. The name must be under 255 characters (not already in use for another source in this organization).

   • Character optical recognition (OCR)

     Check this box if you want Coveo Cloud to extract text from image files and/or PDF files containing images (see Enable Optical Character Recognition). OCR-extracted text is processed as item data, meaning that it is fully searchable and will appear in the item Quick View (see Search Result Quick View).

     Since the OCR feature is available at an extra charge, you must first contact Coveo Sales to add this feature to your organization license. You can then enable it for your source.

   • Index

     When adding a source, if you have more than one logical (non-Elasticsearch) index in your organization, select the index in which the retrieved content will be stored (see Leverage Multiple Coveo Indexes). If your organization only has one index, this drop-down menu is not visible and you have no decision to make.

     • To add a source storing content in an index different than default, you need the View access level on the Logical Index domain (see Privilege Management and Logical Indexes Domain).

     • Once the source is added, you cannot switch to a different index.

   • Content security

     Select who can see items from this source in a search interface that includes this source in its scope (see Content Security). Your options are:

     • Shared: Everyone will be able to find the source content in the search interface.

     • Private: Only the source creator will be able to find the source content in the search interface. They must however authenticate to the search interface with the identity with which they created the source.

3. In the Authentication section, all parameters are optional. Fill the appropriate boxes depending on the authentication type used by the source you want to make searchable.

   • If your source uses a HTTP, Basic, Kerberos, or NTLM authentication protocol, enter the Username and Password of the account with which you want to crawl the source. This fills the username and password fields in your source JSON configuration. The account of which you enter the credentials must have access to all the content that you want to make searchable.

     If the repository allows it, it is recommended to use a dedicated crawling account instead of your own account.

   • If your source uses the OAuth 2.0 authentication protocol, enter your content source Client ID, Client secret and Refresh token in the corresponding boxes.

   • If your source uses an API key to authenticate, enter it in the API key box.

   • If your source does not require authentication, leave all boxes empty.

4. In the Content to Include section, in the JSON configuration box, enter your source JSON configuration (see Generic REST API Source Reference and Generic REST API Source Tutorial).

   • If your sources uses an API key authentication protocol, make sure to include the @APIkey placeholder in the HTTP headers, query parameters, or payload parameters (see Authentication and Generic REST API Source JSON Configuration Examples).

     

   • If your source uses the OAuth 2.0 authentication protocol, make sure to include the OAuth 2.0 configuration with the @ClientID, @ClientSecret, and @RefreshToken placeholders (see Authentication and OAuth 2.0).

5. In the Access tab, determine whether each group and API key can view or edit the source configuration (see Understanding Resource Access):
   1. In the Access Level column, select View or Edit for each available group.
   2. On the left-hand side of the tab, if available, click Groups or API Keys to switch lists.

   If you remove the Edit access level from all the groups of which you are a member, you will not be able to edit the source again after saving. Only administrators and members of other groups that have Edit access on this resource will be able to do so. To keep your ability to edit this resource, you must grant the Edit access level to at least one of your groups.

6. Optionally, consider editing or adding mappings (see Adding and Managing Source Mappings).

   You can only manage mapping rules once you build the source (see Refresh, Rescan, or Rebuild Sources).

7. Click Add Source/Save to add/save your source configuration.

8. If you selected the Create an API key check box:

   1. In the Your Key dialog box that appears, click Copy to copy the API key value to the clipboard, and click OK.

      The Your Key dialog box appears only once and is the only place where you can see and copy the key value.

      If you fail to copy (or lose) the key value, the workaround is to manually create a new API key by selecting only the View and Edit check boxes for the Sources privileges of the Content service (see Add an API Key). In this case, you should also disable or delete the original unused API key (see Disable/Enable an API Key or Delete an API Key).

      An API key is an alphanumeric string that looks like the following:

      xx7fdcf07a-85af-446f-8992-6cd2d5ecc53c

      The key appears on the administration console API Access page and its name is your source name followed by API key.

   2. Paste the API key value with the source name to a safe location of your choice so you can later securely communicate this information to the person who manages the crawling process.

9. Optionally, review the source update schedule.

   By default, your Generic REST API source is rescanned daily to make your source content changes searchable (additions, modifications, or deletions) (see Refresh VS Rescan VS Rebuild). If your content changes very frequently, consider decreasing the rescan time interval (such as every hour or every few hours) to optimize resource consumption (see Edit a Source Schedule).

10. While writing your JSON configuration, you may have decided to populate fields that are not Coveo Cloud default fields. If you have not already created these fields for another source, you must create them in the Fields page before building your source (see Add or Edit Fields and Field Origins).

    • You decided to retrieve picture URIs and to have Coveo Cloud populate the pictureuri field with this data. Your item metadata therefore contains:

        "pictureuri": "%[picture.uri]"
      

      However, since the pictureuri field is not a default field like author or date, you must create it.

    • You have another Generic REST API source populating the custom field facebookaccountid. When creating your new source, you therefore do not need to create this field, as it is already in the Fields page.

11. Ensure that your source maps correctly all the fields to populate (see Adding and Managing Source Mappings). If a field does not have a mapping, you must create one.

    You map the pictureuri field with the following rule: %[pictureuri].

12. On the Sources page, you must click Start initial build or Start required rebuild in the source Status column to add the source content or make your changes effective, respectively.

    Once the source is built or rebuilt, you can review its content in the Content Browser (see Content Browser - Page).