REST API Source Tutorial

This is for:

In this article

Prerequisites
Step 1: Plan the JSON Configuration Structure
Step 2: Generate Authentication Credentials
Step 3: Write the Source JSON Configuration
Step 4: Create the REST API source
Step 5: Map Your Source Fields
Step 6: Build Your Source and Configure Your Search Page

In this REST API source tutorial, you’ll create a REST API cloud source retrieving content from Vimeo, a video-sharing platform. You’ll index a Vimeo user’s videos and the comments associated to each video.

When working on your REST API source configuration, you may also want to refer to the following articles:

Prerequisites

For this tutorial, you need:

A Coveo organization in which:
- REST API sources are allowed
- The Sources limit has not been reached
A Coveo organization member account with the required privileges.
A Vimeo Developer account and a Vimeo application.

Step 1: Plan the JSON Configuration Structure

Depending on the content you want to retrieve and how you want it organized in Coveo, you might need to write a JSON configuration with objects nested within others. This requires you to have a good understanding of the content repository structure and to determine the hierarchy according to which your endpoint calls should be made.

To write a JSON configuration allowing Coveo to gather all the desired metadata and to present it the desired way, you should plan the configuration structure beforehand.

Browse the target web application API documentation to grasp how the repository is organized (see Repository Structure). If the web application requires authentication, check the related documentation as well. Ensure to meet any API call prerequisites.

The Vimeo API offers a single service, which in turn provides an endpoint per item type (for example, user, video, category, channel, etc.). You can then make calls (for example, add, delete, edit, get, etc.) against the endpoint items.

You must authenticate to Vimeo using an OAuth 2.0 access token (API key). Step 2 focuses on creating this access token.
Depending on the content you want to make searchable, determine how you need to structure your calls to the endpoints.

This tutorial follows the use case of indexing videos uploaded by the user account Natural History Museum Denmark. For the purposes of this tutorial, assume that you want the videos in your Coveo search result page to be displayed with a list of their comments underneath (see About Result Folding).

Example

You should therefore call the Users endpoint, then the Videos endpoint as a user sub-item, and then the Videos endpoint again as a video sub-item. As a result, your JSON configuration should make three calls and have the following structure:
- Vimeo service
  - Users endpoint (Get a user)
    
    Videos endpoint (Get all the videos that a user has uploaded)
    
    Videos endpoint (Get all the comments on a video)

Step 2: Generate Authentication Credentials

If your web application API requires authentication, gather the required credentials.

In Vimeo, you can generate an API key using the user interface (see Getting Started with the Vimeo API):

Log in to Vimeo Developer.
At the top of the page, click My Apps.
On your API Apps page, click your application.
In your application menu, under Authentication, click Generate Access Token.
Under Authentication, click Authenticated (you), and then click Generate.
Take note of your access token, as it will only be fully displayed once. You’ll need to provide this key in the Add a REST API panel, under API key authentication, as well as in Postman while preparing your source JSON configuration.

Step 3: Write the Source JSON Configuration

While writing your JSON configuration, you must make calls to the target endpoints of the web application API using an application such as Postman. The JSON responses contain properties you must refer to in the configuration. Use the REST API Source Reference page as a guide to build your JSON configuration.

Before you write your source JSON configuration, browse the available fields on the Fields (platform-ca | platform-eu | platform-au) page. You can use fields that are already used by other sources or create new ones for your source (see Step 5: Map your Source Fields).

Note

The Vimeo API JSON responses below are truncated. To view the full JSON response, make the calls using Postman.

Services

Start your configuration with the service properties. In this scenario, since Vimeo offers a single service, this will be your only service configuration.

  "Services": [
    {
    "Url": "https://api.vimeo.com/",
    "Headers": {
      "Authorization": "Bearer @ApiKey"
    },
    "Paging": {
      "PageSize": 25,
      "OffsetStart": 1,
      "OffsetType": "page",
      "Parameters": {
        "Limit": "per_page",
        "Offset": "page"
      }
    }
    },
    ...
  ]

With this paging configuration, Coveo will fetch 25 items per page, starting at page 1, when calling the API. This isn’t relevant for the first call, when you expect only one user to be returned, but it will be appropriate for the two Videos endpoint calls, which will return many videos and comments (see Inheritable Properties).

User Profile

Make an API call to the Users endpoint to retrieve Natural History Museum Denmark's user profile.

API call

 GET https://api.vimeo.com/users/user12339310/videos HTTP/1.1

 Accept: application/json
 Authorization: Bearer **********-****-****-****-************

200 OK JSON response body (truncated)

 {
   "uri": "/users/12339310",
   "name": "Natural History Museum Denmark",
   "link": "https://vimeo.com/user12339310",
   "location": "Copenhagen",
   "bio": "Welcome to the Natural History Museum of Denmark's Vimeo site.",
   "websites": [
     {
       "name": "Our YouTube Channel",
       "link": "https://www.youtube.com/user/NaturalHistoryDK",
       "description": null
     }
   ],
   "metadata": {
     "connections": {
       "followers": {
         "uri": "/users/12339310/followers",
         "options": [
           "GET"
         ],
         "total": 310
       },
       "videos": {
         "uri": "/users/12339310/videos",
         "options": [
           "GET"
         ],
         "total": 93
       },
       ...
     }
   }
   ...
 }

Using the values provided in the JSON response, add the endpoint properties to the source configuration, that is, the Coveo fields that you want to fill and the corresponding values to index. Often you’ll want to use dynamic values instead of hardcoded values, especially when retrieving more than one item (see About Fields).

Note

If you want to retrieve metadata for which there’s no Coveo field yet, you’ll eventually have to create and map the desired field (see About Fields).

You must also ensure to provide authentication information by referring to the API key you’ll later provide in the Add a REST API source panel.

You add the following to the source JSON configuration:
```
 "Endpoints": [
   {
     "Path": "users/user12339310",
     "Method": "GET",
     "ItemType": "User",
     "Uri": "%[coveo_url]/%[uri]",
     "ClickableUri": "%[coveo_url]/%[uri]",
     "Title": "%[name]",
     "Body": "%[bio]",
     "Metadata": {
       "filetype": "User",
       "name": "%[name]",
       "website": "%[websites[*].link]"
     }
     ...
   }
 ]
 ...
```
- Path: You want to index content from a single user (user ID user12339310), so you can hardcode the Path value.
- ItemPath is omitted because the item is at the root of the response.
- Uri: the field value consists of two dynamic values. Coveo indexes the address resulting from this combination.
- Title and name: Dynamic values for fields that refer to properties found in the JSON response. Since only one user is indexed, you could also hardcode these values. However, entering dynamic values ensures that if the value in the JSON response ever changes, Coveo will be up to date after the next source rescan.
- filetype: If you plan on using the File Type facet on your search page, provide the filetype metadata. Since the File Type facet is based on this field, omitting it will cause display issues in the Content Browser and your search page.
- websites[*].link: Whenever a property is associated to an array, you must specify which of the object properties in the array you want to retrieve, even if there’s only one object. As a result, even if there’s only one link property under websites, you must specify that you want to index all ([*]) link values.

Videos

Make an API call to the Videos endpoint.

API call

GET https://api.vimeo.com/users/user12339310/videos HTTP/1.1

Accept: application/json
Authorization: Bearer **********-****-****-****-************

200 OK JSON response body (truncated)

 {
   "total": 93,
   "page": 1,
   "per_page": 25,
   "paging": {
     "next": "/users/user12339310/videos?page=2",
     "previous": null,
     "first": "/users/user12339310/videos?page=1",
     "last": "/users/user12339310/videos?page=4"
   },
   "data": [
     {
       "uri": "/videos/325417533",
       "name": "SUKKANIUNNEQ / THE RACE",
       "description": "On the western coast of Greenland, a young boy dreams of winning the national championship of dog sledding, an ancient form of transportation which has turned into a highly competitive sport.",
       "type": "video",
       "link": "https://vimeo.com/325417533",
       "duration": 535,
       "created_time": "2019-03-20T12:25:23+00:00",
       "modified_time": "2019-12-13T10:01:00+00:00",
       "release_time": "2019-03-20T12:25:23+00:00",
       "pictures": {
         "uri": "/videos/325417533/pictures/769728361",
       },
       "metadata": {
         "connections": {
           "comments": {
             "uri": "/videos/325417533/comments",
             "options": [
               "GET"
             ],
             "total": 8
           },
           "likes": {
             "uri": "/videos/325417533/likes",
             "options": [
               "GET"
             ],
             "total": 193
           }
         }
       }
     },
     ...
   ]
 }

Add the corresponding configuration based on the information provided in the JSON response (see Get all the videos that a user has uploaded).

In this case, since you want Coveo to consider videos as sub-items of a user, you must enter the Videos endpoint object as a sub-item of the Users endpoint.

You add the following to the source JSON configuration:

 ...
 "SubItems": [
   {
   "Path": "users/user12339310/videos",
   "Method": "GET",
   "ItemPath": "data",
   "ItemType": "Video",
   "Uri": "%[coveo_url]/%[uri]",
   "ClickableUri": "%[coveo_url]/%[uri]",
   "Title": "%[name]",
   "Body": "%[description]",
   "Metadata": {
     "date": "%[created_time]",
     "modifieddate": "%[modified_time]",
     "filetype": "Video",
     "videotitle": "%[name]",
     "author": "%[coveo_parent.name]",
     "foldingcollection": "%[uri]",
     "foldingitemid": "%[uri]",
     "foldingparentvideo": "%[uri]"
   },
   ...
   }
 ]
 ...

ItemPath indicates that the items to retrieve are objects in the data array.
foldingcollection, foldingitemid, and foldingparentvideo: the fields will be used by your search page to determine which comments are children of which video and to display them accordingly (see About Result Folding).

Comments

Make another API call to the Videos endpoint (see Get all the comments on a video). This time, you must provide a video_id as a query parameter. In your last JSON response, look for the uri of any video; the numbers after /videos/ are the video ID.

The video of which you use the ID doesn’t matter, as all responses for this call have an identical structure. Moreover, the video data isn’t relevant in your configuration; to build your source JSON configuration, you need the name of the JSON response properties, while their value is irrelevant.

API call

 GET https://api.vimeo.com/videos/325417533/comments HTTP/1.1

 Accept: application/json
 Authorization: Bearer **********-****-****-****-************

200 OK JSON response body (truncated)

 {
   "total": 4,
   "page": 1,
   "per_page": 25,
   "paging": {
     "next": null,
     "previous": null,
     "first": "/videos/325417533/comments?page=1",
     "last": "/videos/325417533/comments?page=1"
   },
   "data": [
     {
       "uri": "/videos/325417533/comments/17433345",
       "type": "video",
       "text": "Beautiful Video! Thank You so much for sharing!",
       "created_on": "2019-04-30T02:25:09+00:00",
       "user": {
         "uri": "/users/5818471",
         "name": "Wolfmaan",
         "link": "https://vimeo.com/wolfmaan",
         "location": "Ontario Canada"
       },
       "metadata": {
         "connections": {
           "replies": {
             "uri": "/videos/325417533/comments/17433345/replies",
             "options": [
               "GET",
               "POST"
             ],
             "total": 1
           }
         }
       }
     },
     ...
   ]
 }

Using the values provided in the JSON response, write the source configuration determining how you want to index the comment items (see Endpoints). Since you want Coveo to consider comments as sub-items of a video, you must enter the comment endpoint object as a sub-item of the Videos endpoint.

You add the following to the source JSON configuration:

 ...
 "SubItems": [
   {
     "SkippableErrorCodes": "401",
     "Path": "%[coveo_parent.raw.metadata.connections.comments.uri]",
     "Method": "GET",
     "ItemType": "Comment",
     "ItemPath": "data",
     "Uri": "%[coveo_url]%[uri]",
     "ClickableUri": "%[coveo_url]/%[uri]",
     "Title": "%[name]",
     "Body": "%[description]",
     "Metadata": {
       "filetype": "Comment",
       "commenttext": "%[text]",
       "author": "%[user.name]",
       "date": "%[created_on]",
       "foldingcollection": "%[coveo_parent.raw.uri]",
       "foldingitemid": "%[uri]",
       "foldingparentvideo": "%[coveo_parent.raw.uri]"
     }
   }
 ]
 ...

Since you didn’t retrieve the content of metadata.connections.comments.uri in your previous sub-item metadata, you must now add raw to retrieve the data for the Path object.

Step 4: Create the REST API source

In the Coveo Administration Console, create your REST API cloud source.

Save your source without building it, as you need to make sure all the fields you mention in your configuration are mapped.

Step 5: Map Your Source Fields

Add mappings for the fields you mentioned in your source configuration. If you used fields that don’t already exist in your organization, you can add them via the Apply a Mapping on All/Specific Item Types of a Source panel.

Step 6: Build Your Source and Configure Your Search Page

Once the fields to be used by your REST API source are all mapped, you can build your source. On the Sources (platform-ca | platform-eu | platform-au) page, click Launch build in the source Status column to add the source content.

If you chose to enable result folding, see Folding results and the Folding results - Example for implementation guidelines and examples. Folded comments under video search results should look as follows.