Generic REST API Source JSON Configuration Examples

This topic presents examples of how you can configure your Generic REST API source to match the content you want to index. You can edit the source JSON configuration from the Coveo Cloud administration console (see Add/Edit a Generic REST API Source).

See also Indexing Discourse with a Generic REST API Source and Generic REST API Source Tutorial for larger examples.

JSON Responses and Corresponding REST Configurations

The following examples show the REST configuration that one could provide to match the format of the JSON response returned by the repository API.

For the sake of clarity, all these examples use the same base REST API URL (http://example.com/api/v1) and do not have any query parameters (other than pagination-specific parameters under Paging) or headers.

Minimal Configuration

Minimal configuration with one service and one endpoint.

JSON response Corresponding REST configuration 
[
  {
    "id": 1,
    "name": "alice",
    "updated": "2016-11-14 11:29:49"
  },
  {
    "id": 2,
    "name": "bob",
    "updated": "2016-11-15 13:51:24"
  },
  {
    "id": 3,
    "name": "jack",
    "updated": "2016-11-15 13:59:44"
  },
  {
    "id": 4,
    "name": "eve",
    "updated": "2016-11-15 16:09:08"
  },
  {
    "id": 5,
    "name": "isaac",
    "updated": "2016-11-16 17:02:35"
  }
]
 
{
  "Services": [
    {
      "Url": "http://example.com/api/v1",
      "Endpoints": [
        {
          "Path": "/users",
          "Method": "GET",
          "ItemType": "User",
          "Uri": "%[coveo_url]/users/%[id]",
          "ClickableUri": "%[coveo_url]/users/%[id]",
          "Title": "%[name]",
          "ModifiedDate": "%[updated]",
          "Metadata": {
            "id": "%[id]",
            "username": "%[name]"
          }
        }
      ]
    }
  ]
}

Paging

  • Paging configuration at the service level (see Services Paging and the Paging property table). The offset is based on page numbers.

    JSON response Corresponding REST configuration 
    [
    {
      "id": 1,
      "name": "alice",
      "updated": "2016-11-14 11:29:49"
    },
    {
      "id": 2,
      "name": "bob",
      "updated": "2016-11-15 13:51:24"
    },
    {
      "id": 3,
      "name": "jack",
      "updated": "2016-11-15 13:59:44"
    },
    {
      "id": 4,
      "name": "eve",
      "updated": "2016-11-15 16:09:08"
    },
    // Example truncated.
    // There are actually 500 users in this collection!
    {
      "id": 500,
      "name": "isaac",
      "updated": "2016-11-16 17:02:35"
    }
  ]
    		 
    {
    "Services": [
      {
        "Url": "http://example.com/api/v1",
        "Paging": {
          "PageSize": 20,         // Each page contains 20 items
          "OffsetStart": 0,       // The first page is numbered 0
          "OffsetType": "page",   // Offset are based on page numbers, and
                                  // not on total item count
          "Parameters": {
            "Limit": "limit",     // Name of the 'limit' query parameter
            "Offset": "page"      // Name of the 'offset' query parameter
          }
        },
        "Endpoints": [
          {
            "Path": "/users",
            "Method": "GET",
            "ItemType": "User",
            "Uri": "%[coveo_url]/users/%[id]",
            "ClickableUri": "%[coveo_url]/users/%[id]",
            "Title": "%[name]",
            "ModifiedDate": "%[updated]",
            "Metadata": {
              "id": "%[id]",
              "username": "%[name]"
            }
          }
        ]
      }
    ]
  }

  • Paging configuration at the service level (see Services Paging and the Paging property table). The offset page is based on page cursor (token).

    JSON response Corresponding REST configuration 
    {"nextPageToken": "3d170d80-7b3b-4371-9499-2ec6a78c507e&","items": [
    {
      "id": 1,
      "name": "alice",
      "updated": "2016-11-14 11:29:49"
    },
    {
      "id": 2,
      "name": "bob",
      "updated": "2016-11-15 13:51:24"
    },
    {
      "id": 3,
      "name": "jack",
      "updated": "2016-11-15 13:59:44"
    },
    {
      "id": 4,
      "name": "eve",
      "updated": "2016-11-15 16:09:08"
    },
    {
      "id": 5,
      "name": "isaac",
      "updated": "2016-11-16 17:02:35"
    }
  ]
  }
    		 
    {
    "Services": [
      {
        "Url": "http://example.com/api/v1",
        "Paging": {
          "PageSize": 5,         // Each page contains 5 items
          "OffsetType": "cursor",   // Offset are based on cursor
          "NextPageKey": "nextPageToken"
           "Parameters": {
            "Limit": "limit",     // Name of the 'limit' query parameter
            "Offset": "pageToken"      // Name of the 'offset' query parameter
          }
        },
        "Endpoints": [
          {
            "Path": "/users",
            "Method": "GET",
            "ItemType": "User",
            "Uri": "%[coveo_url]/users/%[id]",
            "ClickableUri": "%[coveo_url]/users/%[id]",
            "Title": "%[name]",
            "ModifiedDate": "%[updated]",
            "Metadata": {
              "id": "%[id]",
              "username": "%[name]"
            }
          }
        ]
      }
    ]
  }

Collection of Items

If the items of an endpoint are not at the root of the JSON response, you must specify the path to the array of items using the ItemPath property (see ItemPath). In the configuration below, the ItemPath property indicates that the items are in the entries[0].results array.

JSON response Corresponding REST configuration 
{
  "count": 4,
  "entries": [
    {
      "results": [
        {
          "id": 1,
          "name": "item #1"
        },
        {
          "id": 2,
          "name": "item #2"
        },
        {
          "id": 3,
          "name": "item #3"
        },
        {
          "id": 4,
          "name": "item #4"
        }
      ]
    }
  ]
}
 
{
  "Services": [
    {
      "Url": "http://example.com/api/v1",
      "Endpoints": [
        {
          "Path": "/items",
          "Method": "GET",
          "ItemPath": "entries[0].results",
          "ItemType": "Item",
          "Uri": "%[coveo_url]/items/%[id]",
          "ClickableUri": "%[coveo_url]/items/%[id]",
          "Title": "%[name]",
          "Metadata": {
            "id": "%[id]"
          }
        }
      ]
    }
  ]
}

JSON Response Object Containing a Single Item

You can use the Metadata property to retrieve item metadata provided in the JSON response (see Metadata).

JSON response Corresponding REST configuration 
{
  "id": 1,
  "firstname": "Loyd",
  "lastname": "Sweeney",
  "age": 26,
  "address": {
    "streetAddress": "6974 Rutrum Dr",
    "city": "Ashburn",
    "state": "VA",
    "zip": "25581"
  },
  "email": "JWeakliem@lectus.org",
  "username": "GHohmann",
  "phone": "9556676434",
  "created": "2009-09-12T05:29:48.688Z"
}
 
{
  "Services": [
    {
      "Url": "http://example.com/api/v1",
      "Endpoints": [
        {
          "Path": "/users/1",
          "Method": "GET",
          "ItemType": "User",
          "Uri": "%[coveo_url]/users/%[id]",
          "ClickableUri": "%[coveo_url]/users/%[id]",
          "Title": "%[username]",
          "ModifiedDate": "%[created]",
          "Metadata": {
            "id": "%[id]",
            "firstname": "%[firstname]",
            "lastname": "%[lastname]",
            "age": "%[age]",
            "address": "%[address]",
            "email": "%[email]",
            "phonenumber": "%[phone]"
          }
        }
      ]
    }
  ]
}

In the configuration above, the address metadata contains the whole JSON object. So, to flatten the metadata, you could also write the following configuration:

"Metadata": {
  "id": "%[id]",
  "firstname": "%[firstname]",
  "lastname": "%[lastname]",
  "age": "%[age]",
  "address.street": "%[address.streetAddress]",
  "address.city": "%[address.city]",
  "address.state": "%[address.state]",
  "address.zip": "%[address.zip]",
  "email": "%[email]",
  "phonenumber": "%[phone]"
}

IndexingAction

With this configuration, although 100 posts are available, only the first one will be indexed: all posts with an id greater than 1 are ignored (see IndexingAction).

{
  "Services": [
    {
      "Url": "https://jsonplaceholder.typicode.com",
      "Endpoints": [
        {
          "Path": "/posts",
          "Method": "GET",
          "ItemType": "Post",
          "Uri": "%[coveo_url]/posts/%[id]",
          "ClickableUri": "%[coveo_url]/posts/%[id]",
          "IndexingAction": {
            "ActionOnItem": "Ignore",
            "Condition": "%[id] > 1"
          },
          "Title": "%[title]",
          "Body": "%[body]",
          "Metadata": {
            "id": "%[id]",
            "userid": "%[userId]"
          }
        }
      ]
    }
  ]
}

API Key Authentication

When the web application to make searchable requires you to authenticate using an API key, there are different ways to provide the key in your source JSON configuration, depending on the application API.

As there is no standard for API key authentication, there is no standard way to provide the key in the Generic REST API source. You must therefore manually configure the HTTP headers, query parameters, or payload parameters. Below are examples of the most common API key authentication methods and the corresponding JSON configuration to provide when creating your Generic REST API source (see Add/Edit a Generic REST API Source). You can use the @APIKey mapping to retrieve the API key you entered in the Add a Generic REST API Source panel.

API Key in the HTTP Authorization Header

Bearer is the authorization type and is most commonly used with OAuth 2.0 tokens.

API call Corresponding REST configuration 
GET /users HTTP/1.1
Host: http://example.com/api/v1
Authorization: Bearer Az4adfSyBhZX8fTThUJ4Kb2TC3Ax0RshgrZtUiqM
[other HTTP headers truncated]
 
{
  "Services": [
    {
      "Url": "http://example.com/api/v1",
      "Endpoints": [
        {
          "Path": "/users",
          "Method": "GET",
          "ItemType": "User",
          "Uri": "%[coveo_url]/users/%[id]",
          "ClickableUri": "%[coveo_url]/users/%[id]",
          "Headers": {
            "Authorization": "Bearer @ApiKey"
          }
        }
      ]
    }
  ]
}

API Key in a Custom HTTP Header

API call Corresponding REST configuration 
GET /users HTTP/1.1
Host: http://example.com/api/v1
X-Api-Key: Az4adfSyBhZX8fTThUJ4Kb2TC3Ax0RshgrZtUiqM
[other HTTP headers truncated]
 
{
  "Services": [
    {
      "Url": "http://example.com/api/v1",
      "Endpoints": [
        {
          "Path": "/users",
          "Method": "GET",
          "ItemType": "User",
          "Uri": "%[coveo_url]/users/%[id]",
          "ClickableUri": "%[coveo_url]/users/%[id]",
          "Headers": {
            "X-Api-Key": "@ApiKey"
          }
        }
      ]
    }
  ]
}

Not all custom HTTP headers contain the X- prefix, as this practice was deprecated in 2012.

API Key as a Query Parameter

API call Corresponding REST configuration 
GET /users?api_key=Az4adfSyBhZX8fTThUJ4Kb2TC3Ax0RshgrZtUiqM HTTP/1.1
Host: http://example.com/api/v1
[other HTTP headers truncated]
 
{
  "Services": [
    {
      "Url": "http://example.com/api/v1",
      "Endpoints": [
        {
          "Path": "/users",
          "Method": "GET",
          "ItemType": "User",
          "Uri": "%[coveo_url]/users/%[id]",
          "ClickableUri": "%[coveo_url]/users/%[id]",
          "QueryParameters": {
            "api_key": "@ApiKey"
          }
        }
      ]
    }
  ]
}

API Key as a Payload Parameter

API call Corresponding REST configuration 
POST /users HTTP/1.1
Host: http://example.com/api/v1
[other HTTP headers truncated]
api_key=Az4adfSyBhZX8fTThUJ4Kb2TC3Ax0RshgrZtUiqM
 
{
  "Services": [
    {
      "Url": "http://example.com/api/v1",
      "Endpoints": [
        {
          "Path": "/users",
          "Method": "GET",
          "ItemType": "User",
          "Uri": "%[coveo_url]/users/%[id]",
          "ClickableUri": "%[coveo_url]/users/%[id]",
          "PayloadParameters": {
            "api_key": "@ApiKey"
          }
        }
      ]
    }
  ]
}

Complete Example

The REST configuration below results in the execution of these two queries:

  • GET http://example.com/api/v1/posts?type=post&expand=true&id=120 (along with the two specified headers)

  • POST http://example.com/api/v1/users?expand=true (along with the two specified headers)

{
  "Services": [
    {
      "Url": "http://example.com/api/v1",
      "Authentication": {
        "Username": "admin",
        "Password": "password",
        "ForceBasicAuthentication": true
      },
      "Endpoints": [
        {
          "Path": "/posts",
          "Method": "GET",
          "Paging": {
            "PageSize": 10,
            "OffsetStart": 0,
            "OffsetType": "page",
            "Parameters": {
              "Limit": "limit",
              "Offset": "page"
            }
          },
          "Headers": {
            "key": "value",
            "authentication": "basic"
          },
          "QueryParameters": {
            "id": "120",
            "type": "user",
            "expand": true
          },
          "ItemType": "Post",
          "Uri": "%[coveo_url]/repository/posts/%[id]",
          "ClickableUri": "%[coveo_url]/posts/%[id]",
          "Title": "%[title]",
          "ModifiedDate": "%[updated]",
          "Body": "%[html.content]",
          "Metadata": {
            "author": "%[author]",
            "id": "%[id]"
          }
        },
        {
          "Path": "/users",
          "Method": "POST",
          "ItemPath": "results",
          "PayloadParameters": {
            "id": "120",
            "type": "user",
            "expand": true
          },
          "ItemType": "User",
          "Uri": "%[coveo_url]/repository/users/%[id]",
          "ClickableUri": "%[coveo_url]/users/%[id]",
          "Title": "%[name]",
          "ModifiedDate": "%[updated]",
          "Metadata": {
            "id": "%[id]",
            "name": "%[name]"
          }
        }
      ]
    }
  ]
}

Realistic Configuration Examples

The following JSON configuration examples are realistic configurations. They could be entered with few changes in the Add a Generic REST API source panel to create a source (see Add/Edit a Generic REST API Source). If you do so, you can then open the indexed items in the Content Browser to see how the content retrieved by the JSON configuration appears the Coveo Cloud V2 administration console (see Content Browser - Page).

Blog Posts and the Corresponding Comments as Sub-Items

This example uses the JSONPlaceholder REST API in place of that of a blog.

{
   "Services": [
      {
         "Url": "https://jsonplaceholder.typicode.com",
         "Endpoints": [
            {
               "Path": "/posts",
               "Method": "GET",
               "ItemType": "Post",
               "Uri": "%[coveo_url]/posts/%[id]",
               "ClickableUri": "%[coveo_url]/posts/%[id]",
               "Title": "%[title]",
               "Body": "%[body]",
               "Metadata": {
                  "id": "%[id]",
                  "userid": "%[userId]"
               },
               "SubItems":[
                  {
                     "Path": "/posts/%[coveo_parent.id]/comments",
                     "Method": "GET",
                     "ItemType": "Comment",
                     "Uri": "%[coveo_url]/posts/%[coveo_parent.id]/comments/%[id]",
                     "ClickableUri": "%[coveo_url]/comments/%[id]",
                     "Title": "%[name]",
                     "Body": "%[body]",
                     "Metadata": {
                        "id": "%[id]",
                        "postid": "%[postId]",
                        "name": "%[name]",
                        "email": "%[email]"
                     }
                  }
               ]
            }
         ]
      }
   ]
}

Photos with a Binary Body

This example uses the JSONPlaceholder REST API in place of that of an online photo album.

{
   "Services": [
      {
         "Url": "https://jsonplaceholder.typicode.com",
         "Endpoints": [
            {
               "Path": "/photos",
               "Method": "GET",
               "ItemType": "Photo",
               "Uri": "%[coveo_url]/photos/%[id]",
               "ClickableUri": "%[url]",
               "Title": "%[title]",
               "Body": "%[content]",
               "Metadata": {
                  "id": "%[id]",
                  "albumid": "%[albumId]",
                  "thumbnailurl": "%[thumbnailUrl]",
                  "url": "%[url]"
               },
               "SubQueries": [
                  {
                     "Path": "%[coveo_parent.url]",
                     "Method": "GET",
                     "IsBinaryBody": true
                  }
               ]
            }
         ]
      }
   ]
}

ServiceNow Incidents and Knowledge Articles

{
   "Services": [
      {
         "Url": "https://YOUR_INSTANCE.service-now.com/api/now/",
         "Authentication": {
            "ForceBasicAuthentication": "true",
            "Username": "admin",
            "Password": "PASSWORD_HERE"
         },
         "Endpoints": [
            {
               "Path": "/table/incident",
               "Method": "GET",
               "ItemPath": "result",
               "ItemType": "Incident",
               "Uri": "%[coveo_url]/incident/%[sysid]",
               "ClickableUri": "https://YOUR_INSTANCE.service-now.com/nav_to.do?uri=incident.do?sys_id=%[sysid]%26sysparm_view=ess",
               "Title": "%[short_description]",
               "Body": "%[description]",
               "Paging": {
                  "PageSize": 100,
                  "OffsetStart": 0,
                  "OffsetType": "item",
                  "Parameters": {
                     "Limit": "sysparm_limit",
                     "Offset": "sysparm_offset"
                  }
               },
               "Metadata": {
                  "sysid": "%[sys_id]",
                  "number": "%[number]",
                  "closecode": "%[close_code]",
                  "closenotes": "%[close_notes]",
                  "resolvedat": "%[resolved_at]",
                  "description": "%[description]",
                  "category": "%[category]"
               },
               "SubQueries": [
                  {
                     "Path": "%[coveo_parent.raw.opened_by.link]",
                     "Method": "GET",
                     "Metadata": {
                        "openedbyname": "%[result.name]",
                        "openedbyemail": "%[result.email]"
                     }
                  }
               ]
            },
            {
               "Path": "/table/kb_knowledge",
               "Method": "GET",
               "ItemPath": "result",
               "ItemType": "KB",
               "Uri": "%[coveo_url]/kb/%[sysid]",
               "ClickableUri": "https://YOUR_INSTANCE.service-now.com/nav_to.do?uri=incident.do?sys_id=%[sysid]%26sysparm_view=ess",
               "Title": "%[short_description]",
               "Body": "%[text]",
               "Paging": {
                  "PageSize": 50,
                  "OffsetStart": 0,
                  "OffsetType": "item",
                  "Parameters": {
                     "Limit": "sysparm_limit",
                     "Offset": "sysparm_offset"
                  }
               },
               "Metadata": {
                  "sysid": "%[sys_id]",
                  "number": "%[number]",
                  "description": "%[description]",
                  "published": "%[published]",
                  "meta": "%[meta]",
                  "topic": "%[topic]",
                  "category": "%[category]"
               },
               "SubQueries": [
                  {
                     "Path": "%[coveo_parent.raw.author.link]",
                     "Method": "GET",
                     "Metadata": {
                        "authorname": "%[result.name]",
                        "authoremail": "%[result.email]"
                     }
                  }
               ]
            }
         ]
      }
   ]
}

Videos of a Vimeo User

Before you can create a Generic REST API source to index Vimeo content, you must create a Vimeo application to generate a token (see Getting Started).

{
   "Services": [
      {
         "Url": "https://api.vimeo.com/",
         "Paging": {
            "PageSize": 25,
            "OffsetStart": 1,
            "OffsetType": "page",
            "Parameters": {
               "Limit": "per_page",
               "Offset": "page"
            }
         },
         "Endpoints": [
            {
               "Headers": {
                  "Authorization": "Bearer YOURACCESSTOKEN"
               },
               "Path": "/users/USERNAME/videos",
               "Method": "GET",
               "ItemPath": "data",
               "ItemType": "VimeoVideo",
               "Uri": "%[link]",
               "ClickableUri": "%[link]",
               "Title": "%[name]",
               "ModifiedDate": "%[modified_time]",
               "Body": "%[description]",
               "Metadata": {
                  "ytembedplayerhtml": "%[embed.html]",
                  "link": "%[link]",
                  "name": "%[name]",
                  "author": "%[user.name]",
                  "documenttype": "Video",
                  "filetype": "VimeoVideo",
                  "ytlikecount": "%[metadata.connections.likes.total]",
                  "ytthumbnailurl": "%[pictures.sizes[1].link]",
                  "ytcommentcount": "%[metadata.connections.comments.total]",
                  "ytvideoduration": "%[duration]",
                  "description": "%[description]"
               }
            }
         ]
      }
   ]
}

DynamoDB Content

Before you can create a Generic REST API source to index DynamoDB content, you must create a REST service to your DynamoDB Create Amazon API Gateway (see Using Amazon API Gateway as a proxy for DynamoDB).

{
   "Services": [
      {
         "Url": "https://YOUR_AMAZON_REST_API_URL",
         "Endpoints": [
            {
               "Headers": {
                  "contenttype": "application/json"
               },
               "QueryParameters": {
                  "table": "GEOCODE"
               },
               "Path": "PATH_TO_YOUR_REST_SERVICE",
               "Method": "GET",
               "ItemPath": "Items",
               "ItemType": "Database",
               "Uri": "https://mydb/%[city]",
               "ClickableUri": "https://mydb/%[city]",
               "Title": "%[city]",
               "ModifiedDate": "",
               "Body": "%[city]",
               "Metadata": {
                  "filetype": "Database",
                  "city": "%[city]"
               }
            }
         ]
      }
   ]
}

GitHub Content

{
  "Services": [
    {
      "Url": "https://api.github.com/",
      "Endpoints": [
        {
          "Headers": {
            "Authorization": "token",
            "User-Agent": "MyIndexer"
          },
          "Path": "orgs/myorg/repos",
          "Method": "GET",
          "ItemType": "GitHubOrg",
          "Uri": "%[html_url]",
          "ClickableUri": "%[html_url]",
          "Title": "%[name]",
          "SubItems": [
            {
              "Headers": {
                "Authorization": "token",
                "User-Agent": "MyIndexer"
              },
              "Path": "repos/myrepo/%[coveo_parent.raw.name]/branches/%[coveo_parent.raw.default_branch]",
              "Method": "GET",
              "ItemType": "GitHubRepo",
              "Uri": "%[commit.url]",
              "ClickableUri": "%[commit.url]",
              "Title": "%[name] (%[coveo_parent.raw.name])",
              "ModifiedDate": "",
              "SubItems": [
                {
                  "Headers": {
                    "Authorization": "token",
                    "User-Agent": "MyIndexer"
                  },
                  "Path": "repos/myrepo/%[coveo_parent.coveo_parent.raw.name]/git/trees/%[coveo_parent.raw.commit.sha]",
                  "QueryParameters": {
                    "recursive": "1"
                  },
                  "Method": "GET",
                  "ItemPath": "tree",
                  "ItemType": "GitHubItem",
                  "Uri": "%[coveo_parent.raw._links.html]",
                  "ClickableUri": "%[coveo_parent.raw._links.html]",
                  "Title": "%[path] (%[coveo_parent.coveo_parent.raw.name])",
                  "ModifiedDate": "%[coveo_parent.raw.commit.commit.committer.date]",
                  "Body": "%[content]",
                  "SubQueries": [
                    {
                      "Headers": {
                        "Authorization": "token",
                        "User-Agent": "MyIndexer"
                      },
                      "Path": "https://raw.githubusercontent.com/myusername/%[coveo_parent.coveo_parent.coveo_parent.raw.name]/%[coveo_parent.coveo_parent.coveo_parent.raw.default_branch]/%[coveo_parent.raw.path]",
                      "Method": "GET",
                      "IsBinaryBody": true,
                    }
                  ],
                  "Metadata": {
                    "path": "%[path]",
                    "documenttype": "GitHubItem",
                    "filetype": "txt",
                    "author": "%[coveo_parent.raw.commit.commit.author.name]",
                  }
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}