---
title: REST API source concepts
slug: '3131'
canonical_url: https://docs.coveo.com/en/3131/
collection: index-content
source_format: adoc
---
# REST API source concepts
Coveo has dedicated [connectors](https://docs.coveo.com/en/2734/) for many web and on-premises systems, therefore allowing you to quickly make application content searchable.
See the [Connector directory](https://docs.coveo.com/en/1702/) for the full list.
However, there may be applications of which you want to index the content, but for which there's no dedicated connector.
In such a case, when you have the required privileges, you can use a [generic API connector](https://docs.coveo.com/en/1702#generic-connectors) to retrieve and make the desired content searchable with Coveo.
Many web applications offer a public API that developers can use to leverage the application in their own software.
Coveo takes advantage of such an API to call the application and therefore retrieve its content.
A REST API source allows you to index content from [repositories](https://docs.coveo.com/en/2739/) exposing their data through a REST API.
When [creating your REST API source](https://docs.coveo.com/en/1896), you must provide a JSON configuration allowing Coveo to retrieve content [items](https://docs.coveo.com/en/210/).
This configuration indicates which API calls to execute to fetch the desired items, how to parse the responses to extract relevant metadata, and what type of resources these items represent.
While the [Reference](https://docs.coveo.com/en/1525/) article details the elements to use in a JSON configuration, this article explains some basic concepts that apply to any REST API source and its JSON configuration.
It describes the typical [structure of a repository](#repository-structure) to index, as well as how to use [dynamic values](#dynamic-values) to retrieve your content and how JSON objects properties can be [passed down](#inheritable-properties) to child objects in your JSON configuration.
When working on your REST API source configuration, you may also want to refer to the following articles:
* [Add a REST API source](https://docs.coveo.com/en/1896/)
* [Source configuration reference](https://docs.coveo.com/en/1525/)
* [Permission configuration reference](https://docs.coveo.com/en/3303/)
* [JSON configuration examples](https://docs.coveo.com/en/1996/)
* [Tutorial](https://docs.coveo.com/en/2030/)
## Repository structure
The typical remote repository consists of services, endpoints, and items arranged in a hierarchical fashion.
Each service contains one or more resource endpoints, and each of these endpoints represents a type of item to fetch, such as user profiles, web pages, files, etc.
While this structure applies to most repositories, the item types a repository contains vary.
For example, a video-sharing website such as [Vimeo](https://vimeo.com/) provides not only video items, but also user profiles, channels, groups, etc., while a customer service management system may have support cases and knowledge articles as item types.
When you [create REST API source](https://docs.coveo.com/en/1896/) in the Coveo Administration Console, you must provide a JSON source configuration listing the services and endpoints to crawl.
This JSON configuration must also indicate which API calls to execute to fetch the desired items and how to parse the responses to extract relevant [metadata](https://docs.coveo.com/en/218/).
## Dynamic values
Dynamic values are metadata values acting as placeholders in your REST API source JSON configuration and defining what the Coveo [index](https://docs.coveo.com/en/204/) [fields](https://docs.coveo.com/en/200/) will contain for each item.
Dynamic values are to be replaced with metadata or content from the JSON response returned by the API.
See [REST API source tutorial](https://docs.coveo.com/en/2030/) for examples.
The syntax to use to leverage a dynamic value in your [source JSON configuration](https://docs.coveo.com/en/1896/) is typically `"CoveoFieldName": "%[DynamicValue]"`.
However, if your dynamic value contains whitespace characters, the syntax to use is slightly different.
See [Whitespace characters](#whitespace-characters) for details.
A dynamic value can consist of one or more of the following:
* [`coveo_url`](#coveo_url), a placeholder for your [service](https://docs.coveo.com/en/1525#services-array-required) URL.
* A [JSON path](#json-path) leading to a property specified earlier in your JSON configuration or in the JSON response returned by the API.
* [`coveo_parent`](#coveo_parent), which precedes a JSON path and indicates that this JSON path refers to a property of the parent of an item.
* [`raw`](#raw), an indication inserted between `coveo_parent` and a JSON path, allows you to retrieve a value in the API JSON response that previously provided you with metadata regarding the parent of an item.
A dynamic value can also act as a placeholder for [more than one field value](#multi-value-fields).
For example, if the API returns several tags associated with a blog article, you can use a dynamic value to index some or all of them in the Coveo `tags` field.
However, if a piece of metadata is constant across all items of an endpoint, you don't need a dynamic value.
You can hardcode your value, that is, enter it as a static value that applies to all items.
**Example**
All posts on your blog are in English only.
The value of the `language` field therefore doesn't vary and can be hardcoded: `"language": "English"`.
Conversely, if you have posts both in English and Spanish, the `language` value will vary.
You will therefore need to use a [JSON path](#json-path) dynamic value: `"language": "%[lang]"`.
### `coveo_url`
`coveo_url` is a placeholder for your [service](https://docs.coveo.com/en/1525#services-array-required) URL.
**Example**
You write the following configuration to retrieve user profiles.
Your service URL is `+https://myapplication.com+`.
In addition, user profile URIs are built by a dynamic value consisting of your service URL, `/users/`, and another dynamic value representing the user ID.
As a result, in Coveo, for user profile items, the `Uri` field will be populated with `+https://myapplication.com/users/+`, followed by the corresponding user ID.
```json
{
"Services":[
{
"Url":"https://myapplication.com",
"Endpoints":[
{
"Path":"users/%[userid]",
"Method":"GET",
"ItemType":"User",
"Uri":"%[coveo_url]/users/%[userid]",
"ClickableUri":"%[link]",
"Title":"%[name]",
"Body":"%[bio]",
"Metadata":{
"datebirth":"%[birthday]",
"city":"%[location]"
},
}
]
}
]
}
```
### JSON path
To build your source JSON configuration, you should first make the API calls that you want Coveo to make to retrieve your content.
In the JSON responses, you'll see how the desired data is organized and labeled, and you can then use this information to write a JSON path that directs the Coveo crawler to the data you want to retrieve.
**What if my API returns an XML response?**
Details
Some APIs such as Azure's Blob Storage REST API return XML responses instead of JSON responses.
Coveo's REST API source does support XML responses, which it converts to JSON using the [Newtonsoft.Json library](https://www.newtonsoft.com/json).
Therefore, when writing your source JSON configuration, you can still use JSON paths to index the data in your XML response.
When writing your JSON paths, we recommend you convert your API's XML response to JSON using Newtonsoft.Json.
Recreating the JSON response that Coveo parses will help you write your JSON paths with the right syntax.
Let's use [this sample from the Newtonsoft.Json documentation](https://www.newtonsoft.com/json/help/html/ConvertXmlToJson.htm) to illustrate the process.
If your API returns the following XML response:
```xml
@"
Alan
http://www.google.com
Louis
http://www.yahoo.com
"
```
Coveo will convert it to the following:
```json
{
"?xml": {
"@version": "1.0",
"@standalone": "no"
},
"root": {
"person": [
{
"@id": "1",
"name": "Alan",
"url": "http://www.google.com"
},
{
"@id": "2",
"name": "Louis",
"url": "http://www.yahoo.com"
}
]
}
}
```
As a result, your source JSON configuration could contain the following [`Endpoints` array](https://docs.coveo.com/en/1525#endpoints-array-required):
```json
"Endpoints": [
{
"Path": "/users",
"Method": "GET",
"ItemPath": "root.person",
"ItemType": "User",
"Uri": "%[coveo_url]/users/%[@id]",
"ClickableUri": "%[coveo_url]/users/%[@id]",
"Title": "%[name]",
"Metadata": {
"id": "%[@id]",
"url": "%[url]"
}
}
]
```
With a source indexing Azure Blob Storage content, your source configuration could look as follows:
```json
{
"Services": [
{
"Url": "https://azureopendatastorage.blob.core.windows.net",
"Headers": {
"x-ms-version": "2017-11-09"
},
"Paging": {
"OffsetType": "cursor",
"NextPageKey": "EnumerationResults.NextMarker",
"PageSize": 100,
"Parameters": {
"Offset": "marker",
"Limit": "maxresults"
}
},
"Endpoints": [
{
"Path": "/holidaydatacontainer",
"QueryParameters": {
"restype": "container",
"comp": "list"
},
"Method": "GET",
"ItemPath": "EnumerationResults.Blobs.Blob",
"ItemType": "HolidayBlob",
"Uri": "%[coveo_url]/holidaydatacontainer/%[Name]",
"ClickableUri": "%[coveo_url]/holidaydatacontainer/%[Name]",
"ModifiedDate": "%[Properties.Last-Modified]",
"Title": "%[Name]",
"Metadata": {
"url": "%[coveo_url]/holidaydatacontainer/%[Name]",
"etag": "%[Properties.Etag]",
"contenttype": "%[Properties.Content-Type]",
"blobtype": "%[Properties.BlobType]"
},
"SubQueries": [
{
"Path": "%[coveo_parent.url]",
"Method": "GET",
"IsBinaryBody": true
}
]
}
]
}
]
}
```
Details
You want to index a specific user profile.
When you request the user ID, your API returns the following response:
```json
{
"uri": "/users/4543466",
"userid": "4543466",
"firstname": "Andrew",
"lastname": "Price",
"link": "https://example.com/andrewprice",
"city": "Toronto",
"country": "Canada",
"department": "Parts",
"title": "Electrical Panel Specialist",
"bio": "Andrew Price joined the Parts department in 2008, after working as an Electrical Panel Specialist for a major car manufacturer. He has extensive experience in operations, steering and engines, and is highly skilled in diagnosing and repairing electrical problems. Andrew is a dedicated member of his team, and always puts the safety of his customers first.",
"pictureurl": "https://example.com/andrewprice/pic",
"contact": [
{
"name": "Slack",
"link": "@aprice",
},
{
"name": "Email",
"link": "aprice@barca.group",
}
]
}
```
You want Coveo to index the user's bio returned by the API and to display it as the body of the content item representing the user profile.
When writing your source JSON configuration, you associate the Coveo [fields](https://docs.coveo.com/en/1833/) with the property keys that represent the desired data in your API's response, using [JSONPath syntax](https://goessner.net/articles/JsonPath/index.html#e2) and [dynamic values](#dynamic-values).
For example, if you want the value of the `bio` property to be indexed by Coveo in the `Body` field, your source configuration should contain `"Body": "%[bio]"`.
The Coveo crawler will retrieve the content of the `bio` property in the API response, and store this information as the value of the Coveo `Body` field.
Your full source configuration could therefore look as follows:
```json
{
"Services": [
{
"Url": "http://example.com/api/v1",
"Authentication": {
"Username": "@Username",
"Password": "@Password",
"ForceBasicAuthentication": true
},
"Endpoints": [
{
"Path": "/users/",
"Method": "GET",
"ItemType": "People",
"Uri": "%[coveo_url]/users/%[userid]", <1>
"ClickableUri": "%[coveo_url]/users/%[userid]",
"Title": "%[firstname] %[lastname]", <2>
"Body": "%[bio]",
"Metadata": {
"id": "%[userid]",
"division": "%[department]",
"jobtitle": "%[title]",
"location": "%[city], %[country]",
"email": "%[contact[1].link]", <3>
"picture": "%[pictureurl]"
}
}
]
}
]
}
```
<1> This dynamic value acts as a placeholder for your service URL to build a full working URL.
For more details, see [`coveo_url`](#coveo_url).
<2> Coveo can index multiple pieces of data as a single value.
In this case, Coveo will index the first and last names in the `Title` field, which is dedicated to full names when indexing user profiles.
<3> Following the [JSONPath syntax](https://goessner.net/articles/JsonPath/index.html#e2), this value selects the value of the `link` property in the second object of the `contact` array.
With the information indexed by this source configuration, the user profile returned by your API could look as follows in your Coveo search results:
Details
The following is a truncated response example from a SmartSheet table.
In this response, each row corresponds to a project.
The cells of a row contain project data such as its name and status.
Each column contains a certain type of data regarding the projects in the table.
```json
{
"id": 952283761449296,
"name": "MY SMARTSHEET",
"permalink": "https://app.smartsheet.com/sheets/mC8mcf4M8j37dbnHbBe77fP0bw939PMw96pq961",
"columns": [
{
"id": 7323702856574852,
"version": 0,
"index": 0,
"title": "Project name",
"type": "TEXT_NUMBER"
},
{
"id": 1694203322361732,
"version": 0,
"index": 1,
"title": "Status",
"type": "TEXT_NUMBER"
},
"..."
],
"rows": [
{
"id": 2314743682492292,
"rowNumber": 1,
"cells": [
{
"columnId": 7323702856574852,
"value": "ERP Vendor Selection",
"displayValue": "ERP Vendor Selection"
},
{
"columnId": 1694203322361732,
"value": "Closed",
"displayValue": "Closed"
},
"..."
]
},
{
"id": 6818343309862788,
"rowNumber": 2,
"siblingId": 2314743682492292,
"cells": [
{
"columnId": 7323702856574852,
"value": "IS Dashboard",
"displayValue": "IS Dashboard"
},
{
"columnId": 1694203322361732,
"value": "On Hold",
"displayValue": "On Hold"
},
"..."
]
}
]
}
```
You want to index the name, status, and other data of each project item.
You use a JSONPath with a filter expression based on `columnId` to retrieve the value of the `value` field of each column.
Your JSON source configuration therefore contains:
```json
{
"Endpoints": [
{
"Path": "/sheets/952283761449296",
"Method": "GET",
"ItemPath": "rows",
"ItemType": "Project",
"Uri": "https://app.smartsheet.com/sheets/mC8mcf4M8j37dbnHbBe77fP0bw939PMw96pq961?rowId=%[id]",
"ClickableUri": "https://app.smartsheet.com/sheets/mC8mcf4M8j37dbnHbBe77fP0bw939PMw96pq961?rowId=%[id]",
"Title": "%[cells[?(@.columnId==7323702856574852)].value]",
"CreatedDate": "%[createdAt]",
"ModifiedDate": "%[modifiedAt]",
"Metadata": {
"rowId": "%[id]",
"rowNumber": "%[rowNumber]",
"projectName": "%[cells[?(@.columnId==7323702856574852)].value]",
"projectStatus": "%[cells[?(@.columnId==1694203322361732)].value]"
}
}
]
}
```
Details
Let's say you want to index the posts of a WordPress blog.
The following is a truncated response example from the WordPress API.
The main object represents a blog post.
```json
{
"id": 12345,
"date_gmt": "2024-03-11T15:43:43",
"slug": "coveo-test-page-wp",
"status": "publish",
"link": "https://example.com/coveo-test-page-wp/",
"title": {
"rendered": "Coveo Test Page WP Edited Content"
},
"content": {
"rendered": "This is a sample wordpress page to test integration with Coveo.
\nHeklsjfldsfj;ds
\ndsfjlkdskjafklds
\nlkfjdslkfjldk
\n",
"protected": false
},
"author": 4321,
"_embedded": {
"author": [
{
"id": 4321,
"name": "John Smith",
"url": "",
"description": "",
"link": "https://example.com/author/00u1o7cop0d6hhecl0h8/",
"slug": "00u1o7cop0d6hhecl0h8",
}
],
"wp:term": [
[
{
"id": 73,
"link": "https://example.com/category/docs-testing/",
"name": "DocsTesting",
"taxonomy": "category"
}
],
[
{
"id": 242,
"link": "https://example.com/tag/abc-xyz/",
"name": "abc-xyz",
"taxonomy": "post_tag"
},
{
"id": 243,
"link": "https://example.com/tag/coveo-test-wp/",
"name": "coveo-test-wp",
"taxonomy": "post_tag"
}
],
]
}
}
```
In your search interface, you want to be able to filter posts by tag.
So, when indexing a blog post, you also want to index a list of its tags as metadata.
To do so, you use a JSONPath expression, in which you add a filter expression based on the `taxonomy` property to exclude category names.
In your source JSON configuration, the [`Metadata`](https://docs.coveo.com/en/1525#metadata-object) object may therefore look as follows:
```json
"Metadata": {
"author": "%[_embedded.author[0].name]",
"date": "%[date_gmt]",
"wordpress_status": "%[status]",
"id": "%[id]",
"wordpress_tag_names": "%[_embedded.wp:term..[?(@.taxonomy=='post_tag')].name]"
}
```
As a result, Coveo will index the tags associated with each blog post in the `wordpress_tag_names` field.
In the [**Content Browser**](https://platform.cloud.coveo.com/admin/#/orgid/content/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/browser/)), this should look as follows:
However, since you want Coveo to consider these values separately, you must [enable the **Multi-value facet** option](https://docs.coveo.com/en/1833#facet-and-multi-value-facet) for the `wordpress_tag_names` field.
As a result, the tags will be displayed separately in the facet, like so:
Otherwise, you'll see the tags as a single string in the facet, for example: `abc-xyz;coveo-test-wp`.
Details
The following JSON configuration specifies a service with two endpoints.
The first endpoint has no `SkippableErrorCodes` property, so it inherits the value specified at the service level.
As a result, calls to this endpoint will skip the 404 error.
The second endpoint has a different `SkippableErrorCodes` value than its parent service, so the endpoint value overrides the service value.
Calls to this endpoint will skip both the 404 and 403 errors.
```json
{
"Services": [
{
"Url": "http://example.com/api/v1",
"Headers": {
"Authorization": "Bearer @ApiKey"
},
"SkippableErrorCodes": "404",
"Endpoints": [
{
"Path": "/users",
"Method": "GET",
"ItemType": "User",
"Uri": "%[coveo_url]/users/%[id]",
"ClickableUri": "%[coveo_url]/users/%[id]",
"Title": "%[name]",
"ModifiedDate": "%[updated]",
"Metadata": {
"id": "%[id]",
"username": "%[name]"
}
},
{
"Path": "/posts",
"Method": "GET",
"ItemType": "Post",
"Uri": "%[coveo_url]/posts/%[id]",
"ClickableUri": "%[coveo_url]/posts/%[id]",
"Title": "%[title]",
"ModifiedDate": "%[updated]",
"Metadata": {
"id": "%[id]",
"category": "%[category]"
},
"SkippableErrorCodes": "404;403"
}
]
}
]
}
```