--- title: Add a GraphQL API source slug: n6gh2329 canonical_url: https://docs.coveo.com/en/n6gh2329/ collection: index-content source_format: adoc --- # Add a GraphQL API source Coveo has dedicated [connectors](https://docs.coveo.com/en/2734/) for many web and on-premises systems, therefore allowing you to quickly make application content searchable. See the [Connector directory](https://docs.coveo.com/en/1702/) for the full list. However, there may be applications of which you want to index the content, but for which there's no dedicated connector. In such a case, when you have the [required privileges](#required-privileges), you can use a [generic API connector](https://docs.coveo.com/en/1702#generic-connectors) to retrieve and make the desired content searchable with Coveo. The GraphQL API connector lets you crawl content from a remote repository exposing its data through a GraphQL API. When creating your [source](https://docs.coveo.com/en/246/), in the [Coveo Administration Console](https://docs.coveo.com/en/183/), you must provide a JSON configuration allowing Coveo to retrieve content [items](https://docs.coveo.com/en/210/). This configuration indicates which API calls to execute to fetch the desired items, how to parse the responses to extract relevant [metadata](https://docs.coveo.com/en/218/), and what type of resources these items represent. **Example** Your company uses [GitHub](https://github.com/) to manage your software code. You want to list the PRs made by each user for statistical purposes. So, you create a GraphQL API source that will query GitHub's API to index the desired content. The GraphQL source works very similarly to the [REST API source](https://docs.coveo.com/en/1896/). When working on your GraphQL API source, you may also want to refer to the following articles: * [Concepts](https://docs.coveo.com/en/n7jg0349/) * [Source configuration reference](https://docs.coveo.com/en/o1ae7549/) * [Permission configuration reference](https://docs.coveo.com/en/n7jg0461/) > **Leading practice** > > The number of [items](https://docs.coveo.com/en/210/) that a source processes per hour (crawling speed) depends on various factors, such as network bandwidth and source configuration. > See [About crawling speed](https://docs.coveo.com/en/2078/) for information on what can impact crawling speed, as well as possible solutions. ## Source key characteristics The following table presents the main characteristics of a GraphQL API source. [cols="4",options="header"] |=== 2+|Features |Supported |Additional information .3+.^|[Content update operations](https://docs.coveo.com/en/2039/) |[refresh](https://docs.coveo.com/en/2710/) ^|[check] a|For each [endpoint](https://docs.coveo.com/en/o1ae7549#endpoints-array-required) you define in your source configuration, you can provide a [refresh endpoint](https://docs.coveo.com/en/o1ae7549#refreshendpoints-array) to override the initial endpoint. When you do so, the connector can add, update, or delete specific items in the [index](https://docs.coveo.com/en/204/) instead of refreshing the entire repository. However, the following limitations currently apply: * Only refresh queries using the date of the last refresh operation can be made. Tokens such as `@Now` can't be used. * A sub-item can't be refreshed if its parent item isn't detected as modified. |[rescan](https://docs.coveo.com/en/2711/) ^|[check] |[Takes place every day by default](https://docs.coveo.com/en/1933/). |[rebuild](https://docs.coveo.com/en/2712/) ^|[check] | .3+.^|[Content security](https://docs.coveo.com/en/1779/) options |[Same users and groups as in your content system](https://docs.coveo.com/en/1779#same-users-and-groups-as-in-your-content-system) ^|[check] |If your <> includes the [`PermissionType` parameter](https://docs.coveo.com/en/o1ae7549#permissiontype-string), [provide a JSON configuration](https://docs.coveo.com/en/o1ae7549/) detailing how to extract the [relationships](https://docs.coveo.com/en/243/) of the indexed permissions. |[Specific users and groups](https://docs.coveo.com/en/1779#specific-users-and-groups) ^|[check] | |[Everyone](https://docs.coveo.com/en/1779#everyone) ^|[check] | .3+|[Metadata indexing for search](#index-metadata) |Automatic mapping of [metadata](https://docs.coveo.com/en/218/) to [fields](https://docs.coveo.com/en/200/) that have the same name 2+a|[Enabled by default](https://docs.coveo.com/en/1640#about-the-performfieldmappingusingallorigins-setting). |Automatically indexed [metadata](https://docs.coveo.com/en/218/) 2+a|Examples of [auto-populated default fields](https://docs.coveo.com/en/1833#field-origin) (no user-defined metadata required): * `date` * `language` (autodetected from [`Body`](https://docs.coveo.com/en/o1ae7549#body-string) object content) * `size`   Sample of auto-populated fields (source configuration parameter of matching name required): * `author` (include in [`Metadata`](https://docs.coveo.com/en/o1ae7549#metadata-object) object) * `filetype` (uses [`ItemType`](https://docs.coveo.com/en/o1ae7549#itemtype-string-required) value) * [`title`](https://docs.coveo.com/en/o1ae7549#title-string) * [`uri`](https://docs.coveo.com/en/o1ae7549#uri-string-required)   After a content update, [inspect your item field values](https://docs.coveo.com/en/2053#inspect-search-results) in the **Content Browser**. |Extracted but not indexed metadata 2+a|Parameters specified in the source configuration [`Metadata`](https://docs.coveo.com/en/o1ae7549#metadata-object) object that don't match an existing field name are extracted but not indexed. After a rebuild, review the [**View and map metadata**](https://docs.coveo.com/en/m9ti0339#view-and-map-metadata-subpage) subpage for the list of indexed metadata, and [index additional metadata](https://docs.coveo.com/en/m9ti0339#index-metadata). |=== ## Commerce requirements When using a GraphQL API [source](https://docs.coveo.com/en/246/) to [index](https://docs.coveo.com/en/204/) [commerce-specific content](https://docs.coveo.com/en/n8of5561/), such as products, [variants](https://docs.coveo.com/en/mc7f0326/), and [availabilities](https://docs.coveo.com/en/mc7e9096/), you have to undergo a catalog configuration process to benefit from all commerce-related capabilities. More specifically, you must: * [Enable Coveo Personalization-as-you-go (PAYG) capabilities](#enable-coveo-personalization-as-you-go) in your source. * Associate your source with a [catalog configuration](#catalog-configuration). ### Enable Coveo Personalization-as-you-go [Coveo Machine Learning](https://docs.coveo.com/en/188/) tools include [Coveo Personalization-as-you-go](https://docs.coveo.com/en/m5kd0347/) (PAYG) capabilities for commerce use cases. This suite of advanced features learns from a user's intent and reacts within a few clicks. PAYG models require the building of a [product vector space](https://docs.coveo.com/en/l9gg3565/) to represent the products contained in your source. For GraphQL API sources, Coveo PAYG needs to be enabled in order to produce the product vector space. Contact your Coveo representative to discuss your options. ### Catalog configuration Behind the scenes, the GraphQL API source uses the [Stream API](https://docs.coveo.com/en/12#tag/Stream) to push content to the Coveo [index](https://docs.coveo.com/en/204/). Therefore, GraphQL API sources must be associated with a [catalog entity](https://docs.coveo.com/en/3143/) to ensure a complete configuration. This allows the source to accurately build a [product vector](https://docs.coveo.com/en/l9gg3565/) space. For instructions on how to create a catalog entity, see [Commerce catalog entity](https://docs.coveo.com/en/3139/). ## Add a GraphQL API source Follow the instructions below to add a GraphQL API source using the desired [content retrieval method](https://docs.coveo.com/en/1612/). . On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, click **Add source**. . In the **Add a source of content** panel, click the **Cloud** (icon:cloud-icon[alt=cloud-icon,width=16]) or **Crawling Module** ([crawlingmodule]) tab, depending on your [content retrieval context](https://docs.coveo.com/en/1612/). With the latter, you must [install the Crawling Module](https://docs.coveo.com/en/3263/) to make your source operational. . Click the **GraphQL API** tile. . Configure your source. The [Build the source](#build-the-source) and [Index metadata](#index-metadata) steps are especially important when creating a source of this type. > **Leading practice** > > It's best to create or edit your source in your sandbox organization first. > Once you've confirmed that it indexes the desired content, you can copy your source configuration to your production organization, either [with a snapshot](https://docs.coveo.com/en/3239/) or manually. > > See [About non-production organizations](https://docs.coveo.com/en/2959/) for more information and best practices regarding sandbox organizations. ### "Configuration" tab In the **Add a GraphQL API Source** panel, the **Configuration** tab is selected by default. It contains your source's general and authentication information, as well as other parameters. #### General information ##### Source name Enter a name for your source. > **Leading practice** > > A source name can't be modified once it's saved, therefore be sure to use a short and descriptive name, using letters, numbers, hyphens (`-`), and underscores (`_`). Avoid spaces and other special characters. ##### Paired Crawling Module If your source is a [Crawling Module source](https://docs.coveo.com/en/1612/), and if you have [more than one Crawling Module linked to this organization](https://docs.coveo.com/en/3271#deploying-multiple-crawling-module-instances), select the one with which you want to pair your source. If you change the Crawling Module instance paired with your source, a successful [rebuild](https://docs.coveo.com/en/3390#refresh-rescan-or-rebuild-sources) is required for your change to apply. ##### Optical character recognition (OCR) If you want Coveo to extract text from image files or PDF files containing images, enable the appropriate option. The extracted text is processed as item data, meaning that it's fully searchable and will appear in the item [Quick view](https://docs.coveo.com/en/2760#search-result-quick-view). > **Note** > > When OCR is enabled, ensure the source's relevant [file type configurations](https://docs.coveo.com/en/l3qg9275/) index the item content. > Indexing the item's metadata only or ignoring the item will prevent OCR from being applied. See [Enable optical character recognition](https://docs.coveo.com/en/2937/) for details on this feature. ##### Project Use the **Project** selector to associate your source with one or more Coveo [projects](https://docs.coveo.com/en/n7ef0517/). #### "Authentication" section In the **Authentication** section, all parameters are optional. Fill the appropriate boxes depending on the authentication type used by the source you want to make searchable. * If your source uses a **HTTP, Basic, Kerberos, or NTLM authentication** protocol, enter the **Username** and **Password** of the account with which you want to crawl the source. Then, use the `@Username` and `@Password` placeholders in your source JSON configuration instead of exposing the credentials in clear text. The account of which you enter the credentials must have access to all the content that you want to make searchable. See [Source credentials leading practices](https://docs.coveo.com/en/1920/). * If your source uses the **OAuth 2.0 authentication** protocol, enter your content source **Client ID**, **Client secret** and **Refresh token** in the corresponding boxes. Then, use the `@ClientID`, `@ClientSecret`, and `@RefreshToken` placeholders in your source JSON configuration instead of exposing this information in clear text. * If your source uses an API key to authenticate, enter it in the **API key** box. Then, use the `@ApiKey` placeholder in your source JSON configuration instead of exposing the API key in clear text. * If your source doesn't require authentication, leave all boxes empty. #### "Content to include" section In the **JSON configuration** box, enter your source JSON configuration. Under [`PayloadJsonContent`](https://docs.coveo.com/en/o1ae7549#payloadjsoncontent-string) or [`QueryParameters`](https://docs.coveo.com/en/o1ae7549#queryparameters-object), enter a placeholder stating with `@` for your GraphQL queries, for example, `@MyFirstQuery`. Then, in the [**GraphQL queries** section below](#graphql-queries-section), enter the name you used as a placeholder and your actual query. For more information on the GraphQL API source JSON configuration, see: * [Concepts](https://docs.coveo.com/en/n7jg0349/) * [Reference](https://docs.coveo.com/en/o1ae7549/) #### "GraphQL queries" section Click **Add query** to start adding the GraphQL API queries that you want Coveo to execute to retrieve your content. You may want to use a GraphQL-to-JSON conversion tool such as [Data Fetcher's](https://datafetcher.com/graphql-json-body-converter) to help you write your queries. Under **Query name**, enter a name that you'll use as a placeholder for your query in the [source JSON configuration](#content-to-include-section). This name must start with `@`, for example, `@MyFirstQuery`. Then, under **GraphQL query**, enter your query. For example: ```graphql query { user(login:"jsmith") { pullRequests(first:@pageSize, after:@offset) { totalCount edges { node { createdAt title url } cursor } pageInfo { endCursor hasNextPage } } } } ``` Finally, back in your [source JSON configuration](#content-to-include-section), ensure to reference your query by name (starting with `@`) under [`PayloadJsonContent`](https://docs.coveo.com/en/o1ae7549#payloadjsoncontent-string) or [`QueryParameters`](https://docs.coveo.com/en/o1ae7549#queryparameters-object). For example, if you named your queries `@MyFirstQuery` and `@MySecondQuery`, your source JSON configuration could look as follows: ```json { "Services": [ { "Url": "https://api.github.com/", "authentication": { "username": "@username", "password": "@password", "forceBasicAuthentication": "true" }, "Endpoints": [ { "paging": { "pageSize": 10, "offsetType": "cursor", "nextPageKey": "data.user.pullRequests.pageInfo.endCursor" }, "headers": { "accept": "application/vnd.github.v3+json", "User-Agent": "PostmanRuntime/7.29.0" }, "Path": "graphql", "Method": "POST", "ItemPath": "data.user.pullRequests.edges", "ItemType": "PullRequests", "Uri": "%[node.url]", "ClickableUri": "%[node.url]", "Title": "%[node.title]", "ModifiedDate": "%[node.createdAt]", "PayloadJsonContent": "@MyFirstQuery" } ], "RefreshEndpoints": [ { "PayloadJsonContent": "@MySecondQuery" } ] } ] } ``` Alternatively, if you're using the placeholder under `QueryParameters`, it could contain the following: ```json { "QueryParameters": { "query": "@MyFirstQuery" } } ``` ### "Content security" tab Select who will be able to access the source items through a Coveo-powered [search interface](https://docs.coveo.com/en/2741/). For details on the content security options, see [Content security](https://docs.coveo.com/en/1779/). > **Note** > > If, while [writing your source JSON configuration](#content-to-include-section), you chose to index content access [permissions](https://docs.coveo.com/en/223/) and used the [`PermissionType` parameter](https://docs.coveo.com/en/o1ae7549#permissiontype-string), select the **Same users and groups as in your content system** option and provide a [JSON permission configuration](https://docs.coveo.com/en/o1ae7549/) detailing how to retrieve the [relationships](https://docs.coveo.com/en/243/) of each [security identity](https://docs.coveo.com/en/240/) and how to index this data. ### "Access" tab . On the **Access** tab, specify whether each group (and API key, if applicable) in your [Coveo organization](https://docs.coveo.com/en/185/) can view or edit the current source. For example, when creating a new source, you could decide that members of Group A can edit its configuration, while Group B can only view it. For more information, see [Custom access level](https://docs.coveo.com/en/3151#custom-access-level). On the **Access** tab, specify whether each group (and API key, if applicable) in your [Coveo organization](https://docs.coveo.com/en/185/) can view or edit the current source. For example, when creating a new source, you could decide that members of Group A can edit its configuration, while Group B can only view it. For more information, see [Custom access level](https://docs.coveo.com/en/3151#custom-access-level). ### Build the source . Finish adding or editing your source: ** When you're done editing the source and want to make your changes effective, click **Add and build source**/**Save and rebuild source**. ** When you want to save your source configuration changes without starting a build/rebuild, such as when you know you want to make other changes soon, click **Add source**/**Save**. On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, click **Launch build** or **Start required rebuild** when you're ready to make your changes effective and index your content. > **Leading practice** > > By default, a Jira Software source indexes the entire Jira Software instance content. > To index only certain projects, click **Save**, and then specify the desired address patterns in your [source JSON configuration](https://docs.coveo.com/en/1685/) before launching the initial build. > See [Add source filters](https://docs.coveo.com/en/2006#add-source-filters) for further information. . On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, follow the progress of your source addition or modification. . Once the source is built or rebuilt, [review its content in the Content Browser](https://docs.coveo.com/en/2053/). . Optionally, consider [editing or adding mappings](https://docs.coveo.com/en/1640/). > **Note** > > If you selected **Specific URLs** or **User profiles** in the [**Content**](https://docs.coveo.com/en/1739#content) section, some additional items will appear in the Content Browser. > To retrieve user profiles, Coveo must crawl your SharePoint Online instance, including your host site collection and the documents it contains. > Items encountered during this process are also retrieved and therefore appear in the Content Browser. ### Index metadata To use [metadata](https://docs.coveo.com/en/218/) values in [search interface](https://docs.coveo.com/en/2741/) [facets](https://docs.coveo.com/en/198/) or result templates, the metadata must be [mapped](https://docs.coveo.com/en/217/) to [fields](https://docs.coveo.com/en/200/). With recently created GraphQL API sources, [Coveo automatically maps metadata to fields with the same name](https://docs.coveo.com/en/1640#automatically-and-manually-created-mappings). > **Note** > > To enable this auto-mapping behavior on older GraphQL API sources, set the [`performFieldMappingUsingAllOrigins`](https://docs.coveo.com/en/1640#about-the-performfieldmappingusingallorigins-setting) parameter to `true` in the source JSON configuration. Coveo has some [default fields](https://docs.coveo.com/en/1833#field-origin) for commonly extracted metadata (for example, `author`, `date`). For any custom metadata defined in your source [JSON configuration](https://docs.coveo.com/en/o1ae7549/), you must create a field with the same name to store the metadata values. For example, if you've defined a `department` metadata, you must have a `department` field to store the metadata values. . Review the [Metadata](https://docs.coveo.com/en/o1ae7549#metadata-object) object in your source [JSON configuration](https://docs.coveo.com/en/o1ae7549/) for the list of metadata you're currently extracting from your content. . On the [**Fields**](https://platform.cloud.coveo.com/admin/#/orgid/content/fields/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/fields/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/fields/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/fields/)) page, for each metadata you want to use in facets or result templates, [add a field](https://docs.coveo.com/en/1833#add-a-field) with the same name, unless one already exists. > **Note** > > Fields are shared across sources in your [Coveo organization](https://docs.coveo.com/en/185/). > If a field with the same name as the metadata you want to index already exists and its [configuration](https://docs.coveo.com/en/1833#field-options) suits you, use it for the metadata you want to index. > Otherwise, you can [create a mapping](https://docs.coveo.com/en/1640#manage-mappings) to index the metadata in a new field with a different name. **Example** You decided to retrieve picture URIs from your content using the following metadata definition in your source [JSON configuration](https://docs.coveo.com/en/o1ae7549/): `"pictureuri": "%[picture.uri]"` Since there's no Coveo `pictureuri` default field, you need to create the `pictureuri` field, unless it already exists. . Return to the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page. . To reindex your source with your new mappings, click your source, and then click **More** > **Rebuild** in the Action bar. . Once the source is rebuilt, review your item field values. They should now include the values of the metadata you selected to index. .. On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, click your source, and then click **More** > **Open in Content Browser** in the Action bar. .. Select the card of the item for which you want to inspect properties, and then click **Properties** in the Action bar. .. In the panel that appears, select the **Fields** tab. ## Change the source request rate Your source makes requests to an API to index the desired content. By default, these requests are made immediately one after the other. If Coveo's requests reach the throttling limit of your API, you can increase the delay between each request. To do so, in the [source JSON configuration](https://docs.coveo.com/en/1685#access-the-edit-configuration-with-json-panel) (not the crawling configuration) `parameters` object, add the following code. ```json "RequestsIntervalInMs": { "value": "" <1> } ``` <1> Replace `` with the number of milliseconds Coveo should wait between each request. The default value is `0`, that is, there is no delay between requests. ## Ignoring "no first page" errors in subitems When indexing your content, Coveo may encounter an HTTP error. By default, Coveo stops the crawling process when it encounters such an error. However, you can [configure your source to ignore specific errors](https://docs.coveo.com/en/o1ae7549#skippableerrorcodes-string) and continue indexing. Similarly, when requesting [subitems](https://docs.coveo.com/en/o1ae7549#subitems-array) from your API, Coveo will stop the indexing process if your API returns a 404 error rather than the first page of results. A 404 error on the first page prevents Coveo from indexing any of your subitems, as the missing first page contains the information needed to request the second page, such as a cursor or the URL of the next page. However, you can configure your source to ignore this error and continue indexing. It will therefore finish indexing the other items of the same endpoint, including their subitems if the API returns valid result pages. Then, your source will move on to any other endpoint you've defined. To ignore "no first page" errors in subitems . On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)), select your source, and then, in the **More** menu, click **Edit configuration with JSON**. . In your source JSON configuration, in the `parameters` object, add the following: ``` "SkipNoFirstPageErrorsInSubItems": { "value": "true" }, ``` . Click **Save and rebuild source**. This parameter applies to all subitem requests made by your source. ## Required privileges You can assign privileges to allow access to specific tools in the [Coveo Administration Console](https://docs.coveo.com/en/183/). The following table indicates the privileges required to view or edit elements of the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page and associated panels. See [Manage privileges](https://docs.coveo.com/en/3151/) and [Privilege reference](https://docs.coveo.com/en/1707/) for more information. > **Note** > > The **Edit all** privilege isn't required to create sources. > When granting privileges for the [Sources](https://docs.coveo.com/en/1707#sources-domain) domain, you can grant a group or API key the **View all** or [**Custom**](https://docs.coveo.com/en/3151#custom-access-level) access level, instead of **Edit all**, and then select the **Can Create** checkbox to allow users to create sources. > See [Can Create ability dependence](https://docs.coveo.com/en/3151#can-create-ability-dependence) for more information. ## What's next? * [Schedule source updates](https://docs.coveo.com/en/1933/). [Schedule source updates](https://docs.coveo.com/en/1933/).