Index Optimizely content

This article guides you on how to use the generic GraphQL API source, which is particularly suitable for Optimizely Graph endpoints, to optimize your indexing strategy. This approach ensures that you not only collect but also effectively utilize your content for enhanced search experiences.

Create a new GraphQL API source

When creating a GraphQL API source in the Coveo Administration Console, you must provide a JSON configuration that details what to crawl to retrieve the desired content and how to retrieve each type of item.

This configuration has a single property, Services, whose value must be an array of objects

GraphQL API Source JSON configuration | Coveo Platform

This reference article defines the parameters to include in your JSON configuration for the Optimizely Graph endpoint. The examples below show how to configure the source to index the items "Articles" and "Locations," where each article or location will be a document in the Coveo index available for search and recommendations.

Examples

Example 1

{
  "Services": [
	{
  	"Url": "https://prod.cg.optimizely.com/content",
  	"authentication": {
    	"username": "@username",
    	"password": "@password",
    	"forceBasicAuthentication": "true"
  	},
  	"Endpoints": [
    	{
      	"Path": "v2",
      	"Method": "GET",
      	"ItemPath": "data.Content.items",
      	"SkippableErrorCodes": "404",
      	"ItemType": "Articles",
      	"Uri": "%[Url]",
      	"ClickableUri": "%[Url]",
      	"Title": "%[Name]",
      	"QueryParameters": {
        	"query": "@sampleQuery"
      	}
    	}
  	]
	}
  ]
}

Example 2 - Using Content object with pagination

{
  "Services": [
    {
      "Url": "https://prod.cg.optimizely.com/content",
      "authentication": {
        "username": "@username",
        "password": "@password",
        "forceBasicAuthentication": "true"
      },
      "Endpoints": [
        {
          "paging": {
            "pageSize": 5,
            "offsetType": "item",
            "TotalCountKey": "data.Content.total"
          },
          "Path": "v2",
          "Method": "GET",
          "ItemPath": "data.Content.items",
          "SkippableErrorCodes": "404",
          "ItemType": "Articles",
          "Uri": "%[Url]",
          "ClickableUri": "%[Url]",
          "Title": "%[Name]",
          "ModifiedDate": "%[_modified]",
          "QueryParameters": {
            "query": "query ExampleQuery { Content(locale: en, skip:@offset, limit:@pageSize, where: { Url: { exist: true } }) { items{ Name ContentLink { Id WorkId GuidValue ProviderName Url } Url _modified } total } }"
          }
        }
      ]
    }
  ]
}

Example 3 - Using Locations object, field mapping, and pagination

{
  "Services": [
    {
      "Url": "https://prod.cg.optimizely.com/content",
      "authentication": {
        "username": "@username",
        "password": "@password",
        "forceBasicAuthentication": "true"
      },
      "Endpoints": [
        {
          "paging": {
            "pageSize": 5,
            "offsetType": "item",
            "TotalCountKey": "data.LocationPage.total"
          },
          "Path": "v2",
          "Method": "GET",
          "ItemPath": "data.LocationPage.items",
          "SkippableErrorCodes": "404",
          "ItemType": "LocationPage",
          "Uri": "%[Url]",
          "ClickableUri": "%[Url]",
          "Title": "%[Name]",
          "Body": "%[LocationServices], %[LocationCity], %[LocationPostalCode], %[LocationCountry]",
          "ModifiedDate": "%[_modified]",
          "Metadata": {
            "Address": "%[LocationCity]",
            "ThumbnailImage": "%[LocationImage.Url]",
            "Longitude": "%[LocationLongitude]",
            "Latitude": "%[LocationLatitude]"
          },
          "QueryParameters": {
            "query": "query ExampleQuery { LocationPage(locale: en, where: { Url: { exist: true } }) { items { Name LocationTitle LocationStreet1 LocationStreet2 LocationState LocationCity LocationPostalCode LocationImage { Url } LocationCountry LocationServices LocationLatitude LocationLongitude LocationPhone ContentLink { Id WorkId GuidValue ProviderName Url } Url _modified Changed } total } }"
          }
        }
      ]
    }
  ]
}

Example 4 - Using multiple endpoints, field mapping, indexing action, and pagination

{
  "Services": [
    {
      "Url": "https://prod.cg.optimizely.com/content",
      "authentication": {
        "username": "@username",
        "password": "@password",
        "forceBasicAuthentication": "true"
      },
      "Endpoints": [
        {
          "paging": {
            "pageSize": 20,
            "offsetType": "item",
            "TotalCountKey": "data.Content.total"
          },
          "IndexingAction": {
            "ActionOnItem": "Ignore",
            "Condition": "NOT(%[content_type] == \"ArticlePage\")"
          },
          "Path": "v2",
          "Method": "GET",
          "ItemPath": "data.Content.items",
          "SkippableErrorCodes": "404",
          "ItemType": "Articles",
          "Uri": "%[Url]",
          "ClickableUri": "%[Url]",
          "Title": "%[Name]",
          "Language": "%[Language.Name]",
          "ModifiedDate": "%[_modified]",
          "Body": "<html><body>%[MainBody]</body></html>",
          "Metadata": {
            "content_type": "%[ContentType[1]]",
            "status": "%[Status]",
            "full_text": "%[_fulltext]",
            "image_url": "%[PageImage.Url]",
            "language": "%[Language.Name]"
          },
          "QueryParameters": {
            "query": "query ExampleQuery { Content(skip:@offset, limit:@pageSize, where: { Url: { exist: true } }) { items{ Name Status ContentType _fulltext Url _modified Language { Name } ... on ArticlePage {MainBody PageImage { Url }} } total } }"
          }
        },
        {
          "paging": {
            "pageSize": 20,
            "offsetType": "item",
            "TotalCountKey": "data.Content.total"
          },
          "IndexingAction": {
            "ActionOnItem": "Ignore",
            "Condition": "NOT(%[content_type] == \"LocationPage\")"
          },
          "Path": "v2",
          "Method": "GET",
          "ItemPath": "data.Content.items",
          "SkippableErrorCodes": "404",
          "ItemType": "Articles",
          "Uri": "%[Url]",
          "ClickableUri": "%[Url]",
          "Title": "%[Name]",
          "Language": "%[Language.Name]",
          "ModifiedDate": "%[_modified]",
          "Body": "<html><body>%[LocationText]</body></html>",
          "Metadata": {
            "content_type": "%[ContentType[1]]",
            "status": "%[Status]",
            "full_text": "%[_fulltext]",
            "image_url": "%[LocationImage.Url]",
            "language": "%[Language.Name]"
          },
          "QueryParameters": {
            "query": "query ExampleQuery { Content(skip:@offset, limit:@pageSize, where: { Url: { exist: true } }) { items{ Name Status ContentType _fulltext Url _modified Language { Name } ... on LocationPage {LocationText LocationImage { Url }} } total } }"
          }
        }
      ]
    }
  ]
}

Configure "Services"

"Url" section

The Url value is the Optimizely Execute GraphQL query Url without the endpoint v2.

The GraphQL API source configuration Url will be as follows:

"Url": "https://prod.cg.optimizely.com/content"

"Authentication" section

If your Optimizely Graph endpoint requires authentication, this section ensures the right connection is established using the credentials provided. Enter the required authentication information in the Authentication section of the GraphQL source configuration panel. Use the Optimizely App key as username and Secret as password.

Edit GraphQL API Source

The recommendation is to use the @username and @password placeholders to retrieve the values specified in the GraphQL API source configuration panel.

"Url": "Optimizely Endpoint Name",
      "authentication": {
        "username": "<@username_PLACEHOLDER>",
        "password": "<@password_PLACEHOLDER>",
      }

Configure "Endpoints"

"Paging" section

Pagination is a technique used in web development and APIs to divide a large dataset into smaller, more manageable chunks called pages. When configuring the GraphQL API source, pagination allows users to navigate through the data by requesting and displaying one page at a time.

In the context of the GraphQL API source, pagination is being used to retrieve data from the API sequentially, where each request fetches one page of data. This approach helps in optimizing performance and reducing the amount of data transferred between the client and the server.

When building the query, include the total field to indicate the total number of pages. This is calculated by the server-side GraphQL implementation to provide the total count of items matching the query’s filters.

In the source configuration, include the number of items per page, the offsetType to specify to Coveo how your API paginates its content and the count field that specifies the total expected number of items through the data object.

Then, in your GraphQL query, make sure to include tokens @pageSize and @offset. Coveo will replace @pageSize with the value of the pageSize parameter of your paging configuration. Similarly, @offset will be replaced with the value extracted from the response, depending on the paging method selected in offsetType.

"paging": {
            "pageSize": 5,
            "offsetType": "item",
            "TotalCountKey": "data.Content.total"
}
"QueryParameters": {
            "query": "query ExampleQuery { Content(locale: en, skip:@offset, limit:@pageSize, where: { Url: { exist: true } }) { items{ Name Status ContentType _fulltext Url _modified Language { Name } ... on LocationPage {LocationText LocationImage { Url }} } total } }"
}

"IndexingAction" section

This section determines whether to ignore or retrieve an item when a condition is met. When the condition resolves to true, the Coveo crawler applies the specified action to the item. Possible actions are Retrieve and Ignore, that is, the crawler can either index the item or ignore it. An item ignored with IndexingAction isn’t indexed and therefore not visible in the Content Browser or a search interface.

In the provided example, we only want to index the items whose content type is equal to LocationPage.

"IndexingAction": {
            "ActionOnItem": "Ignore",
            "Condition": "NOT(%[content_type] == \"LocationPage\")"
},

"Path" section

The GraphQL API source configuration will be as follows:

"path": "v2"

"ItemType" section

Whether you’re using Optimizely Graph alone or with an existing Optimizely CMS, you need to define the type of item you want to pass from Optimizely into the Coveo index. In the example of this article, Content Type property is used. In CMS, you’ll have Content Templates that define the document you want to make discoverable through search. In the provided example, the configuration of the query fetches articles. Therefore, the GraphQL API source configuration will be as follows:

"ItemType": "Articles"

"Metadata" section

Each key represents the metadata name of the item, while its value is the value path (simple path or JSONPath) in the JSON response.

Typically, the value path consists of one or more dynamic values, since a static, hardcoded value would result in an identical piece of metadata for all items. However, you could choose to use a hardcoded value so that the corresponding Coveo field is filled even if the API doesn’t provide this information.

When you build your source, Coveo retrieves the desired metadata. You can then review a summary of this metadata in the Administration Console and use it to create mapping rules for your source.

Don’t forget to create the field in the Administration Console related to each metadata name and define if those fields are Facet, Multi-value facet, and Sortable.

This section allows us to associate specific properties of each item in the JSON response with a specific field in Coveo. In the provided example, content_type field will be associated with the second element of the ContentType property, status to Status, full_text to _fulltext, image_url to the property Url of the object PageImage, and language to the property Name of the Language object.

"Metadata": {
            "content_type": "%[ContentType[1]]",
            "status": "%[Status]",
            "full_text": "%[_fulltext]",
            "image_url": "%[PageImage.Url]",
            "language": "%[Language.Name]"
}

"Query Parameters" section

Configure the query to get content type objects Configure the desired query that defines the items and fields. Each item in the Graph query will be a document in the Coveo index that will be a part of search or recommendations, while each field will be either the body of the document or the field of the document populated with appropriate metadata.

Test your graph query with Optimizely’s Graph explorer:

https://cg.optimizely.com/app/graphiql?auth=<YOUR_SECRET_SINGLE_KEY>

See an example query is below:

Optimizely GraphQL Explorer

Note: When configuring the query, ensure the following fields are passed with each item. The documents indexed by Coveo, which are available as search results, must have a URL associated with each document. In the query above, the field URL and query parameter Url: { exist: true } are configured for Coveo index to be populated with documents that have an associated URL. It will allow you to easily configure the printableUri and clickableUrl. Include _modified or Changed fields in your query to ensure that the index picks up the changes in a timely manner and the freshest Optimizely Graph content resides in the index.

Note

If the Url returned in the Query field doesn’t return the desired properties, consider using an indexing pipeline extension (IPE) or opt to index content with a Sitemap source instead.