How to update your catalog data

This is for:

Developer

In this article

Leading practices
Limits
Full item updates
Request template
Partial item updates
Request template
Example
What’s next?

Catalog source updates are required when you need to add, remove, or update items (products, variants, or availabilities) in your source after an initial stream.

Note

To perform the operations listed in this article, you must have a Catalog source or equivalent source with the Coveo Personalization-as-you-go feature enabled. In addition, you must have created and configured a catalog entity in the Coveo Administration Console (platform-ca | platform-eu | platform-au).

Leading practices

The Coveo Platform supports two update mechanisms for indexing updates to items in a Catalog source:

Full item updates let you:
- Update all fields in a given item.
- Add and delete items from your Catalog source.
Partial item updates let you update one or more fields in an existing Catalog source. Using this method reduces the number of calls made to the index and removes the need to rebuild the entire product list.

For example, you may want to use the partial item update mechanism to:
- Update a price field across a selection of documents.
- Add or remove a variant from a store array.
- Update the value of an existing product field.
- Update a dictionary field.

Limits

The Stream API enforces certain limits on request size and frequency.

These limits differ depending on whether the organization to which data is pushed is a production or non-production organization.

The following table indicates the Stream API limits depending on your organization type:

organization type	Maximum API requests per day	Burst limit (requests per 5 minutes)	Maximum file size	Maximum item size^[1]	Maximum items per source^[2]
Production	15,000	250	256 MB	3 MB	1,000,000
Non-production	10,000	150	256 MB	3 MB	1,000,000

1. This limit will be applied starting May 6, 2024.

2. This limit will be applied starting May 20, 2024.

These limits could change at any time without prior notice. If you need to modify these limits, contact your Coveo representative.

Full item updates

To perform a full item update, you must interact with the Coveo Stream API. This section guides you through the different actions that must be taken to update your catalog data.

Required parameters

Refer to the Stream API reference for a comprehensive list of required parameters.

Step 1: Create a file container (full item update)

To perform a full document update, you must first create an Amazon S3 file container.

Use the Create a file container operation to create an Amazon S3 file container for a specific Coveo organization:

Request template

Request definition

POST https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/files?useVirtualHostedStyleUrl=<true|false> HTTP/1.1

Accept: application/json
Content-Type: application/json
Authorization: Bearer <MyAccessToken>

Request body

{}

In the request path:

Replace <MyOrganizationId> with the ID of the target Coveo organization (see Retrieve the organization ID).

In the query string:

Optionally, set useVirtualHostedStyleUrl to true if you want the service to return a virtual hosted-style URL, such as coveo-nprod-customerdata.s3.amazonaws.com/.... The default value is currently false, which means that the service returns path-style URLs, such as s3.amazonaws.com/coveo-nprod-customerdata/....

The useVirtualHostedStyleUrl query string parameter will soon be deprecated as part of the path-style URL deprecation. From this point onwards, the service will only return virtual hosted-style URLs.

In the Authorization HTTP header:

Replace <MyAccessToken> with an access token that grants Organization - View privileges in the target Coveo organization. For more information, see:

The body of a successful response contains important information about the temporary, private, and encrypted Amazon S3 file container that you just created:

Successful response - 201 Created

{
    "uploadUri": "<UPLOAD-URI>", 
    "fileId": "b5e8767e-8f0d-4a89-9095-1127915c89c7", 
    "requiredHeaders": { 
        "x-amz-server-side-encryption": "AES256",
        "Content-Type": "application/octet-stream"
    }
}

The uploadUri property contains a pre-signed URI to use in the PUT request of step 2.

Notes

The Amazon S3 file container applies AES-256 server-side encryption to your data.
The file container is automatically deleted as soon as its content has been successfully forwarded to the service.
The uploadUri automatically expires after 60 minutes.

Therefore, it’s safe to upload sensitive information into the Amazon S3 file container.

The fileId property contains the unique identifier of your file container.

The requiredHeaders property contains the required HTTP headers for sending in the PUT request of step 2.

Step 2: Upload the full item content into the file container

To upload the content to update into the Amazon S3 file container you got from step 1, perform the following PUT request:

Request template

PUT <MyUploadURI> HTTP/1.1

<HTTPHeaders>

Where you replace:

<MyUploadURI> with the value of the uploadUri property you received the response when you created your file container in step 1.
<HTTPHeaders> with the key-value pairs of the requiredHeaders object property you received the response when you created your file container in step 1.

You can now upload your update data (JSON file) in the body of the request. For example, the following update data is structured in JSON and has items that must be updated and an item that must be deleted:

Payload example

{
  "addOrUpdate": [ 
    {
      "objecttype": "Product",
      "documentId": "product://010",
      "ec_name": "Sneaker 010",
      "ec_product_id": "010",
      "ec_category": "Sneakers",
      "gender": "Unisex",
      "departement": "Shoes"
    },
    {
      "objecttype": "Product",
      "documentId": "product://011",
      "ec_name": "Sneaker 011",
      "ec_product_id": "011",
      "ec_category": "Sneakers",
      "gender": "Unisex",
      "departement": "Shoes"
    },
    {
      "objecttype": "Variant",
      "documentId": "variant://010-blue",
      "ec_name": "Sneaker 010 Royal Blue",
      "ec_product_id": "010",
      "ec_variant_id": "010-blue",
      "width": "wide",
      "productSize": "9"
    },
  ],
  "delete": [ 
    {
      "documentId": "store://s000001"
    },
  ]
}

In the request body:

	For each item you include in the `addOrUpdate` array, specifying a unique `documentId` value for each item is mandatory. Therefore, you should make sure that all of your items contain a `documentId` for which the value is a URI that uniquely identifies the item. This value must be a valid URL with a proper URI prefix, such as `product://`, or any other scheme that fits your catalog data.
	For each item you include in the `delete` array, specifying a unique `documentId` value for each item is mandatory. This value must be a valid URL with a proper URI prefix, such as `product://`, or any other scheme that fits your catalog data.

A successful response has no content, but indicates that the content update was successfully uploaded to the Amazon S3 file container, as in the following example:

200 OK

{}

When the payload exceeds 256 MB, it must be chunked into 256 MB parts.

Step 3: Send the file container to update your source (full item update)

To push the Amazon S3 file container into your source, use the Update a catalog stream source operation as follows:

Request template

PUT https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/sources/<MySourceId>/stream/update?fileId=<MyFileId> HTTP/1.1

Content-Type: application/json
Authorization: Bearer <MyAccessToken>

Payload

{}

In the request path:

Replace <MyOrganizationId> with the ID of the target Coveo organization (see Retrieve the organization ID).
Replace <MySourceId> with the ID of the source which contains the catalog data that you want to update.

In the query string:

Replace <MyFileId> with the fileId you got from step 1.

In the Authorization HTTP header:

Replace <MyAccessToken> with an access token that grants the Organization - View and Sources - Edit privileges in the target organization (see Create an API key programmatically and Get the privileges of an access token).

A successful response indicates that the operation was successfully forwarded to the service and that the batch of items is now enqueued to be processed by the Coveo indexing pipeline. The response body contains an orderingId that indicates the time your request was received, as well as the requestId which is the unique identifier for your request.

Example

202 Accepted

{
  "orderingId": 1716387965000,
  "requestId": "498ef728-1dc2-4b01-be5f-e8f8f1154a99"
}

Partial item updates

To perform a partial item update, you must interact with the Coveo Stream API. This section guides you through the different actions that must be taken to update your items.

Structure and operations

The following example represents the basic structure of a JSON file to send for a partial item update:

{
  "partialUpdate": [
    {
      "documentId": "<DOCUMENT_ID>",
      "operator": "<PARTIAL_UPDATE_OPERATOR>",
      "field": "<FIELD_NAME>",
      "value": <VALUE>
    },
  ]
}

Where you replace:

<DOCUMENT_ID> with the required unique identifier of the document to apply a partial update operation to. Therefore, you should make sure that all of your items contain a documentId for which the value is a URI that uniquely identifies the item. This value must be a valid URL with a proper URI prefix, such as product://, or any other scheme that fits your catalog data.
<PARTIAL_UPDATE_OPERATOR> with one of the partial item update operators.
<FIELD_NAME> with the field name you wish to update.
<VALUE> depending on the chosen <PARTIAL_UPDATE_OPERATOR>, the <VALUE> can differ:
- To add items, see Add items
- To remove items, see Remove items
- To replace a field value, see Replace a field value
- To update a dictionary field, see Dictionary field update

The partialUpdate operation is an addition to the existing update endpoint as a top-level sibling to the addOrUpdate and delete operations.

Note

The partialUpdate operation can only be performed during an update, after an initial upload.

Partial item update operators

The following partial item update operators are treated sequentially.

Operator Description

Operator	Description
`arrayAppend`	Adds a list of elements to an array attribute. For example, adding unique identifiers for products or variants to a store, or making products or variants available.
`arrayRemove`	Removes a list of elements from an array attribute. For example, removing unique identifiers for products or variants from a store, or making products or variants unavailable.
`fieldValueReplace`	Replaces a field value regardless of the original value type.
`dictionaryPut`	Adds a key-value pair to the dictionary as long as the field already exists. `{ "key1": "value1" }`
`dictionaryRemove`	Removes a key-value pair as long as the key value already exists. Removing a single key: `"value": "key1"` Removing multiple keys: `"value": [ "key1", "key2", "Key3" ]`

arrayAppend

Adds a list of elements to an array attribute. For example, adding unique identifiers for products or variants to a store, or making products or variants available.

arrayRemove

Removes a list of elements from an array attribute. For example, removing unique identifiers for products or variants from a store, or making products or variants unavailable.

fieldValueReplace

Replaces a field value regardless of the original value type.

dictionaryPut

Adds a key-value pair to the dictionary as long as the field already exists.

{
  "key1": "value1"
}

dictionaryRemove

Removes a key-value pair as long as the key value already exists.

Removing a single key:

"value": "key1"

Removing multiple keys:

"value": [ "key1", "key2", "Key3" ]

WARNING

If an operation in the partial item update is found to be invalid, it will be ignored.

Protected fields

The following fields aren’t updated during a partial update operation:

documentid
permanentid
Product, variant, or availability object type fields
Product, variant, or availability ID fields

Object types and ID fields may vary depending on your catalog configuration.

Step 1: Create a file container (partial item update)

To perform a partial document update, you must first create an Amazon S3 file container.

Use the Create a file container operation to create an Amazon S3 file container for a specific Coveo organization:

Request template

Request definition

POST https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/files?useVirtualHostedStyleUrl=<true|false> HTTP/1.1

Accept: application/json
Content-Type: application/json
Authorization: Bearer <MyAccessToken>

Request body

{}

In the request path:

Replace <MyOrganizationId> with the ID of the target Coveo organization (see Retrieve the organization ID).

In the query string:

In the Authorization HTTP header:

Replace <MyAccessToken> with an access token that grants Organization - View privileges in the target Coveo organization. For more information, see:

The body of a successful response contains important information about the temporary, private, and encrypted Amazon S3 file container that you just created:

Successful response - 201 Created

{
    "uploadUri": "<UPLOAD-URI>", 
    "fileId": "b5e8767e-8f0d-4a89-9095-1127915c89c7", 
    "requiredHeaders": { 
        "x-amz-server-side-encryption": "AES256",
        "Content-Type": "application/octet-stream"
    }
}

The uploadUri property contains a pre-signed URI to use in the PUT request of step 2.

Notes

The Amazon S3 file container applies AES-256 server-side encryption to your data.
The file container is automatically deleted as soon as its content has been successfully forwarded to the service.
The uploadUri automatically expires after 60 minutes.

Therefore, it’s safe to upload sensitive information into the Amazon S3 file container.

The fileId property contains the unique identifier of your file container.

The requiredHeaders property contains the required HTTP headers for sending in the PUT request of step 2.

Step 2: Upload the partial item content into the file container

To upload the update file into the Amazon S3 file container you got from step 1, perform the following PUT request:

Request template

PUT <MyUploadURI> HTTP/1.1

<HTTPHeaders>

Where you replace:

<MyUploadURI> with the value of the uploadUri property you got in the response when you created your file container in step 1.
<HTTPHeaders> with the key-value pairs of the requiredHeaders object property you got in the response when you created your file container in step 1.

A successful response has no content, but indicates that the content update was successfully uploaded to the Amazon S3 file container, as in the following example:

200 OK

{}

Add items

The following example shows how to make products or variants available for a store:

{
  "partialUpdate": [
    {
      "documentId": "store://s000001",
      "operator": "arrayAppend",
      "field": "ec_available_items",
      "value": ["sku-224", "sku-225"]
    },
  ]
}

Remove items

The following example shows how to remove products or variants from a store:

{
  "partialUpdate": [
    {
      "documentId": "store://s000001",
      "operator": "arrayRemove",
      "field": "ec_available_items",
      "value": ["sku-221", "sku-220"]
    },
  ]
}

Replace a field value

Note

When performing an update to the value of a product field that doesn’t exist, the field will be created.

The following example shows how to modify a price and a description of a specific item, as well as change the name of another item during the same update:

{
  "partialUpdate": [
    {
      "documentId": "product://010",
      "operator": "fieldValueReplace",
      "field": "ec_price",
      "value": 23.50
    },
    {
      "documentId": "product://010",
      "operator": "fieldValueReplace",
      "field": "ec_description",
      "value": "Written representation of the change in description"
    },
    {
      "documentId": "product://040",
      "operator": "fieldValueReplace",
      "field": "ec_name",
      "value": "Product name X"
    },
  ]
}

Dictionary field update

You can use the partial item updates method to partially update dictionary fields on a Catalog source item. When performing this update, if the dictionary field to update doesn’t exist, the field will be added as a single-item dictionary. The following examples show how to update dictionary fields for items of the product catalog object. However, this method is applicable to any other catalog object type, such as variants or availability.

You should exercise caution when using distinct HTTP calls to send multiple updates targeting the same dictionary field in rapid succession. If an update to apply is received, and is older than the most recently applied update, the system will ignore the older message. This emphasizes the importance of ensuring proper order and sequencing when sending updates to prevent unintentional data inconsistencies.

Consider this initial product metadata. It contains dictionary field key-value pairs of types of ice-cream and their respective unique identifiers:

{
  "addOrUpdate": [
    {
      "documentId": "product://ice-cream_0001",
      "objecttype": "Product",
      "ec_product_id": "ice-cream_0001",
      "ec_price": 9.99
      "flavors": {
        "Vanilla": "SKU-0001",
        "Chocolate": "SKU-0002",
        "Pumpkin": "SKU-0003",
        "Cinnamon": "SKU-0004",
        "Candies": "SKU-0005"
      }
    }
  ]
}

The user decides to add a new flavor, "Moka": "SKU-123456", to the existing list. The JSON file of the partial dictionary update would look like the following:

{
  "partialUpdate": [
    {
      "documentId": "product://ice-cream_0001",
      "operator": "dictionaryPut",
      "field": "flavors",
      "value": {
        "Moka": "SKU-123456"
      }
    }
  ]
}

After which, the resulting updated dictionary would look like this:

{
  "documentId": "product://ice-cream_0001",
  "objecttype": "Product",
  "ec_product_id": "ice-cream_0001",
  "ec_price": 9.99
  "flavors": {
    "Vanilla": "SKU-0001",
    "Chocolate": "SKU-0002",
    "Pumpkin": "SKU-0003",
    "Cinnamon": "SKU-0004",
    "Candies": "SKU-0005",
    "Moka": "SKU-123456"
  }
}

To remove a key, such as the "Pumpkin" flavor, from the existing list, you need only identify the key as follows:

{
  "partialUpdate": [
    {
      "documentId": "product://ice-cream_0001",
      "operator": "dictionaryRemove",
      "field": "flavors",
      "value": "Pumpkin"
    }
  ]
}

To remove multiple keys, such as the "Pumpkin", "Cinnamon", and "Candies" flavors, from the dictionary, enter all of the keys in an array as follows:

{
  "partialUpdate": [
    {
      "documentId": "product://ice-cream_0001",
      "operator": "dictionaryRemove",
      "field": "flavors",
      "value": [ "Pumpkin", "Cinnamon", "Candies" ]
    }
  ]
}

Step 3: Send the file container to update your source (partial item update)

To push the Amazon S3 file container into your source, use the Update a catalog stream source operation as follows:

Request template

PUT https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/sources/<MySourceId>/stream/update?fileId=<MyFileId> HTTP/1.1

Content-Type: application/json
Authorization: Bearer <MyAccessToken>

Payload

{}

In the request path:

Replace <MyOrganizationId> with the ID of the target Coveo organization (see Retrieve the organization ID).
Replace <MySourceId> with the ID of the source which contains the catalog data that you want to update.

In the query string:

Replace <MyFileId> with the fileId you got from step 1.

In the Authorization HTTP header:

Replace <MyAccessToken> with an access token that grants the Organization - View and Sources - Edit privileges in the target organization (see Create an API key programmatically and Get the privileges of an access token).

Example

202 Accepted

{
  "orderingId": 1716387965000,
  "requestId": "498ef728-1dc2-4b01-be5f-e8f8f1154a99"
}

Example

The following is a complete example in which both full and partial update operators are used to update documents in a source.

{
  "addOrUpdate": [
    {
      "objecttype": "Product",
      "documentId": "product://010",
      "ec_name": "Sneaker 010",
      "ec_product_id": "010",
      "ec_category": "Sneakers",
      "gender": "Unisex",
      "departement": "Shoes"
    }
  ],
  "delete": [
    {
      "documentId": "store://s000001"
    }
  ],
  "partialUpdate": [
    {
      "documentId": "store://s000003",
      "operator": "arrayAppend",
      "field": "ec_available_items",
      "value": [
        "sku-224",
        "sku-225"
      ]
    },
    {
      "documentId": "store://s000005",
      "operator": "arrayRemove",
      "field": "ec_available_items",
      "value": [
        "sku-221",
        "sku-220"
      ]
    },
    {
      "documentId": "product://030",
      "operator": "fieldValueReplace",
      "field": "ec_price",
      "value": 23.5
    }
  ]
}

What’s next?

Once you’re done updating your catalog data, we strongly recommend that you inspect your content and properties to ensure that your content was indexed correctly.