Partial catalog data updates

This is for:

Developer

To partially update your catalog data in your source (typically a Catalog source), you have to interact with the Coveo Stream API. The Coveo Platform supports two types of operations to perform partial updates on your catalog data:

  • Partial item updates let you update one or more fields in an existing source. Using this method reduces the number of calls made to the index and removes the need to rebuild the entire product list.

    For example, you may want to use the partial item update mechanism to:

    • Update a price field across a selection of documents.

    • Add or remove a variant from a store array.

    • Update the value of an existing product field.

    • Update a dictionary field.

  • Shallow merge operations: This operation allows you to update one or more fields in a catalog object. If the item doesn’t exist, it will be created, and if the field is missing, it will be added. In other words, shallow merge operations:

    • Add metadata to existing items.

    • Update existing metadata on items.

    • Preserve existing metadata values that aren’t specified in the payload of the merge operation.

    • index new items if the product identifier mentioned in the payload doesn’t already exist in the source. Unlike partial item updates, this method allows you to index items for the first time. However, the full item update operation is still the recommended approach if you’re only indexing new items.

Leading practices

Any partial update to your catalog data should be performed using the partial item update operation. This operation is the most efficient way to update your catalog data in your source.

Shallow merge operations should be used only when it’s impossible to validate the existence of an item in the Catalog source before partially updating it.

Prerequisites

To perform the operations listed in this article, you must have:

Partial item updates

Partial item updates let you update one or more fields in an existing source. Using this method reduces the number of calls made to the index and removes the need to rebuild the entire product list.

For example, you may want to use the partial item update mechanism to:

To perform a partial item update, you must interact with the Coveo Stream API. This section guides you through the different actions that must be taken to update your items.

Structure and operations

The following example represents the basic structure of a JSON file to send for a partial item update:

{
  "partialUpdate": [
    {
      "documentId": "<DOCUMENT_ID>", 1
      "operator": "<PARTIAL_UPDATE_OPERATOR>", 2
      "field": "<FIELD_NAME>", 3
      "value": <VALUE> 4
    },
  ]
}

Where you replace:

1 <DOCUMENT_ID> with the required unique identifier of the document to apply a partial update operation to. Therefore, you should make sure that all of your items contain a documentId for which the value is a URI that uniquely identifies the item. This value must be a valid URL with a proper URI prefix, such as product://, or any other scheme that fits your catalog data.
2 <PARTIAL_UPDATE_OPERATOR> with one of the following operators. Note that they’re treated sequentially:
Operator Description

arrayAppend

Adds a list of elements to an array attribute. For example, adding unique identifiers for products or variants to a store, or making products or variants available.

arrayRemove

Removes a list of elements from an array attribute. For example, removing unique identifiers for products or variants from a store, or making products or variants unavailable.

fieldValueReplace

Replaces a field value regardless of the original value type.

dictionaryPut

Adds a key-value pair to the dictionary as long as the field already exists.

{
  "key1": "value1"
}

dictionaryRemove

Removes a key-value pair as long as the key value already exists.

Removing a single key:

"value": "key1"

Removing multiple keys:

"value": [ "key1", "key2", "Key3" ]
Warning
WARNING

If an operation in the partial item update is found to be invalid, it will be ignored.
3 <FIELD_NAME> with the field name you wish to update.
4 <VALUE> depending on the chosen <PARTIAL_UPDATE_OPERATOR>, the <VALUE> can differ:

Protected fields

The following fields aren’t updated during a partial update operation:

  • documentid

  • permanentid

  • Product, variant, or availability object type fields

  • Product, variant, or availability ID fields

Object types and ID fields may vary depending on your catalog configuration.

Step 1: Create a file container (partial item update)

Important

Make sure that you meet the prerequisites before performing this operation.

To perform a partial document update, you must first create an Amazon S3 file container. Use the Create a file container operation to create an Amazon S3 file container for a specific Coveo organization:

Request template

POST https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/files?useVirtualHostedStyleUrl=<true|false> HTTP/1.1
​
Accept: application/json
Content-Type: application/json
Authorization: Bearer <MyAccessToken>

In the request path:

In the query string:

  • Optionally, set useVirtualHostedStyleUrl to true if you want the service to return a virtual hosted-style URL, such as coveo-nprod-customerdata.s3.amazonaws.com/.... The default value is currently false, which means that the service returns path-style URLs, such as s3.amazonaws.com/coveo-nprod-customerdata/....

    Important

    The useVirtualHostedStyleUrl query string parameter will soon be deprecated as part of the path-style URL deprecation. From this point onwards, the service will only return virtual hosted-style URLs.

In the Authorization HTTP header:

  • Replace <MyAccessToken> with an access token, such as an API key that has the required privileges to push content to the source.

Payload

{}

The body of a successful response contains important information about the temporary, private, and encrypted Amazon S3 file container that you just created:

{
    "uploadUri": "<UPLOAD-URI>", 1
    "fileId": "<FILE_ID>", 2
    "requiredHeaders": { 3
        "x-amz-server-side-encryption": "AES256",
        "Content-Type": "application/octet-stream"
    }
}
1 The uploadUri property contains a pre-signed URI to use in the PUT request of step 2.
Notes

  • The Amazon S3 file container applies AES-256 server-side encryption to your data.

  • The file container is automatically deleted as soon as its content has been successfully forwarded to the service.

  • The uploadUri automatically expires after 60 minutes.

Therefore, it’s safe to upload sensitive information into the Amazon S3 file container.
2 The fileId property contains the unique identifier of your file container. You must use this value to send the file container to the source in step 3.
3 The requiredHeaders property contains the required HTTP headers for sending in the PUT request of step 2.

Step 2: Upload the partial item content into the file container

To upload the update file into the Amazon S3 file container you got from step 1, perform the following PUT request:

Request template

PUT <MyUploadURI> HTTP/1.1
​
<HTTPHeaders>

Where you replace:

  • <MyUploadURI> with the value of the uploadUri property you got in the response when you created your file container in step 1.

  • <HTTPHeaders> with the key-value pairs of the requiredHeaders object property you got in the response when you created your file container in step 1.

You can now upload your update data (JSON file) in the body of the request. Expand the following sections to see examples of how to structure your JSON file for each of the available operations:

Add items

Add items

The following example shows how to make products or variants available for a store:

{
  "partialUpdate": [
    {
      "documentId": "store://s000001",
      "operator": "arrayAppend",
      "field": "ec_available_items",
      "value": ["sku-224", "sku-225"]
    },
  ]
}
Remove items

Remove items

The following example shows how to remove products or variants from a store:

{
  "partialUpdate": [
    {
      "documentId": "store://s000001",
      "operator": "arrayRemove",
      "field": "ec_available_items",
      "value": ["sku-221", "sku-220"]
    },
  ]
}
Replace a field value

Replace a field value

Note

When performing an update to the value of a product field that doesn’t exist, the field will be created.

The following example shows how to modify a price and a description of a specific item, as well as change the name of another item during the same update:

{
  "partialUpdate": [
    {
      "documentId": "product://010",
      "operator": "fieldValueReplace",
      "field": "ec_price",
      "value": 23.50
    },
    {
      "documentId": "product://010",
      "operator": "fieldValueReplace",
      "field": "ec_description",
      "value": "Written representation of the change in description"
    },
    {
      "documentId": "product://040",
      "operator": "fieldValueReplace",
      "field": "ec_name",
      "value": "Product name X"
    },
  ]
}
Dictionary field update

Dictionary field update

You can use the partial item updates method to partially update dictionary fields on a Catalog source item. When performing this update, if the dictionary field to update doesn’t exist, the field will be added as a single-item dictionary. The following examples show how to update dictionary fields for items of the product catalog object. However, this method is applicable to any other catalog object type, such as variants or availability.

Important

You should exercise caution when using distinct HTTP calls to send multiple updates targeting the same dictionary field in rapid succession. If an update to apply is received, and is older than the most recently applied update, the system will ignore the older message. This emphasizes the importance of ensuring proper order and sequencing when sending updates to prevent unintentional data inconsistencies.

Consider this initial product metadata. It contains dictionary field key-value pairs of types of ice-cream and their respective unique identifiers:

{
  "addOrUpdate": [
    {
      "documentId": "product://ice-cream_0001",
      "objecttype": "Product",
      "ec_product_id": "ice-cream_0001",
      "ec_price": 9.99
      "flavors": {
        "Vanilla": "SKU-0001",
        "Chocolate": "SKU-0002",
        "Pumpkin": "SKU-0003",
        "Cinnamon": "SKU-0004",
        "Candies": "SKU-0005"
      }
    }
  ]
}

The user decides to add a new flavor, "Moka": "SKU-123456", to the existing list. The JSON file of the partial dictionary update would look like the following:

{
  "partialUpdate": [
    {
      "documentId": "product://ice-cream_0001",
      "operator": "dictionaryPut",
      "field": "flavors",
      "value": {
        "Moka": "SKU-123456"
      }
    }
  ]
}

After which, the resulting updated dictionary would look like this:

{
  "documentId": "product://ice-cream_0001",
  "objecttype": "Product",
  "ec_product_id": "ice-cream_0001",
  "ec_price": 9.99
  "flavors": {
    "Vanilla": "SKU-0001",
    "Chocolate": "SKU-0002",
    "Pumpkin": "SKU-0003",
    "Cinnamon": "SKU-0004",
    "Candies": "SKU-0005",
    "Moka": "SKU-123456"
  }
}

To remove a key, such as the "Pumpkin" flavor, from the existing list, you need only identify the key as follows:

{
  "partialUpdate": [
    {
      "documentId": "product://ice-cream_0001",
      "operator": "dictionaryRemove",
      "field": "flavors",
      "value": "Pumpkin"
    }
  ]
}

To remove multiple keys, such as the "Pumpkin", "Cinnamon", and "Candies" flavors, from the dictionary, enter all of the keys in an array as follows:

{
  "partialUpdate": [
    {
      "documentId": "product://ice-cream_0001",
      "operator": "dictionaryRemove",
      "field": "flavors",
      "value": [ "Pumpkin", "Cinnamon", "Candies" ]
    }
  ]
}

A successful response has no content, but indicates that the content update was successfully uploaded to the Amazon S3 file container, as follows:

200 OK

{}
Important

When the payload exceeds 256 MB, it must be chunked into 256 MB parts. See Uploading large catalog data files for instructions.

Step 3: Send the file container to update your source (partial item update)

To push the Amazon S3 file container into your source, use the Update a catalog stream source operation as follows:

Request template

PUT https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/sources/<MySourceId>/stream/update?fileId=<MyFileId> HTTP/1.1
​
Content-Type: application/json
Authorization: Bearer <MyAccessToken>

Payload

{}

Where you replace:

  • <MyOrganizationId> with the ID of the target Coveo organization (see Retrieve the organization ID).

  • <MySourceId> with the ID of the source which contains the catalog data that you want to update.

  • <MyFileId> with the fileId you got from step 1.

  • <MyAccessToken> with an access token, such as an API key that has the required privileges to push content to the source.

A successful response (202) indicates that the operation was successfully forwarded to the service and that the batch of items is now enqueued to be processed by the Coveo indexing pipeline. For example:

202 Accepted

{
  "orderingId": 1716387965000, 1
  "requestId": "498ef728-1dc2-4b01-be5f-e8f8f1154a99" 2
}

Where:

1 orderingId indicates the time your request was received.
2 requestId is the unique identifier for your request.
Tip

The contents of a file container can be pushed to multiple sources in the same Coveo organization. Just update the target sourceId and Authorization HTTP header access token in your other Stream API update or merge requests.

The file container remains available for 4 days.

Example

The following is a complete example of a partial update used to update documents in a source.

{
  "partialUpdate": [
    {
      "documentId": "store://s000003",
      "operator": "arrayAppend",
      "field": "ec_available_items",
      "value": [
        "sku-224",
        "sku-225"
      ]
    },
    {
      "documentId": "store://s000005",
      "operator": "arrayRemove",
      "field": "ec_available_items",
      "value": [
        "sku-221",
        "sku-220"
      ]
    },
    {
      "documentId": "product://030",
      "operator": "fieldValueReplace",
      "field": "ec_price",
      "value": 23.5
    }
  ]
}

Shallow merge operations

The Catalog source supports shallow merge operations, enabling updates to one or more fields in a catalog object. If the item doesn’t exist, it will be created, and if the field is missing, it will be added.

This operation should be used when it’s impossible to validate the existence of an item in the Catalog source before partially updating it.

The merge aspect means that the operation allows for updating one or multiple fields on an item that already exists in the Catalog source. If the item specified in the payload of the operation doesn’t exist in the source, the item is created.

The shallow aspect means that the operation doesn’t allow for partial updates on dictionary fields as it overwrites the entire value(s) of the field. See Shallow merge operations and dictionary fields for more information about the effect of using shallow merge operations with dictionary fields.

In other words, shallow merge operations:

  • Add metadata to existing items.

  • Update existing metadata on items.

  • Preserve existing metadata values that aren’t specified in the payload of the merge operation.

  • index new items if the product identifier mentioned in the payload doesn’t already exist in the source. Unlike partial item updates, this method allows you to index items for the first time. However, the full item update operation is still the recommended approach if you’re only indexing new items.

Warning

Shallow merge operations shouldn’t be used to append or remove values from a dictionary field. See Shallow merge operations and dictionary fields for more information about the effect of using shallow merge operations with dictionary fields.

Usage example

Consider an item of the Product catalog object, configured with the following metadata:

{
  "objecttype": "Product",
  "documentId": "product://010",
  "ec_name": "Sneaker 010",
  "productId": "010",
  "ec_category": "Sneakers",
  "gender": "Unisex",
  "departement": "Shoes"
}

You can perform a shallow merge operation to add a new field to this item, such as ec_brand, and update the gender field.

{
  "addOrMerge": [
    {
      "documentId": "product://010",
      "gender": "Women",
      "ec_brand": "ACME"
    }
  ]
}

In this example, the ec_brand field is added to the item, and the gender field is updated to Women. The item now appears in the source with the following metadata:

{
  "objecttype": "Product",
  "documentId": "product://010",
  "ec_name": "Sneaker 010",
  "productId": "010",
  "ec_category": "Sneakers",
  "gender": "Women",
  "departement": "Shoes",
  "ec_brand": "ACME"
}

Merging with existing items

To perform a shallow merge on your commerce items, you must interact with the Coveo Stream API. This section guides you through the different actions that must be taken to update your commerce items.

Refer to the Coveo Stream API for a comprehensive list of required parameters.

Step 1: Create a file container

Important

Make sure that you meet the prerequisites before performing this operation.

To perform a merge document update, you must first create an Amazon S3 file container. Use the Create a file container operation to create an Amazon S3 file container for a specific Coveo organization:

Request template

POST https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/files?useVirtualHostedStyleUrl=<true|false> HTTP/1.1
​
Accept: application/json
Content-Type: application/json
Authorization: Bearer <MyAccessToken>

In the request path:

In the query string:

  • Optionally, set useVirtualHostedStyleUrl to true if you want the service to return a virtual hosted-style URL, such as coveo-nprod-customerdata.s3.amazonaws.com/.... The default value is currently false, which means that the service returns path-style URLs, such as s3.amazonaws.com/coveo-nprod-customerdata/....

    Important

    The useVirtualHostedStyleUrl query string parameter will soon be deprecated as part of the path-style URL deprecation. From this point onwards, the service will only return virtual hosted-style URLs.

In the Authorization HTTP header:

  • Replace <MyAccessToken> with an access token, such as an API key that has the required privileges to push content to the source.

Payload

{}

The body of a successful response contains important information about the temporary, private, and encrypted Amazon S3 file container that you just created:

{
    "uploadUri": "<UPLOAD-URI>", 1
    "fileId": "<FILE_ID>", 2
    "requiredHeaders": { 3
        "x-amz-server-side-encryption": "AES256",
        "Content-Type": "application/octet-stream"
    }
}
1 The uploadUri property contains a pre-signed URI to use in the PUT request of step 2.
Notes

  • The Amazon S3 file container applies AES-256 server-side encryption to your data.

  • The file container is automatically deleted as soon as its content has been successfully forwarded to the service.

  • The uploadUri automatically expires after 60 minutes.

Therefore, it’s safe to upload sensitive information into the Amazon S3 file container.
2 The fileId property contains the unique identifier of your file container. You must use this value to send the file container to the source in step 3.
3 The requiredHeaders property contains the required HTTP headers for sending in the PUT request of step 2.

Step 2: Upload the item content to merge into the file container

To upload the content to merge into the file container you created in step 1, perform the following PUT request:

Request template

PUT <MyUploadURI> HTTP/1.1

<HTTPHeaders>

Where you replace:

You can now upload your update data (JSON file) in the body of the request. For example, the following payload illustrates how you could replace metadata on existing commerce items:

Payload example

{
   "addOrMerge": [ 1
       {
           "objecttype": "Product",
           "documentId": "product://010",
           "ec_name": "Sneaker 010",
           "productId": "010",
           "ec_category": "Sneakers",
           "gender": "Unisex",
           "departement": "Shoes"
       },
       {
           "objecttype": "Variant",
           "documentId": "variant://010-blue",
           "width": "wide"
       },
       // ...More items to add or merge...
   ]
}

In the request body:

1 For each item you include in the addOrMerge array, you must specify a unique documentId value. Therefore, make sure that all of your items contain a unique documentId whose value is an URI. This value must be a valid URL with a proper URI prefix, such as product://, or any other scheme that fits your catalog data.

A successful response has no content, but indicates that the content update was successfully uploaded to the Amazon S3 file container, as in the following example:

200 OK

{}
Important

When the payload exceeds 256 MB, it must be chunked into 256 MB parts. See Uploading large catalog data files for instructions.

Step 3: Send the file container to update your catalog (full item update)

To push the Amazon S3 file container into your source, use the Merge documents of a catalog stream source operation as follows:

Request template

PUT https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/sources/<MySourceId>/stream/merge?fileId=<MyFileId> HTTP/1.1

Content-Type: application/json
Authorization: Bearer <MyAccessToken>

Payload

{}

In the request path:

  • Replace <MyOrganizationId> with the ID of the target Coveo organization (see Retrieve the organization ID).

  • Replace <MySourceId> with the ID of the source which contains the catalog data that you want to merge with.

In the query string:

In the Authorization HTTP header:

  • Replace <MyAccessToken> with an access token, such as an API key that has the required privileges to push content to the source.

A successful response indicates that the operation was successfully forwarded to the service and that the batch of items is now enqueued to be processed by the Coveo indexing pipeline. The response body contains an orderingId that indicates the time your request was received, as well as the requestId which is the unique identifier for your request.

Example

202 Accepted

{
  "orderingId": 1716387965000,
  "requestId": "498ef728-1dc2-4b01-be5f-e8f8f1154a99"
}
Tip

The contents of a file container can be pushed to multiple sources in the same Coveo organization. Just update the target sourceId and Authorization HTTP header access token in your other Stream API merge or update requests.

The file container remains available for 4 days.

Shallow merge operations and dictionary fields

Performing a shallow merge operation on an existing dictionary field replaces the entire value(s). This means that this operation shouldn’t be used to append or remove values from a dictionary field.

For example, consider the following dictionary field:

{
   ...
   "documentId": "product://010",
   "price_dict": {
       "": "28.00",
       "store1": "28.00",
       "store2": "30.00"
   },
   ...
}

You want to update the price_dict field to add a new store. You use a shallow merge operation such as the following:

{
  "addOrMerge": [
    {
      "documentId": "product://010",
      "price_dict": {
        "store3": "32.00"
      }
    }
  ]
}

The price_dict field is replaced by the new value, and the existing values are lost. The item now appears in the source with the following metadata:

{
   ...
   "documentId": "product://010",
   "price_dict": {
       "store3": "32.00"
   },
   ...
}

To append or remove values from a dictionary field, you should instead use a partial item update operation.

Stream API limits

The Stream API enforces certain limits on request size and frequency.

These limits differ depending on whether the organization to which data is pushed is a production or non-production organization.

The following table indicates the Stream API limits depending on your organization type:

organization type Maximum API requests per day Burst limit (requests per 5 minutes) Maximum upload requests per day Maximum file size Maximum item size[1] Maximum items per source[2]

Production

15,000

250

96

256 MB

3 MB

1,000,000

Non-production

10,000

150

96

256 MB

3 MB

1,000,000

1. This limit will be applied starting May 6, 2024.

2. This limit will be applied starting May 20, 2024.

Important

These limits could change at any time without prior notice. To modify these limits, contact your Coveo representative.

Stream API error codes

If a request to the Stream API fails because one of the limits has been exceeded, the API will trigger one of the following response status codes:

Status code Triggered when

413

The total Stream API request size exceeds 256 MB when pushing a large file container. See Uploading large catalog files.

429

The amount of total Stream API (upload and update) requests exceeds 15,000 per day (10,000 for non-production organizations). The quota is reset at midnight UTC.

The amount of total Stream API upload requests exceeds 96 per day (4 per hour). The quota is reset at midnight UTC.

The amount of total Stream API requests exceeds 250 (150 for non-production organizations) within a 5 minute period. The retry-after header indicates how long the user agent should wait before making another request.

Coveo declined your request due to a reduced indexing capacity.

Uploading large catalog data files

The Stream API limits the size of your catalog data JSON file to 256 MB. If your catalog data file exceeds the limit, you must upload multiple JSON files.

To upload multiple JSON files:

When you initially open the stream, you receive an uploadUri. This URI is used to upload your first set of metadata (JSON file).

  1. After uploading the first file, make a POST request to the following endpoint to get a new uploadUri:

    POST https://api.cloud.coveo.com/push/v1/organizations/{organizationId}/sources/{sourceId}/stream/{streamId}/chunk HTTP/1.1

Content-Type: application/json
Accept: application/json
Authorization: Bearer <MY_ACCESS_TOKEN>

    This request returns a new uploadUri that you can use for the next step.

  2. Make a PUT request using the uploadUri your received in the previous step. The body of the request must contain the catalog data chunk (maximum 256 MB) that you want to upload.

    PUT {uploadUri} HTTP/1.1

x-amz-server-side-encryption: AES256
Content-Type: application/octet-stream

    If your request to upload the catalog data is successful, you’ll receive a 200 HTTP response code.

  3. If you have more catalog data files to upload, repeat this process until all of your catalog data has been uploaded. For each file, first obtain a new uploadUri, then upload the file.

Required privileges

The following table indicates the privileges required for your organization's groups to view or edit elements of the Catalogs (platform-ca | platform-eu | platform-au) page and its associated panels (see Manage privileges and Privilege reference). The Commerce domain is only available to organizations in which Coveo for Commerce features are enabled.

Action Service - Domain Required access level

View catalogs

Commerce - Catalogs
Content - Sources
Content - Fields
Organization - Organization

View

Edit catalogs

Content - Fields
Content - Sources
Organization - Organization

View

Commerce - Catalogs

Edit

Search - Execute Query

Allowed