How to merge items with your catalog

This is for:

Developer

The Catalog source supports shallow merge operations, enabling updates to one or more fields in a catalog object. If the item doesn’t exist, it will be created, and if the field is missing, it will be added.

This operation should be used when it’s impossible to validate the existence of an item in the Catalog source before partially updating it.

The merge aspect means that the operation allows for updating one or multiple fields on an item that already exists in the Catalog source. If the item specified in the payload of the operation doesn’t exist in the source, the item is created.

The shallow aspect means that the operation doesn’t allow for partial updates on dictionary fields as it overwrites the entire value(s) of the field. See Shallow merge operations and dictionary fields for more information about the effect of using shallow merge operations with dictionary fields.

In other words, shallow merge operations:

  • Add metadata to existing items.

  • Update existing metadata on items.

  • Preserve existing metadata values that aren’t specified in the payload of the merge operation.

  • index new items if the product identifier mentioned in the payload doesn’t already exist in the source. Unlike partial item updates, this method allows you to index items for the first time. However, the full item update operation is still the recommended approach if you’re only indexing new items.

Note

Shallow merges on commerce items are only possible with Catalog sources.

In addition, you must have created and configured a catalog entity in the Coveo Administration Console. Attempting to perform a merge on a source without an associated catalog entity won’t index any items.

Leading practices

When designing your indexing implementation approach, consider if these options are the best fit for your use case before proceeding with a merge operation:

  • Full item updates are recommended over merge operations when indexing complete items, such as new products. Full item updates offer superior indexing performance, and will delete metadata no longer found in your source system.

  • Partial item updates can be used instead to perform more granular modifications, such as appending a value to an array or a dictionary field.

Warning

Shallow merge operations shouldn’t be used to append or remove values from a dictionary field. See Shallow merge operations and dictionary fields for more information about the effect of using shallow merge operations with dictionary fields.

Usage example

Consider an item of the Product catalog object, configured with the following metadata:

{
  "objecttype": "Product",
  "documentId": "product://010",
  "ec_name": "Sneaker 010",
  "productId": "010",
  "ec_category": "Sneakers",
  "gender": "Unisex",
  "departement": "Shoes"
}

You can perform a shallow merge operation to add a new field to this item, such as ec_brand, and update the gender field.

{
  "addOrMerge": [
    {
      "documentId": "product://010",
      "gender": "Women",
      "ec_brand": "ACME"
    }
  ]
}

In this example, the ec_brand field is added to the item, and the gender field is updated to Women. The item now appears in the source with the following metadata:

{
  "objecttype": "Product",
  "documentId": "product://010",
  "ec_name": "Sneaker 010",
  "productId": "010",
  "ec_category": "Sneakers",
  "gender": "Women",
  "departement": "Shoes",
  "ec_brand": "ACME"
}

Limits

The Stream API enforces certain limits on request size and frequency.

These limits differ depending on whether the organization to which data is pushed is a production or non-production organization.

The following table indicates the Stream API limits depending on your organization type:

organization type Maximum API requests per day Burst limit (requests per 5 minutes) Maximum file size Maximum item size[1] Maximum items per source[2]

Production

15,000

250

256 MB

3 MB

1,000,000

Non-production

10,000

150

256 MB

3 MB

1,000,000

1. This limit will be applied starting May 6, 2024.

2. This limit will be applied starting May 20, 2024.

Important

These limits could change at any time without prior notice. If you need to modify these limits, contact your Coveo representative.

Merging with existing items

To perform a shallow merge on your commerce items, you must interact with the Coveo Stream API. This section guides you through the different actions that must be taken to update your commerce items.

Required parameters

Refer to the Coveo Stream API for a comprehensive list of required parameters.

Step 1: Create a file container

To perform a merge document update, you must first create an Amazon S3 file container.

Use the Create a file container operation to create an Amazon S3 file container for a specific Coveo organization:

Request template

Request definition

POST https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/files?useVirtualHostedStyleUrl=<true|false> HTTP/1.1
​
Accept: application/json
Content-Type: application/json
Authorization: Bearer <MyAccessToken>

Request body

{}

In the request path:

In the query string:

  • Optionally, set useVirtualHostedStyleUrl to true if you want the service to return a virtual hosted-style URL, such as coveo-nprod-customerdata.s3.amazonaws.com/.... The default value is currently false, which means that the service returns path-style URLs, such as s3.amazonaws.com/coveo-nprod-customerdata/....

    Important

    The useVirtualHostedStyleUrl query string parameter will soon be deprecated as part of the path-style URL deprecation. From this point onwards, the service will only return virtual hosted-style URLs.

In the Authorization HTTP header:

The body of a successful response contains important information about the temporary, private, and encrypted Amazon S3 file container that you just created:

Successful response - 201 Created

{
    "uploadUri": "<UPLOAD-URI>", 1
    "fileId": "b5e8767e-8f0d-4a89-9095-1127915c89c7", 2
    "requiredHeaders": { 3
        "x-amz-server-side-encryption": "AES256",
        "Content-Type": "application/octet-stream"
    }
}
1 The uploadUri property contains a pre-signed URI to use in the PUT request of step 2.
Notes

  • The Amazon S3 file container applies AES-256 server-side encryption to your data.

  • The file container is automatically deleted as soon as its content has been successfully forwarded to the service.

  • The uploadUri automatically expires after 60 minutes.

Therefore, it’s safe to upload sensitive information into the Amazon S3 file container.
2 The fileId property contains the unique identifier of your file container.
3 The requiredHeaders property contains the required HTTP headers for sending in the PUT request of step 2.

Step 2: Upload the item content to merge into the file container

To upload the content to merge into the file container you created in step 1, perform the following PUT request:

Request template

PUT <MyUploadURI> HTTP/1.1

<HTTPHeaders>

Where you replace:

You can now upload your update data (JSON file) in the body of the request. For example, the following payload illustrates how you could replace metadata on existing commerce items:

Payload example

{
   "addOrMerge": [ 1
       {
           "objecttype": "Product",
           "documentId": "product://010",
           "ec_name": "Sneaker 010",
           "productId": "010",
           "ec_category": "Sneakers",
           "gender": "Unisex",
           "departement": "Shoes"
       },
       {
           "objecttype": "Variant",
           "documentId": "variant://010-blue",
           "width": "wide"
       },
       // ...More items to add or merge...
   ]
}

In the request body:

1 For each item you include in the addOrMerge array, you must specify a unique documentId value. Therefore, make sure that all of your items contain a unique documentId whose value is an URI. This value must be a valid URL with a proper URI prefix, such as product://, or any other scheme that fits your catalog data.

A successful response has no content, but indicates that the content update was successfully uploaded to the Amazon S3 file container, as in the following example:

200 OK

{}
Important

The payload must be chunked into parts of no more than 256 MB.

Step 3: Send the file container to update your catalog (full item update)

To push the Amazon S3 file container into your source, use the Merge documents of a catalog stream source operation as follows:

Request template

PUT https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/sources/<MySourceId>/stream/merge?fileId=<MyFileId> HTTP/1.1

Content-Type: application/json
Authorization: Bearer <MyAccessToken>

Payload

{}

In the request path:

  • Replace <MyOrganizationId> with the ID of the target Coveo organization (see Retrieve the organization ID).

  • Replace <MySourceId> with the ID of the source which contains the catalog data that you want to merge with.

In the query string:

In the Authorization HTTP header:

A successful response indicates that the operation was successfully forwarded to the service and that the batch of items is now enqueued to be processed by the Coveo indexing pipeline. The response body contains an orderingId that indicates the time your request was received, as well as the requestId which is the unique identifier for your request.

Example

202 Accepted

{
  "orderingId": 1716387965000,
  "requestId": "498ef728-1dc2-4b01-be5f-e8f8f1154a99"
}

Shallow merge operations and dictionary fields

Performing a shallow merge operation on an existing dictionary field replaces the entire value(s). This means that this operation shouldn’t be used to append or remove values from a dictionary field.

For example, consider the following dictionary field:

{
   ...
   "documentId": "product://010",
   "price_dict": {
       "": "28.00",
       "store1": "28.00",
       "store2": "30.00"
   },
   ...
}

You want to update the price_dict field to add a new store. You use a shallow merge operation such as the following:

{
  "addOrMerge": [
    {
      "documentId": "product://010",
      "price_dict": {
        "store3": "32.00"
      }
    }
  ]
}

The price_dict field is replaced by the new value, and the existing values are lost. The item now appears in the source with the following metadata:

{
   ...
   "documentId": "product://010",
   "price_dict": {
       "store3": "32.00"
   },
   ...
}

To append or remove values from a dictionary field, you should instead use a partial item update operation.

What’s next?

Once you’re done merging your catalog data, we strongly recommend that you inspect your content and properties to ensure that your content was indexed correctly.