How to merge items with your catalog

This is for:

Developer

The Catalog source supports shallow merge operations that can be leveraged to update multiple fields on commerce items (products, variants, or availabilities) in your source all at once. Unlike partial item updates, you can use this method to index items for the first time.

This operation is provided to enable implementers who can’t distinguish created and updated items in the system they wish to index.

Note

Shallow merges on catalog items are only possible with Catalog sources.

In addition, you must have created and configured a commerce catalog in the Coveo Administration Console. Attempts to perform merges on sources without an associated catalog won’t result in any items being indexed.

Leading practices

When designing your indexing implementation approach, consider if these options are the best fit for your use case before proceeding with a merge operation:

  • Full item updates are recommended over merge operations when indexing complete items, such as new products. Full item updates offer superior indexing performance, and will delete metadata no longer found in your source system.

  • Partial item updates can be used instead to perform more granular modifications, such as appending a value to an array.

Limits

The Push API enforces certain limits on request size and frequency.

These limits differ depending on whether the organization to which data is pushed is a production or non-production organization.

The following table indicates the Push API limits depending on your organization type:

organization type Maximum API requests per day Burst limit (requests per 5 minutes) Maximum file size Maximum item size[1] Maximum items per source[2]

Production

15,000

250

256 MB

3 MB

1,000,000

Non-production

10,000

150

256 MB

3 MB

1,000,000

1. This limit will be applied starting May 6, 2024.

2. This limit will be applied starting May 20, 2024.

Important

These limits could change at any time without prior notice. If you need to modify these limits, contact your Coveo representative.

Merging with catalog items

To perform a shallow merge on your catalog items, you must interact with the Coveo Push API. This section guides you through the different actions that must be taken to update your catalog items.

Required parameters

Refer to the Push API (V1) reference for a comprehensive list of required parameters.

Step 1: Create a file container

To perform a merge document update, you must first create an Amazon S3 file container.

Use the Create a file container operation to create an Amazon S3 file container for a specific Coveo organization:

Request template

POST https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/files?useVirtualHostedStyleUrl=<true|false> HTTP/1.1
​
Accept: application/json
Content-Type: application/json
Authorization: Bearer <MyAccessToken>

Payload

{}

In the request path:

In the query string:

  • Optionally, set useVirtualHostedStyleUrl to true if you want the service to return a virtual hosted-style URL, such as coveo-nprod-customerdata.s3.amazonaws.com/.... The default value is currently false, which means that the service returns path-style URLs, such as s3.amazonaws.com/coveo-nprod-customerdata/....

    Important

    The useVirtualHostedStyleUrl query string parameter will soon be deprecated as part of the path-style URL deprecation. From this point onwards, the service will only return virtual hosted-style URLs.

In the Authorization HTTP header:

The body of a successful response contains important information about the temporary, private, and encrypted Amazon S3 file container that you just created:

201 Created

{
    "uploadUri": "<UPLOAD-URI>", 1
    "fileId": "b5e8767e-8f0d-4a89-9095-1127915c89c7", 2
    "requiredHeaders": { 3
        "x-amz-server-side-encryption": "AES256",
        "Content-Type": "application/octet-stream"
    }
}
1 The uploadUri property contains a pre-signed URI to use in the PUT request of step 2.
Notes

  • The Amazon S3 file container applies AES-256 server-side encryption to your data.

  • The file container is automatically deleted as soon as its content has been successfully forwarded to the service.

  • The uploadUri automatically expires after 60 minutes.

Therefore, it’s safe to upload sensitive information into the Amazon S3 file container.
2 The fileId property contains the unique identifier of your file container.
3 The requiredHeaders property contains the required HTTP headers for sending in the PUT request of step 2.

Step 2: Upload the item content to merge into the file container

To upload the content to merge into the file container you created in step 1, perform the following PUT request:

Request template

PUT <MyUploadURI> HTTP/1.1

<HTTPHeaders>

Where you replace:

You can now upload your update data (JSON file) in the body of the request. For example, the following payload illustrates how you could replace metadata on existing catalog items:

Payload example

{
   "addOrMerge": [ 1
       {
           "objecttype": "Product",
           "documentId": "product://010",
           "ec_name": "Sneaker 010",
           "productId": "010",
           "ec_category": "Sneakers",
           "gender": "Unisex",
           "departement": "Shoes"
       },
       {
           "objecttype": "Variant",
           "documentId": "variant://010-blue",
           "width": "wide"
       },
       // ...More items to add or merge...
   ]
}

In the request body:

1 For each item you include in the addOrMerge array, you must specify a unique documentId value. Therefore, make sure that all of your items contain a unique documentId whose value is an URI. This value must be a valid URL with a proper URI prefix, such as product://, or any other scheme that fits your catalog’s configuration.

A successful response has no content, but indicates that the content update was successfully uploaded to the Amazon S3 file container, as in the following example:

200 OK

{}
Important

The payload must be chunked into parts of no more than 256 MB.

Step 3: Send the file container to update your catalog (full item update)

To push the Amazon S3 file container into your source, use the Merge documents of a catalog stream source operation as follows:

Request template

PUT https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/sources/<MySourceId>/stream/merge?fileId=<MyFileId> HTTP/1.1

Content-Type: application/json
Authorization: Bearer <MyAccessToken>

Payload

{}

In the request path:

  • Replace <MyOrganizationId> with the ID of the target Coveo organization (see Retrieve the organization ID).

  • Replace <MySourceId> with the ID of the source which contains the catalog data that you want to merge with.

In the query string:

In the Authorization HTTP header:

A successful response indicates that the operation was successfully forwarded to the service and that the batch of items is now enqueued to be processed by the Coveo indexing pipeline. The response body contains an orderingId that indicates the time your request was received, as well as the requestId which is the unique identifier for your request.

Example

202 Accepted

{
  "orderingId": 1716387965000,
  "requestId": "498ef728-1dc2-4b01-be5f-e8f8f1154a99"
}

What’s next?

Once you’re done merging your catalog data, we strongly recommend that you inspect your content and properties to ensure that your content was indexed correctly.