How to merge items with your catalog

This is for:

Developer

In this article

Leading practices
Limits
Merging with catalog items
What’s next?

The Catalog source supports shallow merge operations that can be leveraged to update multiple fields on commerce items (products, variants, or availabilities) in your source all at once. Unlike partial item updates, you can use this method to index items for the first time.

This operation is provided to enable implementers who can’t distinguish created and updated items in the system they wish to index.

Note

Shallow merges on catalog items are only possible with Catalog sources.

In addition, you must have created and configured a commerce catalog in the Coveo Administration Console. Attempts to perform merges on sources without an associated catalog won’t result in any items being indexed.

Leading practices

When designing your indexing implementation approach, consider if these options are the best fit for your use case before proceeding with a merge operation:

Full item updates are recommended over merge operations when indexing complete items, such as new products. Full item updates offer superior indexing performance, and will delete metadata no longer found in your source system.
Partial item updates can be used instead to perform more granular modifications, such as appending a value to an array.

Limits

The Push API enforces certain limits on request size and frequency.

These limits differ depending on whether the organization to which data is pushed is a production or non-production organization.

The following table indicates the Push API limits depending on your organization type:

organization type	Maximum API requests per day	Burst limit (requests per 5 minutes)	Maximum file size	Maximum item size^[1]	Maximum items per source^[2]
Production	15,000	250	256 MB	3 MB	1,000,000
Non-production	10,000	150	256 MB	3 MB	1,000,000

1. This limit will be applied starting May 6, 2024.

2. This limit will be applied starting May 20, 2024.

These limits could change at any time without prior notice. If you need to modify these limits, contact your Coveo representative.

Merging with catalog items

To perform a shallow merge on your catalog items, you must interact with the Coveo Push API. This section guides you through the different actions that must be taken to update your catalog items.

Required parameters

Refer to the Push API (V1) reference for a comprehensive list of required parameters.

Step 1: Create a file container

To perform a merge document update, you must first create an Amazon S3 file container.

Use the Create a file container operation to create an Amazon S3 file container for a specific Coveo organization:

Request template

POST https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/files?useVirtualHostedStyleUrl=<true|false> HTTP/1.1

Accept: application/json
Content-Type: application/json
Authorization: Bearer <MyAccessToken>

Payload

{}

In the request path:

Replace <MyOrganizationId> with the ID of the target Coveo organization (see Retrieve the organization ID).

In the query string:

Optionally, set useVirtualHostedStyleUrl to true if you want the service to return a virtual hosted-style URL, such as coveo-nprod-customerdata.s3.amazonaws.com/.... The default value is currently false, which means that the service returns path-style URLs, such as s3.amazonaws.com/coveo-nprod-customerdata/....

The useVirtualHostedStyleUrl query string parameter will soon be deprecated as part of the path-style URL deprecation. From this point onwards, the service will only return virtual hosted-style URLs.

In the Authorization HTTP header:

Replace <MyAccessToken> with an access token that grants Organization - View privileges in the target Coveo organization. For more information, see:

The body of a successful response contains important information about the temporary, private, and encrypted Amazon S3 file container that you just created:

201 Created

{
    "uploadUri": "<UPLOAD-URI>", 
    "fileId": "b5e8767e-8f0d-4a89-9095-1127915c89c7", 
    "requiredHeaders": { 
        "x-amz-server-side-encryption": "AES256",
        "Content-Type": "application/octet-stream"
    }
}

The uploadUri property contains a pre-signed URI to use in the PUT request of step 2.

Notes

The Amazon S3 file container applies AES-256 server-side encryption to your data.
The file container is automatically deleted as soon as its content has been successfully forwarded to the service.
The uploadUri automatically expires after 60 minutes.

Therefore, it’s safe to upload sensitive information into the Amazon S3 file container.

The fileId property contains the unique identifier of your file container.

The requiredHeaders property contains the required HTTP headers for sending in the PUT request of step 2.

Step 2: Upload the item content to merge into the file container

To upload the content to merge into the file container you created in step 1, perform the following PUT request:

Request template

PUT <MyUploadURI> HTTP/1.1

<HTTPHeaders>

Where you replace:

<MyUploadURI> with the value of the uploadUri property from the step 1 HTTP response.
<HTTPHeaders> with the key-value pairs of the requiredHeaders object property from the step 1 HTTP response.

You can now upload your update data (JSON file) in the body of the request. For example, the following payload illustrates how you could replace metadata on existing catalog items:

Payload example

{
   "addOrMerge": [ 
       {
           "objecttype": "Product",
           "documentId": "product://010",
           "ec_name": "Sneaker 010",
           "productId": "010",
           "ec_category": "Sneakers",
           "gender": "Unisex",
           "departement": "Shoes"
       },
       {
           "objecttype": "Variant",
           "documentId": "variant://010-blue",
           "width": "wide"
       },
       // ...More items to add or merge...
   ]
}

In the request body:

For each item you include in the addOrMerge array, you must specify a unique documentId value. Therefore, make sure that all of your items contain a unique documentId whose value is an URI.

A successful response has no content, but indicates that the content update was successfully uploaded to the Amazon S3 file container, as in the following example:

200 OK

{}

The payload must be chunked into parts of no more than 256 MB.

Step 3: Send the file container to update your catalog (full item update)

To push the Amazon S3 file container into your source, use the Merge documents of a catalog stream source operation as follows:

Request template

PUT https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/sources/<MySourceId>/stream/merge?fileId=<MyFileId> HTTP/1.1

Content-Type: application/json
Authorization: Bearer <MyAccessToken>

Payload

{}

In the request path:

Replace <MyOrganizationId> with the ID of the target Coveo organization (see Retrieve the organization ID).
Replace <MySourceId> with the ID of the source which contains the catalog data that you want to merge with.

In the query string:

Replace <MyFileId> with the fileId from the step 1 HTTP response.

In the Authorization HTTP header:

Replace <MyAccessToken> with an access token that grants the Organization - View and Sources - Edit privileges in the target organization (see Create an API key programmatically and Get the privileges of an access token).

A successful response indicates that the operation was successfully forwarded to the service and that the batch of items is now enqueued to be processed by the Coveo indexing pipeline. It also contains a unique identifier for your merge request (requestId), as well as an orderingId that containing the time when your request was received, as in the following example:

202 Accepted

{
  "orderingId": 1716387965000,
  "requestId": "498ef728-1dc2-4b01-be5f-e8f8f1154a99"
}

What’s next?

Once you’re done merging your catalog data, we strongly recommend that you inspect your content and properties to ensure that your content was indexed correctly.