How to update your catalog
How to update your catalog
Catalog source updates are required when you need to add, remove, or update items (product, variant, or availabilities) in your source after an initial stream.
Leading practices
Coveo supports two update mechanisms for indexing updates to items in a Catalog source:
-
Full item updates let you:
-
Update all fields in a given item.
-
Add and delete items from your Catalog source.
-
-
Partial item updates let you update one or more fields in an existing Catalog source. Using this method reduces the number of calls made to the index and removes the need to rebuild the entire product list.
For example, you may want to use the partial item update mechanism to:
Full item updates
To perform a full item update, you must interact with the Coveo Push API. This section guides you through the different actions that must be taken to update your catalog items.
Required parameters
Refer to the Push API (V1) reference for a comprehensive list of required parameters.
Step 1: Create a file container
To perform a full document update, you must first create an Amazon S3 file container.
Use the Create a file container operation to create an Amazon S3 file container for a specific Coveo organization:
Request template
POST https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/files?useVirtualHostedStyleUrl=<true|false> HTTP/1.1
Content-Type: application/json
Accept: application/json
Authorization: Bearer <MyAccessToken>
Payload
{}
In the request path:
-
Replace
<MyOrganizationId>
with the ID of the target Coveo organization (see Retrieve the organization ID).
In the query string:
-
Optionally, set
useVirtualHostedStyleUrl
totrue
if you want the service to return the URL in virtual hosted-style (for example,coveo-nprod-customerdata.s3.amazonaws.com/...
). Default value is currentlyfalse
, which means that the service returns path-style URLs (for example,s3.amazonaws.com/coveo-nprod-customerdata/...
).The
useVirtualHostedStyleUrl
query string parameter will soon be deprecated as part of the path-style URL deprecation. From this point onwards, the service will only return virtual hosted-style URLs.
In the Authorization
HTTP header:
-
Replace
<MyAccessToken>
with an access token that grants Organization - View privileges in the target Coveo organization (see Get your Coveo access token, Create an API key, Manage privileges, and Get the privileges of an access token).
The body of a successful response contains important information about the temporary, private, and encrypted Amazon S3 file container you just created:
|
Successful response - 200 OK
|
-
The
uploadUri
property contains a pre-signed URI to use in thePUT
request of step 2.Notes-
The Amazon S3 file container applies AES-256 server side encryption to your data (see Advanced Encryption Standard).
-
The file container is automatically deleted as soon as its content has been successfully forwarded to the service.
-
The
uploadUri
automatically expires after 60 minutes.
Therefore, it’s perfectly safe to upload sensitive information into the Amazon S3 file container.
-
-
The
fileId
property contains the unique identifier of your file container. -
The
requiredHeaders
property contains the required HTTP headers for sending in thePUT
request of step 2.
Step 2: Upload the full item content into the file container
To upload the content to update into the Amazon S3 file container you got from step 1, perform the following PUT
request:
Request template
PUT <MyUploadURI>
<HTTPHeaders>
Where you replace:
You can now upload your update data (JSON file) in the body of the request. For example, the following update data is structured in JSON and has items that must be updated and an item that must be deleted:
Payload example
{
"addOrUpdate": [
{
"objecttype": "Product",
"documentId": "product://010",
"ec_name": "Sneaker 010",
"productId": "010",
"ec_category": "Sneakers",
"gender": "Unisex",
"departement": "Shoes"
},
{
"objecttype": "Product",
"documentId": "product://011",
"ec_name": "Sneaker 011",
"productId": "011",
"ec_category": "Sneakers",
"gender": "Unisex",
"departement": "Shoes"
},
{
"objecttype": "Variant",
"documentId": "variant://010-blue",
"ec_name": "Sneaker 010 Royal Blue",
"productId": "010",
"sku": "010-blue",
"width": "wide",
"productSize": "9"
},
// ...More items to add or update...
],
"delete": [
{
"documentId": "store://s000001"
},
// ...More items to delete...
]
}
In the request body (see Item models - BatchDocumentBody):
For each item you include in the addOrUpdate array (see Item models - DocumentBody), specifying a unique documentId value for each item is mandatory.
Therefore, you should make sure that all your items contain a documentId for which the value is a URI that uniquely identifies the item. |
|
For each item you include in the delete array (see Item models - DeletedItem), specifying a unique documentId value for each item is mandatory. |
|
A successful response (
|
|
When the payload exceeds 256 MB, it must be chunked in 256 MB parts. |
Step 3: Send the file container to update your catalog
To push the Amazon S3 file container into your source, use the add, update, or delete a large number of encrypted items in a source operation as follows:
Request template
PUT https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/sources/<MySourceId>/stream/update?fileId=<MyFileId>
Content-Type: application/json
Authorization: Bearer <MyAccessToken>
Payload
{}
In the request path:
-
Replace
<MyOrganizationId>
with the ID of the target Coveo organization (see Retrieve the organization ID). -
Replace
<MySourceId>
with the ID of the source that contains the catalog data you want to update.
In the query string:
-
Replace
<MyFileId>
with thefileId
you got from step 1.
In the Authorization
HTTP header:
-
Replace
<MyAccessToken>
with an access token that grants the Organization - View and Sources - Edit privileges in the target organization (see Create an API key programmatically and Get the privileges of an access token).
|
A successful response ( Successful response - 202 Accepted
|
Partial item updates
To perform a partial item update, you must interact with the Coveo Push API. This section guides you through the different actions that must be taken to update your catalog items.
Structure and operations
The following example represents the basic structure of a JSON file to send for a partial item update:
{
...,
"partialUpdate": [
{
"documentId": "<DOCUMENT_ID>",
"operator": "<PARTIAL_UPDATE_OPERATOR>",
"field": "<FIELD_NAME>",
"value": <VALUE>
},
...// more items to apply a partial update
]
}
Where you replace:
-
<DOCUMENT_ID>
with the required unique identifier of the document to apply a partial update operation to. Therefore, you should make sure that all your items contain adocumentId
for which the value is a URI that uniquely identifies the item. -
<PARTIAL_UPDATE_OPERATOR>
with one of the partial item update operators. -
<FIELD_NAME>
with the field name you wish to update. -
<VALUE>
depending on the chosen<PARTIAL_UPDATE_OPERATOR>
, the<VALUE>
can differ:-
To add items, see Add items
-
To remove items, see Remove items
-
To replace a field value, see Replace a field value
-
To update a dictionary field, see Dictionary field update
-
The partialUpdate
operation is an addition to the existing update endpoint as a top-level sibling to the addOrUpdate
and delete
operations.
|
Note
The |
Partial item update operators
The following partial item update operators are treated sequentially.
Operator | Description |
---|---|
|
Adds a list of elements to an array attribute, such as adding SKUs or making them available. |
|
Removes a list of elements from an array attribute, such as removing SKUs or making them unavailable. |
|
Replaces a field value regardless of the original value type. |
|
Adds a key-value pair to the dictionary as long as the field already exists.
|
|
Removes a key-value pair as long as the key value already exists. Removing single key:
When removing multiple keys:
|
|
WARNING
If an operation in the partial item update is found to be invalid, it will be ignored. |
Protected fields
The following fields aren’t updated during a partial update operation:
-
documentid
-
permanentid
-
A catalog’s product, variant, or availability object type field
-
A catalog’s product, variant, or availability id field
Object types and ID fields may vary depending on your configuration.
Step 1: Create a file container
To perform a partial document update, you must first create an Amazon S3 file container.
Use the Create a file container operation to create an Amazon S3 file container for a specific Coveo organization:
Request template
POST https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/files?useVirtualHostedStyleUrl=<true|false> HTTP/1.1
Content-Type: application/json
Accept: application/json
Authorization: Bearer <MyAccessToken>
Payload
{}
In the request path:
-
Replace
<MyOrganizationId>
with the ID of the target Coveo organization (see Retrieve the organization ID).
In the query string:
-
Optionally, set
useVirtualHostedStyleUrl
totrue
if you want the service to return the URL in virtual hosted-style (for example,coveo-nprod-customerdata.s3.amazonaws.com/...
). Default value is currentlyfalse
, which means that the service returns path-style URLs (for example,s3.amazonaws.com/coveo-nprod-customerdata/...
).The
useVirtualHostedStyleUrl
query string parameter will soon be deprecated as part of the path-style URL deprecation. From this point onwards, the service will only return virtual hosted-style URLs.
In the Authorization
HTTP header:
-
Replace
<MyAccessToken>
with an access token that grants Organization - View privileges in the target Coveo organization (see Get your Coveo access token, Create an API key, Manage privileges, and Get the privileges of an access token).
The body of a successful response contains important information about the temporary, private, and encrypted Amazon S3 file container you just created:
|
Successful response - 200 OK
|
-
The
uploadUri
property contains a pre-signed URI to use in thePUT
request of step 2.Notes-
The Amazon S3 file container applies AES-256 server side encryption to your data (see Advanced Encryption Standard).
-
The file container is automatically deleted as soon as its content has been successfully forwarded to the service.
-
The
uploadUri
automatically expires after 60 minutes.
Therefore, it’s perfectly safe to upload sensitive information into the Amazon S3 file container.
-
-
The
fileId
property contains the unique identifier of your file container. -
The
requiredHeaders
property contains the required HTTP headers for sending in thePUT
request of step 2.
Step 2: Upload the partial item content into the file container
To upload the update file into the Amazon S3 file container you got from step 1, perform the following PUT
request:
Request template
PUT <MyUploadURI>
<HTTPHeaders>
Where you replace:
|
A successful response ( Successful response - 200 OK
|
Add items
The following example shows how to make products or variants available for a store.
{
...,
"partialUpdate": [
{
"documentId": "store://s000001",
"operator": "arrayAppend",
"field": "availableSkus",
"value": ["sku-224", "sku-225"]
},
... // more partial updates
]
}
Remove items
The following example shows how to remove products or variants from a store.
{
...,
"partialUpdate": [
{
"documentId": "store://s000001",
"operator": "arrayRemove",
"field": "availableSkus",
"value": ["sku-221", "sku-220"]
},
... // more partial updates
]
}
Replace a field value
|
Note
When performing an update to the value of a product field that doesn’t exist, the field will be created. |
The following example shows how to modify a price and a description of a specific item, as well as change the name of another item during the same update.
{
...,
"partialUpdate": [
{
"documentId": "product://010",
"operator": "fieldValueReplace",
"field": "ec_price",
"value": 23.50
},
{
"documentId": "product://010",
"operator": "fieldValueReplace",
"field": "ec_description",
"value": "Written representation of the change in description"
},
{
"documentId": "product://040",
"operator": "fieldValueReplace",
"field": "ec_name",
"value": "Product name X"
},
... // more partial updates
]
}
Dictionary field update
This partial item updates method allows you to partially update dictionary fields on a Catalog source item. When performing this update, if the dictionary field to update doesn’t exist, the field will be added as a single-item dictionary. The examples below demonstrate how to update dictionary fields for items of the product object type. However, this method is applicable to any other catalog object type, such as variants or availability.
|
Caution should be exercised when sending multiple updates targeting the same dictionary field in rapid succession through distinct HTTP calls. If an update to apply is received, and is older than the most recently applied update, the system will ignore the older message. This emphasizes the importance of ensuring proper order and sequencing when sending updates to prevent unintentional data inconsistencies. |
Consider this initial product metadata that contains dictionary field key-value pairs of types of ice-cream and their respective SKUs:
{
"addOrUpdate":
[
{
"documentId": "product://ice-cream_0001",
"objecttype": "Product",
"productid": "ice-cream_0001",
"ec_price": 9.99
"flavours": {
"Vanilla": "SKU-0001",
"Chocolate": "SKU-0002",
"Pumpkin": "SKU-0003",
"Cinnamon": "SKU-0004",
"Candies": "SKU-0005"
}
}
]
}
The user decides to add a new flavour "Moka": "SKU-123456"
to the existing list.
The JSON file of the partial dictionary update would look like the following:
{
"partialUpdate": [
{
"documentId": "product://ice-cream_0001",
"operator": "dictionaryPut",
"field": "flavours",
"value": {
"Moka": "SKU-123456"
}
}
]
}
Afterwich, the resulting updated dictionary would look like this:
{
"documentId": "product://ice-cream_0001",
"objecttype": "Product",
"productid": "ice-cream_0001",
"ec_price": 9.99
"flavours": {
"Vanilla": "SKU-0001",
"Chocolate": "SKU-0002",
"Pumpkin": "SKU-0003",
"Cinnamon": "SKU-0004",
"Candies": "SKU-0005",
"Moka": "SKU-123456"
}
}
To remove a key, such as the "Pumpkin"
flavour from the existing list, you need only identify the key as follows:
{
"partialUpdate": [
{
"documentId": "product://ice-cream_0001",
"operator": "dictionaryRemove",
"field": "flavours",
"value": "Pumpkin"
}
]
}
To remove multiple keys, such as the Pumpkin, Cinnamon, and Candies flavours from the dictionary, enter all the keys in an array as follows:
{
"partialUpdate": [
{
"documentId": "product://ice-cream_0001",
"operator": "dictionaryRemove",
"field": "flavours",
"value": [ "Pumpkin", "Cinnamon", "Candies" ]
}
]
}
Step 3: Send the file container to update your catalog
To push the Amazon S3 file container into your source, use the add, update, or delete a large number of encrypted items in a source operation as follows:
Request template
PUT https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/sources/<MySourceId>/stream/update?fileId=<MyFileId>
Content-Type: application/json
Authorization: Bearer <MyAccessToken>
Payload
{}
In the request path:
-
Replace
<MyOrganizationId>
with the ID of the target Coveo organization (see Retrieve the organization ID). -
Replace
<MySourceId>
with the ID of the source that contains the catalog data you want to update.
In the query string:
-
Replace
<MyFileId>
with thefileId
you got from step 1.
In the Authorization
HTTP header:
-
Replace
<MyAccessToken>
with an access token that grants the Organization - View and Sources - Edit privileges in the target organization (see Create an API key programmatically and Get the privileges of an access token).
|
A successful response ( Successful response - 202 Accepted
|
Example
Below is a complete example where both full and partial update operators are used to update documents in a source.
{
"addOrUpdate": [
{
"objecttype": "Product",
"documentId": "product://010",
"ec_name": "Sneaker 010",
"productId": "010",
"ec_category": "Sneakers",
"gender": "Unisex",
"departement": "Shoes"
}
],
"delete": [
{
"documentId": "store://s000001"
}
],
"partialUpdate": [
{
"documentId": "store://s000003",
"operator": "arrayAppend",
"field": "availableSkus",
"value": [
"sku-224",
"sku-225"
]
},
{
"documentId": "store://s000005",
"operator": "arrayRemove",
"field": "availableSkus",
"value": [
"sku-221",
"sku-220"
]
},
{
"documentId": "product://030",
"operator": "fieldValueReplace",
"field": "ec_price",
"value": 23.5
}
]
}
What’s next
Once you’re done updating your catalog data, we strongly recommend that you inspect your content and properties to ensure that your content was correcly indexed.