How to update your catalog

Ecommerce Developer Product Documentation

In this article

Leading practices
Full document updates
Partial document updates
Example

Catalog source updates are required when you need to add, remove, or update product documents in your source after an initial catalog upload (see How to stream your catalog data to your source) .

Leading practices

We support two update mechanisms for indexing updates to documents in a source:

Full document updates let you update or delete complete documents in your catalog.
Partial document updates let you update specific field(s) in an existing catalog source in order to reduce the number of calls made to the index and eliminate the need to rebuild the entire product list.

Reasons for using this update mechanism include:
- Updating a price field across a selection of documents.
- Adding or removing a variant's SKU from a store array.
- Updating the value of an existing product field.

The catalog manager responsible for the content identifies the file(s) and the field(s) that require an update from the full load, and pushes the corresponding documentId to the Stream API endpoint.

Note

The partialUpdate operation can only be performed during a catalog update, after an initial catalog upload.

Full document updates

Required parameters

Refer to the Push API (V1) reference for a comprehensive list of required parameters.

When the payload exceeds 256 MB, it must be chunked in 256 MB parts.

Step 1: create a file container

To perform a full catalog update, you must first create a file container.

Step 2: Upload the full document content into the file container

To upload the stream operations into the Amazon S3 file container you got from step 1, perform the following PUT request:

Request template

PUT <MyUploadURI>

<HTTPHeaders>

Where you replace:

<MyUploadURI> with the value of the uploadUri property you got in the response when you created your file container in step 1.
<HTTPHeaders> with the key-value pairs of the requiredHeaders object property you got in the response when you created your file container in step 1.

Payload example

{
   "addOrUpdate": [ 
       {
           "objecttype": "Product",
           "documentId": "https://www.acme.com/product/010",
           "ec_name": "Sneaker 010",
           "productId": "010",
           "ec_category": "Sneakers",
           "gender": "Unisex",
           "departement": "Shoes"
       },
       {
           "objecttype": "Product",
           "documentId": "https://www.acme.com/product/011",
           "ec_name": "Sneaker 011",
           "productId": "011",
           "ec_category": "Sneakers",
           "gender": "Unisex",
           "departement": "Shoes"
       },
       {
           "objecttype": "Variant",
           "documentId": "https://www.acme.com/product/010-blue",
           "ec_name": "Sneaker 010 Royal Blue",
           "productId": "010",
           "sku": "010-blue",
           "width": "wide",
           "productSize": "9"
       },
       // ...More items to add or update...
   ],
   "delete": [ 
       {
           "documentId": "https://www.acme.com/store/montreal"
       },
        // ...More items to delete...
   ]
}

In the request body (see Item models - BatchDocumentBody):

	For each item you include in the `addOrUpdate` array (see Item models - DocumentBody), specifying a unique `documentId` value for each item is mandatory.
	For each item you include in the `delete` array (see Item models - DeletedItem), specifying a unique `documentId` value for each item is mandatory.

A successful response (200 OK) has no content, but indicates that the content update was successfully uploaded to the Amazon S3 file container.

{}

Step 3: Send the file container to update your catalog

To push the Amazon S3 file container into a stream source, use the add, update, or delete a large number of encrypted items in a source operation as follows:

Request template

PUT https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/sources/<MySourceId>/stream/update?fileId=<MyFileId>

Content-Type: application/json
Authorization: Bearer <MyAccessToken>

Payload

{}

In the request path:

Replace <MyOrganizationId> with the actual ID of the target Coveo organization (see Retrieve the organization ID).
Replace <MySourceId> with the actual ID of the target stream source (see How to stream your catalog data to your source).

In the query string:

Replace <MyFileId> with the fileId you got from Step 1: create a file container.

In the Authorization HTTP header:

Replace <MyAccessToken> with an access token that grants the Organization - View and Sources - Edit privileges in the target organization (see Create an API key programatically and Get the privileges of an access token).

A successful response (202 Accepted) has no content, but indicates that the operation was successfully forwarded to the service and that the batch of items is now enqueued to be processed by the Coveo indexing pipeline, for example:

Successful response - 202 Accepted

null

Partial document updates

Structure and operations

The following example represents the basic structure of a partial document update:

{
  ...,
  "partialUpdate": [
    {
      "documentId": "<DOCUMENT_ID>",
      "operator": "<PARTIAL_UPDATE_OPERATOR>",
      "field": "<FIELD_NAME>",
      "value": <VALUE>
    },
    ...// more partial updates
  ]
}

Where you replace:

<DOCUMENT_ID> with the required unique identifier of the document to apply a partial update operation to.
<PARTIAL_UPDATE_OPERATOR> with one of the partial document update operators.
<FIELD_NAME> with the field name you wish to update.
<VALUE> with the operator value, depending on the partial update operation.

The partialUpdate operation is an addition to the existing update endpoint as a top-level sibling to the addOrUpdate and delete operations.

Partial document update operators

The following partial document update operators are treated sequentially.

Operator Description

Operator	Description
`arrayAppend`	Adds a list of elements to an array attribute, such as adding SKUs or making them available.
`arrayRemove`	Removes a list of elements from an array attribute, such as removing SKUs or making them unavailable.
`fieldValueReplace`	Replaces a field value regardless of the original value type.

arrayAppend

Adds a list of elements to an array attribute, such as adding SKUs or making them available.

arrayRemove

Removes a list of elements from an array attribute, such as removing SKUs or making them unavailable.

fieldValueReplace

Replaces a field value regardless of the original value type.

WARNING

If an operation in the partial document update is found to be invalid, it will be ignored.

Protected fields

The following fields aren’t updated during a partial update operation:

documentid
permanentid
A catalog’s product, variant, or availability object type field
A catalog’s product, variant, or availability id field

Object types and ID fields may vary depending on your configuration.

Step 1: create a file container

To perform a partial catalog update, you must first create a file container.

Step 2: Upload the partial document content into the file container

To upload the stream operations into the Amazon S3 file container you got from Step 1: create a file container, perform the following PUT request:

Request template

PUT <MyUploadURI>

<HTTPHeaders>

Where you replace:

<MyUploadURI> with the value of the uploadUri property you got in the response when you created your file container in Step 1: create a file container.
<HTTPHeaders> with the key-value pairs of the requiredHeaders object property you got in the response when you created your file container in Step 1: create a file container.

Add items

The following example shows how to make products or variants available for a store.

{
  ...,
  "partialUpdate": [
    {
      "documentId": "https://www.acme.com/store/010",
      "operator": "arrayAppend",
      "field": "availableSkus",
      "value": ["sku-224", "sku-225"]
    },
    ... // more partial updates
  ]
}

Remove items

The following example shows how to remove products or variants from a store.

{
  ...,
  "partialUpdate": [
    {
      "documentId": "https://www.acme.com/store/020",
      "operator": "arrayRemove",
      "field": "availableSkus",
      "value": ["sku-221", "sku-220"]
    },
    ... // more partial updates
  ]
}

Replace a field value

Note

When performing an update to the value of a product field that doesn’t exist, the field will be created.

The following example shows how to modify a price and a description of a specific item, as well as change the name of another item during the same update.

{
  ...,
  "partialUpdate": [
    {
      "documentId": "https://www.acme.com/product/030",
      "operator": "fieldValueReplace",
      "field": "ec_price",
      "value": 23.50
    },
    {
      "documentId": "https://www.acme.com/product/030",
      "operator": "fieldValueReplace",
      "field": "ec_description",
      "value": "Written representation of the change in description"
    },
    {
      "documentId": "https://www.acme.com/product/040",
      "operator": "fieldValueReplace",
      "field": "ec_name",
      "value": "Product name X"
    },
    ... // more partial updates
  ]
}

A successful response (200 OK) has no content, but indicates that the content update was successfully uploaded to the Amazon S3 file container.

Successful response - 200 OK

{}

Step 3: Send the file container to update your catalog

To push the Amazon S3 file container into a stream source, use the add, update, or delete a large number of encrypted items in a source operation as follows:

Request template

PUT https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/sources/<MySourceId>/stream/update?fileId=<MyFileId>

Content-Type: application/json
Authorization: Bearer <MyAccessToken>

Payload

{}

In the request path:

Replace <MyOrganizationId> with the actual ID of the target Coveo organization (see Retrieve the organization ID).
Replace <MySourceId> with the actual ID of the target stream source (see How to stream your catalog data to your source).

In the query string:

Replace <MyFileId> with the fileId you got from Step 1: create a file container.

In the Authorization HTTP header:

Replace <MyAccessToken> with an access token that grants the Organization - View and Sources - Edit privileges in the target organization (see Create an API key programatically and Get the privileges of an access token).

Successful response - 202 Accepted

null

Example

Below is a complete example where both full and partial update operators are used to update documents in a source.

{
  "addOrUpdate": [
    {
      "objecttype": "Product",
      "documentId": "https://www.acme.com/product/010",
      "ec_name": "Sneaker 010",
      "productId": "010",
      "ec_category": "Sneakers",
      "gender": "Unisex",
      "departement": "Shoes"
    }
  ],
  "delete": [
    {
      "documentId": "https://www.acme.com/store/montreal"
    }
  ],
  "partialUpdate": [
    {
      "documentId": "https://www.acme.com/store/010",
      "operator": "arrayAppend",
      "field": "availableSkus",
      "value": [
        "sku-224",
        "sku-225"
      ]
    },
    {
      "documentId": "https://www.acme.com/store/020",
      "operator": "arrayRemove",
      "field": "availableSkus",
      "value": [
        "sku-221",
        "sku-220"
      ]
    },
    {
      "documentId": "https://www.acme.com/product/030",
      "operator": "fieldValueReplace",
      "field": "ec_price",
      "value": 23.5
    }
  ]
}