How to update your catalog

This is for:

Developer

Catalog source updates are required when you need to add, remove, or update product documents (also known as items) in your source after an initial stream.

Leading practices

Coveo supports two update mechanisms for indexing updates to documents in a source:

Full document updates

To perform a full document update, you must interact with the Coveo Push API. This section guides you through the different actions that must be taken to update your catalog items.

Required parameters

Refer to the Push API (V1) reference for a comprehensive list of required parameters.

Step 1: Create a file container

To perform a full document update, you must first create an Amazon S3 file container.

Use the Create a file container operation to create an Amazon S3 file container for a specific Coveo organization:

Request template

POST https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/files?useVirtualHostedStyleUrl=<true|false> HTTP/1.1

Content-Type: application/json
Accept: application/json
Authorization: Bearer <MyAccessToken>

Payload

{}

In the request path:

In the query string:

  • Optionally, set useVirtualHostedStyleUrl to true if you want the service to return the URL in virtual hosted-style (e.g., coveo-nprod-customerdata.s3.amazonaws.com/...). Default value is currently false, which means that the service returns path-style URLs (e.g., s3.amazonaws.com/coveo-nprod-customerdata/...).

    Important

    The useVirtualHostedStyleUrl query string parameter will soon be deprecated as part of the path-style URL deprecation. From this point onwards, the service will only return virtual hosted-style URLs.

In the Authorization HTTP header:

The body of a successful response contains important information about the temporary, private, and encrypted Amazon S3 file container you just created:

  • The uploadUri property contains a pre-signed URI to use in the PUT request of step 2.

    Notes

    • The Amazon S3 file container applies AES-256 server side encryption to your data (see Advanced Encryption Standard).

    • The file container is automatically deleted as soon as its content has been successfully forwarded to the service.

    • The uploadUri automatically expires after 60 minutes.

    Therefore, it’s perfectly safe to upload sensitive information into the Amazon S3 file container.

  • The fileId property contains the unique identifier of your file container.

  • The requiredHeaders property contains the required HTTP headers for sending in the PUT request of step 2.

Step 2: Upload the full document content into the file container

To upload the content to update into the Amazon S3 file container you got from step 1, perform the following PUT request:

Request template

PUT <MyUploadURI>

<HTTPHeaders>

Where you replace:

  • <MyUploadURI> with the value of the uploadUri property you received the response when you created your file container in step 1.

  • <HTTPHeaders> with the key-value pairs of the requiredHeaders object property you received the response when you created your file container in step 1.

You can now upload your update data (JSON file) in the body of the request. For example, the following update data is structured in JSON and has items that must be updated and an item that must be deleted:

Payload example

{
   "addOrUpdate": [ 1
       {
           "objecttype": "Product",
           "documentId": "https://www.acme.com/product/010",
           "ec_name": "Sneaker 010",
           "productId": "010",
           "ec_category": "Sneakers",
           "gender": "Unisex",
           "departement": "Shoes"
       },
       {
           "objecttype": "Product",
           "documentId": "https://www.acme.com/product/011",
           "ec_name": "Sneaker 011",
           "productId": "011",
           "ec_category": "Sneakers",
           "gender": "Unisex",
           "departement": "Shoes"
       },
       {
           "objecttype": "Variant",
           "documentId": "https://www.acme.com/product/010-blue",
           "ec_name": "Sneaker 010 Royal Blue",
           "productId": "010",
           "sku": "010-blue",
           "width": "wide",
           "productSize": "9"
       },
       // ...More items to add or update...
   ],
   "delete": [ 2
       {
           "documentId": "https://www.acme.com/store/montreal"
       },
        // ...More items to delete...
   ]
}

In the request body (see Item models - BatchDocumentBody):

1 For each item you include in the addOrUpdate array (see Item models - DocumentBody), specifying a unique documentId value for each item is mandatory. Therefore, you should make sure that all your items contain a documentId for which the value is a URI that uniquely identifies the item.
2 For each item you include in the delete array (see Item models - DeletedItem), specifying a unique documentId value for each item is mandatory.
Tip

A successful response (200 OK) has no content, but indicates that the content update was successfully uploaded to the Amazon S3 file container.

{}
Important

When the payload exceeds 256 MB, it must be chunked in 256 MB parts.

Step 3: Send the file container to update your catalog

To push the Amazon S3 file container into your source, use the add, update, or delete a large number of encrypted items in a source operation as follows:

Request template

PUT https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/sources/<MySourceId>/stream/update?fileId=<MyFileId>

Content-Type: application/json
Authorization: Bearer <MyAccessToken>

Payload

{}

In the request path:

  • Replace <MyOrganizationId> with the ID of the target Coveo organization (see Retrieve the organization ID).

  • Replace <MySourceId> with the ID of the source that contains the catalog data you want to update.

In the query string:

  • Replace <MyFileId> with the fileId you got from step 1.

In the Authorization HTTP header:

Tip

A successful response (202 Accepted) has no content, but indicates that the operation was successfully forwarded to the service and that the batch of items is now enqueued to be processed by the Coveo indexing pipeline, for example:

Successful response - 202 Accepted

null

Partial document updates

To perform a partial document update, you must interact with the Coveo Push API. This section guides you through the different actions that must be taken to update your catalog items.

Structure and operations

The following example represents the basic structure of a JSON file to send for a partial document update:

{
  ...,
  "partialUpdate": [
    {
      "documentId": "<DOCUMENT_ID>",
      "operator": "<PARTIAL_UPDATE_OPERATOR>",
      "field": "<FIELD_NAME>",
      "value": <VALUE>
    },
    ...// more items to apply a partial update
  ]
}

Where you replace:

  • <DOCUMENT_ID> with the required unique identifier of the document to apply a partial update operation to. Therefore, you should make sure that all your items contain a documentId for which the value is a URI that uniquely identifies the item.

  • <PARTIAL_UPDATE_OPERATOR> with one of the partial document update operators.

  • <FIELD_NAME> with the field name you wish to update.

  • <VALUE> depending on the chosen <PARTIAL_UPDATE_OPERATOR>, the <VALUE> can differ:

The partialUpdate operation is an addition to the existing update endpoint as a top-level sibling to the addOrUpdate and delete operations.

Note

The partialUpdate operation can only be performed during a catalog update, after an initial catalog upload.

Partial document update operators

The following partial document update operators are treated sequentially.

Operator Description

arrayAppend

Adds a list of elements to an array attribute, such as adding SKUs or making them available.

arrayRemove

Removes a list of elements from an array attribute, such as removing SKUs or making them unavailable.

fieldValueReplace

Replaces a field value regardless of the original value type.
Warning
WARNING

If an operation in the partial document update is found to be invalid, it will be ignored.

Protected fields

The following fields aren’t updated during a partial update operation:

  • documentid

  • permanentid

  • A catalog’s product, variant, or availability object type field

  • A catalog’s product, variant, or availability id field

Object types and ID fields may vary depending on your configuration.

Step 1: Create a file container

To perform a partial document update, you must first create an Amazon S3 file container.

Use the Create a file container operation to create an Amazon S3 file container for a specific Coveo organization:

Request template

POST https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/files?useVirtualHostedStyleUrl=<true|false> HTTP/1.1

Content-Type: application/json
Accept: application/json
Authorization: Bearer <MyAccessToken>

Payload

{}

In the request path:

In the query string:

  • Optionally, set useVirtualHostedStyleUrl to true if you want the service to return the URL in virtual hosted-style (e.g., coveo-nprod-customerdata.s3.amazonaws.com/...). Default value is currently false, which means that the service returns path-style URLs (e.g., s3.amazonaws.com/coveo-nprod-customerdata/...).

    Important

    The useVirtualHostedStyleUrl query string parameter will soon be deprecated as part of the path-style URL deprecation. From this point onwards, the service will only return virtual hosted-style URLs.

In the Authorization HTTP header:

The body of a successful response contains important information about the temporary, private, and encrypted Amazon S3 file container you just created:

  • The uploadUri property contains a pre-signed URI to use in the PUT request of step 2.

    Notes

    • The Amazon S3 file container applies AES-256 server side encryption to your data (see Advanced Encryption Standard).

    • The file container is automatically deleted as soon as its content has been successfully forwarded to the service.

    • The uploadUri automatically expires after 60 minutes.

    Therefore, it’s perfectly safe to upload sensitive information into the Amazon S3 file container.

  • The fileId property contains the unique identifier of your file container.

  • The requiredHeaders property contains the required HTTP headers for sending in the PUT request of step 2.

Step 2: Upload the partial document content into the file container

To upload the update file into the Amazon S3 file container you got from step 1, perform the following PUT request:

Request template

PUT <MyUploadURI>

<HTTPHeaders>

Where you replace:

  • <MyUploadURI> with the value of the uploadUri property you got in the response when you created your file container in step 1.

  • <HTTPHeaders> with the key-value pairs of the requiredHeaders object property you got in the response when you created your file container in step 1.

Tip

A successful response (200 OK) has no content, but indicates that the content update was successfully uploaded to the Amazon S3 file container.

Successful response - 200 OK

{}

Add items

The following example shows how to make products or variants available for a store.

{
  ...,
  "partialUpdate": [
    {
      "documentId": "https://www.acme.com/store/010",
      "operator": "arrayAppend",
      "field": "availableSkus",
      "value": ["sku-224", "sku-225"]
    },
    ... // more partial updates
  ]
}

Remove items

The following example shows how to remove products or variants from a store.

{
  ...,
  "partialUpdate": [
    {
      "documentId": "https://www.acme.com/store/020",
      "operator": "arrayRemove",
      "field": "availableSkus",
      "value": ["sku-221", "sku-220"]
    },
    ... // more partial updates
  ]
}

Replace a field value

Note

When performing an update to the value of a product field that doesn’t exist, the field will be created.

The following example shows how to modify a price and a description of a specific item, as well as change the name of another item during the same update.

{
  ...,
  "partialUpdate": [
    {
      "documentId": "https://www.acme.com/product/030",
      "operator": "fieldValueReplace",
      "field": "ec_price",
      "value": 23.50
    },
    {
      "documentId": "https://www.acme.com/product/030",
      "operator": "fieldValueReplace",
      "field": "ec_description",
      "value": "Written representation of the change in description"
    },
    {
      "documentId": "https://www.acme.com/product/040",
      "operator": "fieldValueReplace",
      "field": "ec_name",
      "value": "Product name X"
    },
    ... // more partial updates
  ]
}

Step 3: Send the file container to update your catalog

To push the Amazon S3 file container into your source, use the add, update, or delete a large number of encrypted items in a source operation as follows:

Request template

PUT https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/sources/<MySourceId>/stream/update?fileId=<MyFileId>

Content-Type: application/json
Authorization: Bearer <MyAccessToken>

Payload

{}

In the request path:

  • Replace <MyOrganizationId> with the ID of the target Coveo organization (see Retrieve the organization ID).

  • Replace <MySourceId> with the ID of the source that contains the catalog data you want to update.

In the query string:

  • Replace <MyFileId> with the fileId you got from step 1.

In the Authorization HTTP header:

Tip

A successful response (202 Accepted) has no content, but indicates that the operation was successfully forwarded to the service and that the batch of items is now enqueued to be processed by the Coveo indexing pipeline, for example:

Successful response - 202 Accepted

null

Example

Below is a complete example where both full and partial update operators are used to update documents in a source.

{
  "addOrUpdate": [
    {
      "objecttype": "Product",
      "documentId": "https://www.acme.com/product/010",
      "ec_name": "Sneaker 010",
      "productId": "010",
      "ec_category": "Sneakers",
      "gender": "Unisex",
      "departement": "Shoes"
    }
  ],
  "delete": [
    {
      "documentId": "https://www.acme.com/store/montreal"
    }
  ],
  "partialUpdate": [
    {
      "documentId": "https://www.acme.com/store/010",
      "operator": "arrayAppend",
      "field": "availableSkus",
      "value": [
        "sku-224",
        "sku-225"
      ]
    },
    {
      "documentId": "https://www.acme.com/store/020",
      "operator": "arrayRemove",
      "field": "availableSkus",
      "value": [
        "sku-221",
        "sku-220"
      ]
    },
    {
      "documentId": "https://www.acme.com/product/030",
      "operator": "fieldValueReplace",
      "field": "ec_price",
      "value": 23.5
    }
  ]
}

What’s next

Once you are done with your updating your catalog data, we strongly recommend that you inspect your content and properties to ensure that your content was correcly indexed.