Manage batches of items in a Push source

This is for:

In this article

Step 1: Create a file container
Step 2: Upload the content update into the file container
Step 3: Push the file container into a Push source

Managing items in batches is the best way to reduce the risk of getting throttled by the service. For details about the Push API rate limits, see Maximum number of items per time period. Batch Push API requests allow you to forward many item updates to the service using only a few Push API requests, rather than performing hundreds (or thousands) of single item requests to achieve the same results.

Performing batch update Push API requests is slightly more complex than performing single Push API requests, as doing so involves three distinct steps which are detailed in this article.

Leading practice: Perform a security identity update

Whenever you add or update content in a secured Push source, you should ensure that the security identities referenced in item permission models are up to date in the security identity cache.

In essence, you should always perform a security identity update on the security identity provider of a secured Push source before you perform a content update in that source. For more information, see Manage security identities in a security identity provider.

Step 1: Create a file container

Perform a Create a file container HTTP request.

Step 2: Upload the content update into the file container

Perform the following PUT uploadUri request to upload the content update into the Amazon S3 file container you got from step 1.

Sequence diagram for PUT upload URI request

Sequence diagram highlighting the PUT uploadUri request in the context of a batch update sequence and illustrating how the Coveo indexing pipeline handles the batch update.

Request template

PUT <MyUploadURI> HTTP/1.1

<HTTPHeaders>

Request parameters:

Parameters

Replace <MyUploadUri> with the value of the uploadUri property you got in the response when you created your file container in step 1.
Replace <HTTPHeaders> with the key-value pairs of the requiredHeaders object property you got in the response when you created your file container in step 1.

Request body:

{
  "addOrUpdate": [
    {
      <MyItemMetadata>*,
      "documentId": <MyItemToAddOrUpdateURI>,
      <"data"|"compressedBinaryData"|"compressedBinaryDataFileId">: <MyItemDataOrFileId>,
      "compressionType": <"UNCOMPRESSED"|"DEFLATE"|"GZIP"|"LZMA"|"ZLIB">,
      "fileExtension": <MyItemDataFileExtension>,
      "parentId": <MyItemParentId>,
      "permissions": <MyItemPermissionModel>
    }*
  ],
  "delete": [
    {
      "documentId": <MyItemToDeleteURI>,
      "deleteChildren": <true|false>
    }*
  ]
}

Parameters

The request body must implement the BatchDocumentBody model.

For each item you include in the addOrUpdate array:

Replace <MyItemMetadata>* with any number of arbitrary metadata key-values you want to include along with the item you’re adding or updating.
Replace <MyItemToAddOrUpdateURI> with the URI of the item to add or update (for example, http://www.example.com/toadd.html). Specifying a unique documentId value for each item is mandatory.
Replace <"data"|"compressedBinaryData"|"compressedBinaryDataFileId"> with the property you want to use to push the item data. You must also replace <MyItemDataOrFileId> accordingly. For more information, see Pushing item data.
If you’re using the compressedBinaryData or the compressedBinaryDataFileId property to push item data, replace <"UNCOMPRESSED"|"DEFLATE"|"GZIP"|"LZMA"|"ZLIB"> with the actual compression algorithm that was applied to the item data.

Otherwise, you don’t need to include the compressionType property.
(Recommended) Replace <MyItemDataFileExtension> with the actual file extension which the Push API should use to interpret the item data (for example, .txt, .html, etc.). This value must include the leading dot (.) character.
To establish a parent-child relationship between the item and other items in the same Push source, replace <MyItemParentId> with the documentId of the parent item, or the documentId of the item you’re adding or updating if this item is the parent.

Otherwise, you don’t need to include the parentId property in your request body.
If the target Push source is secured, replace <MyItemPermissionModel> with a valid permission model for the pushed item. See Push API reference - Item permission models, Simplified permission model, and lComplete permission model for details.

Otherwise, you don’t need to include the permissions property.

For each item you include in the delete array:

Replace <MyItemToDeleteURI> with the URI of the item to delete (for example, http://www.example.com/todelete.html).
Set the deleteChildren property to true to delete all items whose documentId is nested under the documentId to delete.

The deleteChildren property is set to false by default.

Successful response: 200 OK

A successful response has no content, but indicates that the content update was successfully uploaded to the Amazon S3 file container.

Sample request

The following example shows how to upload a batch of item updates into a file container.

PUT link:https://coveo-nprod-customerdata.s3.amazonaws.com/proda/blobstore/mycoveocloudv2organizationg8tp8wu3/b5e8767e-8f0d-4a89-9095-1127915c89c7[...] HTTP/1.1

x-amz-server-side-encryption: AES256
Content-Type: application/octet-stream

Request body:

{
  "addOrUpdate": [
    {
      // ...Metadata...
      "documentId": "http://www.example.com/toadd.html",
      "compressedBinaryData": "ZUp5enlTakp6YkhqVWdBQ200elV4QlFvVXgrSm5aU2ZVZ2xoUWxRWjJybFdKT1lXNUtRQ1ZSa2lTUlRZaFdSa0Zpc0FVV0tlUWlwRWlaNk5mb0VkRjZlTlBzUVFvQTZRZFFETWhCOEU=",
      "compressionType": "DEFLATE",
      "fileExtension": ".html",
      "parentId": "http://www.example.com/",
      "permissions": [
        {
          "allowAnonymous": false,
          "allowedPermissions": [
            // ...Allowed security identities...
          ],
          "deniedPermissions": [
            // ...Denied security identities...
          ]
        },
        // ...More permission sets...
      ]
    },
    // ...More items to add or update...
  ],
  "delete": [
    {
      "documentId": "http://www.example.com/todelete.html",
      "deleteChildren": true
    },
    // ...More items to delete...
  ]
}

Step 3: Push the file container into a Push source

Use the Add, update, and/or delete a batch of items request to push the Amazon S3 file container into a Push source.

Sequence diagram highlighting the Add, update, and/or delete a batch of items request in the context of a batch update sequence and illustrating how the Coveo indexing pipeline handles the batch update.

Request template

US East region

PUT https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/sources/<MySourceId>/documents/batch?fileId=<MyFileId> HTTP/1.1

Accept: application/json
Authorization: Bearer <MyAccessToken>

Canada region

PUT https://api-ca.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/sources/<MySourceId>/documents/batch?fileId=<MyFileId> HTTP/1.1

Accept: application/json
Authorization: Bearer <MyAccessToken>

Ireland region

PUT https://api-eu.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/sources/<MySourceId>/documents/batch?fileId=<MyFileId> HTTP/1.1

Accept: application/json
Authorization: Bearer <MyAccessToken>

Australia region

PUT https://api-au.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/sources/<MySourceId>/documents/batch?fileId=<MyFileId> HTTP/1.1

Accept: application/json
Authorization: Bearer <MyAccessToken>

Request parameters:

Parameters

In the request path:

Replace <MyOrganizationId> with your organization ID.
Replace <MySourceId> with the ID of the target Push source.

In the query string:

Replace <MyFileId> with the filedId you got from step 1.

In the Authorization HTTP header:

Replace <MyAccessToken> with a Push source API key that grants the set of privileges required to push items to your source.

Request body: None

Successful response: 202 Accepted

A successful response (202 Accepted) has no content, but indicates that the operation was successfully forwarded to the service and that the batch of items is now enqueued to be processed by the Coveo indexing pipeline. This doesn’t imply that all items in the batch were successfully added, updated and/or deleted in the target Push source. See About the Push API processing delay for details.

The contents of a file container can be pushed to multiple sources or security identity providers in the same Coveo organization. Just update the target sourceId/providerId and Authorization HTTP header access token in your other item or security identity batch requests.

The file container remains available for 4 days.

Sample request

The following example shows how to push a file container into a Push source.

PUT https://api.cloud.coveo.com/push/v1/organizations/mycoveocloudv2organizationg8tp8wu3/sources/mycoveocloudv2organization-rp5rxzbdz753uhndklv2ztkfgy/documents/batch?fileId=b5e8767e-8f0d-4a89-9095-1127915c89c7 HTTP/1.1

Authorization: Bearer **********-****-****-****-************
Accept: application/json