Managing Batches of Items in a Push Source

Managing items in batches is the best way to ensure that you never get throttled by the service (see Push API Limits - Recommended Maximum Number of Items/Security Identities per Hour). A batch Push API operation allows you to forward a large number of push operations to the service using only a few Push API calls, rather than performing hundreds (or thousands) of single Push API calls to achieve the same results.

Performing batch Push API operations is slightly more complex than performing single Push API operations, as doing so involves three distinct steps which are detailed in this topic.

Step 1 - Create a File Container

See Creating a File Container.

Step 2 - Upload the Content Update into the File Container

Perform the following PUT request to upload the push operations required for your content update into the Amazon S3 file container you got from Step 1 - Create a File Container.

PUT <MyUploadURI> HTTP/1.1
 
<HTTPHeaders>

{
  "addOrUpdate": [
    {
      <MyItemMetadata>*,
      "documentId": <MyItemToAddOrUpdateURI>,
      <"data"|"compressedBinaryData"|"compressedBinaryDataFileId">: <MyItemDataOrFileId>,
      "compressionType": <"UNCOMPRESSED"|"DEFLATE"|"GZIP"|"LZMA"|"ZLIB">,
      "fileExtension": <MyItemDataFileExtension>,
      "permissions": <MyItemPermissionModel>
    }*
  ],
  "delete": [
    {
      "documentId": <MyItemToDeleteURI>,
      "deleteChildren": <true|false>
    }*
  ]
}

Replace:

• <MyUploadUri> by the value of the uploadUri property you got in the response when you created your file container at Step 1 - Create a File Container.
• <HTTPHeaders> by the keys-value pairs of the requiredHeaders object property you got in the response when you created your file container at Step 1 - Create a File Container.

In the request body (see Item Models - BatchDocumentBody):

• For each item you include in the addOrUpdate array (see Item Models - DocumentBody):
  • Replace <MyItemMetadata>* by any number of arbitrary metadata key-values you want to include along with the item you are adding or updating (see Understanding Push Source Item Metadata).
  • Replace <MyItemToAddOrUpdateURI> by the URI of the item to add or update (e.g., http://www.example.com/toadd.html).
  • Replace <"data"|"compressedBinaryData"|"compressedBinaryDataFileId"> by the property you want to use to push the item data. You must also replace <MyItemDataOrFileId> accordingly (see Pushing Item Data).

  • If you are using the compressedBinaryData or the compressedBinaryDataFileId property to push item data (see Using the compressedBinaryData Property and Using the compressedBinaryDataFileId Property):

    • Replace <"UNCOMPRESSED"|"DEFLATE"|"GZIP"|"LZMA"|"ZLIB"> by the actual compression algorithm that was applied to the item data.

      The compressionType value is case sensitive.

      Otherwise, you do not need to include the compressionType property at all.

  • Replace <MyItemDataFileExtension> by the actual file extension which the Push API should use to interpret the item data (e.g., .txt, .html, etc.). This value must include a preceding dot (.) character.

    While specifying a fileExtension is optional, doing so is considered best practice.

  • If you want to identify the item as the child of another item in the index, replace <MyItemParentId> by the URI of the parent item (see Understanding the parentId Property).

    Otherwise, you do not need to include the parentId property at all in your request body.

  • If the target Push source is secured:

    • Replace <MyItemPermissionModel> by a valid permission model for the pushed item (see Simple Permission Model Definition Examples or Complex Permission Model Definition Example).

      Otherwise, you do not need to include the permissions property at all.

• For each item you include in the delete array (see Item Models - DeletedItem):
  • Replace <MyItemToDeleteURI> by the URI of the item to delete (e.g., http://www.example.com/todelete.html).

  • Set the deleteChildren property to true if you want to delete all items identified as descendants of the specified item to delete, or set it to false otherwise (see Understanding the parentId Property).

    The deleteChildren property is set to false by default.

A successful response (200 OK) has no content, but indicates that the content update was successfully uploaded to the Amazon S3 file container.

Sample Request

PUT https://s3.amazonaws.com/coveo-nprod-customerdata/proda/blobstore/mycoveocloudv2organizationg8tp8wu3/b5e8767e-8f0d-4a89-9095-1127915c89c7[...] HTTP/1.1
 
x-amz-server-side-encryption: AES256
Content-Type: application/octet-stream

{
  "addOrUpdate": [
    {
      // ...Metadata...
      "documentId": "http://www.example.com/toadd.html",
      "compressedBinaryData": "ZUp5enlTakp6YkhqVWdBQ200elV4QlFvVXgrSm5aU2ZVZ2xoUWxRWjJybFdKT1lXNUtRQ1ZSa2lTUlRZaFdSa0Zpc0FVV0tlUWlwRWlaNk5mb0VkRjZlTlBzUVFvQTZRZFFETWhCOEU=",
      "compressionType": "Deflate",
      "fileExtension": ".html",
      "parentId": "http://www.example.com/",
      "permissions": [
        {
          "allowAnonymous": false,
          "allowedPermissions": [
            // ...Allowed security identities...
          ],
          "deniedPermissions": [
            // ...Denied security identities...
          ]
        },
        // ...More permission sets...
      ]
    },
    // ...More items to add or update...
  ],
  "delete": [
    {
      "documentId": "http://www.example.com/todelete.html",
      "deleteChildren": true
    },
    // ...More items to delete...
  ]
}

{}

Step 3 - Push the File Container into a Push Source

Use the Add, update or delete a large number of encrypted items in a source operation to push the Amazon S3 file container into a Push source.

PUT https://push.cloud.coveo.com/v1/organizations/<MyOrganizationId>/sources/<MySourceId>/documents/batch?fileId=<MyFileId> HTTP/1.1
 
Content-Type: application/json
Accept: application/json
Authorization: Bearer <MyAccessToken>

{}

In the request path:

• Replace <MyOrganizationId> by the actual ID of the target Coveo Cloud V2 organization (see Retrieving the ID of a Coveo Cloud V2 Organization).
• Replace <MySourceId> by the actual ID of the target Push source (see Creating a Push Source and Retrieving the ID of a Source).

In the query string:

• Replace <MyFileId> by the filedId you got from Step 1 - Create a File Container.

In the Authorization HTTP header:

• Replace <MyAccessToken> by an access token that grants the Organization - View and Sources - View/Edit privileges in the target organization (see Creating an API key, Getting the Privileges of an Access Token, and Getting Your Coveo Cloud V2 Platform Access Token).

A successful response (202 Accepted) has no content, but indicates that the operation was successfully forwarded to the service and that the batch of items is now enqueued to be processed by the Coveo Cloud V2 indexing pipeline.

Sample Request

PUT https://push.cloud.coveo.com/v1/organizations/mycoveocloudv2organizationg8tp8wu3/sources/mycoveocloudv2organization-rp5rxzbdz753uhndklv2ztkfgy/documents/batch?fileId=b5e8767e-8f0d-4a89-9095-1127915c89c7 HTTP/1.1
 
Content-Type: application/json
Accept: application/json
Authorization: Bearer **********-****-****-****-************

{}

{}