Manage batches of items in a Push source

This is for:

In this article

Step 1: Create a File Container
Step 2: Upload the Content Update into the File Container
Step 3: Push the File Container into a Push Source

Manage items in batches is the best way to ensure that you never get throttled by the service (see Push API Limits - Recommended Maximum Number of Items/Security Identities per Hour). A batch Push API operation allows you to forward many push operations to the service using only a few Push API calls, rather than performing hundreds (or thousands) of single Push API calls to achieve the same results.

Performing batch Push API operations is slightly more complex than performing single Push API operations, as doing so involves three distinct steps which are detailed in this article.

Perform a security identity update

Whenever you add or update content in a secured Push source, you should ensure that the security identities referenced in item permission models are up to date in the security identity cache.

In essence, you should always perform a security identity update on the security identity provider of a secured Push source before you perform a content update in that source (see Manage security identities in a security identity provider).

Step 1: Create a File Container

Perform a POST /files HTTP request.

Step 2: Upload the Content Update into the File Container

Perform the following PUT uploadUri request to upload the push operations required for your content update into the Amazon S3 file container you got from Step 1: Create a File Container.

Sequence diagram for PUT uploadUri request

Sequence diagram highlighting the PUT uploadUri request in the context of a batch update sequence and illustrating how the Coveo indexing pipeline handles the batch update.

Request template

PUT <MyUploadURI> HTTP/1.1
  
<HTTPHeaders>

Payload (see Item Models - BatchDocumentBody)

{
  "addOrUpdate": [
    {
      <MyItemMetadata>*,
      "documentId": <MyItemToAddOrUpdateURI>,
      <"data"|"compressedBinaryData"|"compressedBinaryDataFileId">: <MyItemDataOrFileId>,
      "compressionType": <"UNCOMPRESSED"|"DEFLATE"|"GZIP"|"LZMA"|"ZLIB">,
      "fileExtension": <MyItemDataFileExtension>,
      "permissions": <MyItemPermissionModel>
    }*
  ],
  "delete": [
    {
      "documentId": <MyItemToDeleteURI>,
      "deleteChildren": <true|false>
    }*
  ]
}

Replace:

<MyUploadUri> with the value of the uploadUri property you got in the response when you created your file container in Step 1: Create a File Container.
<HTTPHeaders> with the keys-value pairs of the requiredHeaders object property you got in the response when you created your file container in Step 1: Create a File Container.

In the request body (see Item Models - BatchDocumentBody):

For each item you include in the addOrUpdate array (see Item Models - DocumentBody):
- Replace <MyItemMetadata>* with any number of arbitrary metadata key-values you want to include along with the item you’re adding or updating (see About Push Source Item Metadata).
- Replace <MyItemToAddOrUpdateURI> with the URI of the item to add or update (for example, http://www.example.com/toadd.html).
  
  Specifying a unique documentId value for each item is mandatory.
- Replace <"data"|"compressedBinaryData"|"compressedBinaryDataFileId"> with the property you want to use to push the item data. You must also replace <MyItemDataOrFileId> accordingly (see Pushing Item Data).
- If you’re using the compressedBinaryData or the compressedBinaryDataFileId property to push item data (see Using the compressedBinaryData Property and Using the compressedBinaryDataFileId Property):
  - Replace <"UNCOMPRESSED"|"DEFLATE"|"GZIP"|"LZMA"|"ZLIB"> with the actual compression algorithm that was applied to the item data.
    
    The compressionType value is case sensitive.
    
    Otherwise, you don’t need to include the compressionType property at all.
- Replace <MyItemDataFileExtension> with the actual file extension which the Push API should use to interpret the item data (for example, .txt, .html, etc.). This value must include a preceding dot (.) character.
  
  Although doing so is optional, you should generally specify a fileExtension.
- If you want to establish a parent-child relationship between the item and other items in the same Push source:
  - Replace <MyItemParentId> with the @documentId of the parent item, or by the @documentId of the item your are adding or updating if this item itself is the parent (see About the parentId Property).
    
    Otherwise, you don’t need to include the parentId property at all in your request body.
- If the target Push source is secured:
  - Replace <MyItemPermissionModel> by a valid permission model for the pushed item. See:
    - PermissionSetsModel / Simple Permission Model Definition Examples
    - PermissionLevelsModel / Complex Permission Model Definition Example
    Otherwise, you don’t need to include the permissions property at all.
For each item you include in the delete array (see Item Models - DeletedItem):
- Replace <MyItemToDeleteURI> with the URI of the item to delete (for example, http://www.example.com/todelete.html).
- Set the deleteChildren property to true if you want to recursively delete all items whose documentId is nested under the documentId to delete, or set it to false otherwise
  
  The deleteChildren property is set to false by default.

A successful response (200 OK) has no content, but indicates that the content update was successfully uploaded to the Amazon S3 file container.

Sample Request

Uploading a batch of push operations into a file container

PUT https://coveo-nprod-customerdata.s3.amazonaws.com/proda/blobstore/mycoveocloudv2organizationg8tp8wu3/b5e8767e-8f0d-4a89-9095-1127915c89c7[...] HTTP/1.1
  
x-amz-server-side-encryption: AES256
Content-Type: application/octet-stream

Payload

{
  "addOrUpdate": [
    {
      // ...Metadata...
      "documentId": "http://www.example.com/toadd.html",
      "compressedBinaryData": "ZUp5enlTakp6YkhqVWdBQ200elV4QlFvVXgrSm5aU2ZVZ2xoUWxRWjJybFdKT1lXNUtRQ1ZSa2lTUlRZaFdSa0Zpc0FVV0tlUWlwRWlaNk5mb0VkRjZlTlBzUVFvQTZRZFFETWhCOEU=",
      "compressionType": "DEFLATE",
      "fileExtension": ".html",
      "parentId": "http://www.example.com/",
      "permissions": [
        {
          "allowAnonymous": false,
          "allowedPermissions": [
            // ...Allowed security identities...
          ],
          "deniedPermissions": [
            // ...Denied security identities...
          ]
        },
        // ...More permission sets...
      ]
    },
    // ...More items to add or update...
  ],
  "delete": [
    {
      "documentId": "http://www.example.com/todelete.html",
      "deleteChildren": true
    },
    // ...More items to delete...
  ]
}

Successful response - 200 OK

{}

Step 3: Push the File Container into a Push Source

Use the PUT /sources/{sourceId}/documents/batch HTTP request to push the Amazon S3 file container into a Push source.

Sequence diagram for PUT batch request

Sequence diagram highlighting the PUT /sources/{sourceId}/documents/batch request in the context of a batch update sequence and illustrating how the Coveo indexing pipeline handles the batch update.

Request template

PUT https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/sources/<MySourceId>/documents/batch?fileId=<MyFileId> HTTP/1.1
  
Content-Type: application/json
Authorization: Bearer <MyAccessToken>

Payload

{}

In the request path:

Replace <MyOrganizationId> with the actual ID of the target Coveo organization (see Retrieve the organization ID).
Replace <MySourceId> with the actual ID of the target Push source (see Create a Push Source).

In the query string:

Replace <MyFileId> with the filedId you got from Step 1: Create a File Container.

In the Authorization HTTP header:

Replace <MyAccessToken> with an access token that grants the Sources - Edit privilege on the target source (see Create an API key, Get the privileges of an access token, and Get your Coveo access token).

A successful response (202 Accepted) has no content, but indicates that the operation was successfully forwarded to the service and that the batch of items is now enqueued to be processed by the Coveo indexing pipeline.

This doesn’t imply that all items in the batch were successfully added, updated and/or deleted in the target Push source (see About the Push API Processing Delay).

Update the status of your Push source

If you set your Push source to an active status before, you should consider setting it back to the IDLE status once this operation has successfully returned, assuming you have no more operations to perform as part of this content update (see Updating the Status of a Push Source).

Sample Request

Pushing a file container into a Push source

PUT https://api.cloud.coveo.com/push/v1/organizations/mycoveocloudv2organizationg8tp8wu3/sources/mycoveocloudv2organization-rp5rxzbdz753uhndklv2ztkfgy/documents/batch?fileId=b5e8767e-8f0d-4a89-9095-1127915c89c7 HTTP/1.1
  
Content-Type: application/json
Authorization: Bearer **********-****-****-****-************

Payload

{}

Successful response - 202 Accepted

null