Manage batches of items in a Push source
Manage batches of items in a Push source
Managing items in batches is the best way to ensure that you never get throttled by the service (see Push API limits - Recommended maximum number of items/security identities per hour). Batch Push API requests allow you to forward many item updates to the service using only a few Push API requests, rather than performing hundreds (or thousands) of single item requests to achieve the same results.
Performing batch update Push API requests is slightly more complex than performing single Push API requests, as doing so involves three distinct steps which are detailed in this article.
|
Leading practice: Perform a security identity update
Whenever you add or update content in a secured Push source, you should ensure that the security identities referenced in item permission models are up to date in the security identity cache. In essence, you should always perform a security identity update on the security identity provider of a secured Push source before you perform a content update in that source (see Manage security identities in a security identity provider). |
Step 1: Create a file container
Perform a Create a file container
HTTP request.
Step 2: Upload the content update into the file container
Perform the following PUT uploadUri
request to upload the content update into the Amazon S3 file container you got from step 1.
PUT uploadUri
request in the context of a batch update sequence and illustrating how the Coveo indexing pipeline handles the batch update.Request template
PUT <MyUploadURI> HTTP/1.1
<HTTPHeaders>
Request parameters:
Parameters
Request body:
{
"addOrUpdate": [
{
<MyItemMetadata>*,
"documentId": <MyItemToAddOrUpdateURI>,
<"data"|"compressedBinaryData"|"compressedBinaryDataFileId">: <MyItemDataOrFileId>,
"compressionType": <"UNCOMPRESSED"|"DEFLATE"|"GZIP"|"LZMA"|"ZLIB">,
"fileExtension": <MyItemDataFileExtension>,
"parentId": <MyItemParentId>,
"permissions": <MyItemPermissionModel>
}*
],
"delete": [
{
"documentId": <MyItemToDeleteURI>,
"deleteChildren": <true|false>
}*
]
}
Parameters
The request body must implement the BatchDocumentBody
model.
For each item you include in the addOrUpdate
array:
-
Replace
<MyItemMetadata>*
with any number of arbitrary metadata key-values you want to include along with the item you’re adding or updating (see About Push source item metadata). -
Replace
<MyItemToAddOrUpdateURI>
with the URI of the item to add or update (for example,http://www.example.com/toadd.html
). Specifying a uniquedocumentId
value for each item is mandatory. -
Replace
<"data"|"compressedBinaryData"|"compressedBinaryDataFileId">
with the property you want to use to push the item data. You must also replace<MyItemDataOrFileId>
accordingly (see Pushing item data). -
If you’re using the
compressedBinaryData
or thecompressedBinaryDataFileId
property to push item data, replace<"UNCOMPRESSED"|"DEFLATE"|"GZIP"|"LZMA"|"ZLIB">
with the actual compression algorithm that was applied to the item data.Otherwise, you don’t need to include the
compressionType
property. -
(Recommended) Replace
<MyItemDataFileExtension>
with the actual file extension which the Push API should use to interpret the item data (for example,.txt
,.html
, etc.). This value must include the leading dot (.
) character. -
To establish a parent-child relationship between the item and other items in the same Push source, replace
<MyItemParentId>
with thedocumentId
of the parent item, or by thedocumentId
of the item your are adding or updating if this item itself is the parent.Otherwise, you don’t need to include the
parentId
property in your request body. -
If the target Push source is secured, replace
<MyItemPermissionModel>
with a valid permission model for the pushed item. See Push API reference - Item permission models, Simplified permission model, and lComplete permission model for details.Otherwise, you don’t need to include the
permissions
property.
For each item you include in the delete
array:
-
Replace
<MyItemToDeleteURI>
with the URI of the item to delete (for example,http://www.example.com/todelete.html
). -
Set the
deleteChildren
property totrue
to delete all items whosedocumentId
is nested under thedocumentId
to delete.The
deleteChildren
property is set tofalse
by default.
Successful response: 200 OK
{}
A successful response has no content, but indicates that the content update was successfully uploaded to the Amazon S3 file container.
Sample request
The following example shows how to upload a batch of item updates into a file container.
PUT link:https://coveo-nprod-customerdata.s3.amazonaws.com/proda/blobstore/mycoveocloudv2organizationg8tp8wu3/b5e8767e-8f0d-4a89-9095-1127915c89c7[...] HTTP/1.1
x-amz-server-side-encryption: AES256
Content-Type: application/octet-stream
Request body:
{
"addOrUpdate": [
{
// ...Metadata...
"documentId": "http://www.example.com/toadd.html",
"compressedBinaryData": "ZUp5enlTakp6YkhqVWdBQ200elV4QlFvVXgrSm5aU2ZVZ2xoUWxRWjJybFdKT1lXNUtRQ1ZSa2lTUlRZaFdSa0Zpc0FVV0tlUWlwRWlaNk5mb0VkRjZlTlBzUVFvQTZRZFFETWhCOEU=",
"compressionType": "DEFLATE",
"fileExtension": ".html",
"parentId": "http://www.example.com/",
"permissions": [
{
"allowAnonymous": false,
"allowedPermissions": [
// ...Allowed security identities...
],
"deniedPermissions": [
// ...Denied security identities...
]
},
// ...More permission sets...
]
},
// ...More items to add or update...
],
"delete": [
{
"documentId": "http://www.example.com/todelete.html",
"deleteChildren": true
},
// ...More items to delete...
]
}
Step 3: Push the file container into a Push source
Use the Add, update, and/or delete a batch of items
request to push the Amazon S3 file container into a Push source.
Add, update, and/or delete a batch of items
request in the context of a batch update sequence and illustrating how the Coveo indexing pipeline handles the batch update.Request template
US East region
PUT https://api.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/sources/<MySourceId>/documents/batch?fileId=<MyFileId> HTTP/1.1
Accept: application/json
Authorization: Bearer <MyAccessToken>
Canada region
PUT https://api-ca.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/sources/<MySourceId>/documents/batch?fileId=<MyFileId> HTTP/1.1
Accept: application/json
Authorization: Bearer <MyAccessToken>
Ireland region
PUT https://api-eu.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/sources/<MySourceId>/documents/batch?fileId=<MyFileId> HTTP/1.1
Accept: application/json
Authorization: Bearer <MyAccessToken>
Australia region
PUT https://api-au.cloud.coveo.com/push/v1/organizations/<MyOrganizationId>/sources/<MySourceId>/documents/batch?fileId=<MyFileId> HTTP/1.1
Accept: application/json
Authorization: Bearer <MyAccessToken>
Request parameters:
Parameters
In the request path:
-
Replace
<MyOrganizationId>
with the actual ID of the target Coveo organization. See Retrieve the organization ID for details. -
Replace
<MySourceId>
with the ID of the target Push source.
In the query string:
-
Replace
<MyFileId>
with thefiledId
you got from step 1.
In the Authorization
HTTP header:
-
Replace
<MyAccessToken>
with a Push source API key that grants the set of privileges required to push items to your source.
Request body:
{}
Successful response: 202 Accepted
null
A successful response (202 Accepted
) has no content, but indicates that the operation was successfully forwarded to the service and that the batch of items is now enqueued to be processed by the Coveo indexing pipeline.
This doesn’t imply that all items in the batch were successfully added, updated and/or deleted in the target Push source (see About the Push API processing delay).
|
The contents of a file container can be pushed to multiple sources or security identity providers in the same Coveo organization.
Just update the target The file container remains available for 4 days. |
Sample request
The following example shows how to push a file container into a Push source.
PUT https://api.cloud.coveo.com/push/v1/organizations/mycoveocloudv2organizationg8tp8wu3/sources/mycoveocloudv2organization-rp5rxzbdz753uhndklv2ztkfgy/documents/batch?fileId=b5e8767e-8f0d-4a89-9095-1127915c89c7 HTTP/1.1
Content-Type: application/json
Authorization: Bearer **********-****-****-****-************
Request body:
{}