--- title: Ingest data into a data collection slug: q6980563 canonical_url: https://docs.coveo.com/en/q6980563/ collection: data-collections source_format: adoc --- # Ingest data into a data collection This article explains how to ingest data into a [data collection](https://docs.coveo.com/en/q65b0594.md). You can use [update operations](#update-operations) to send incremental changes or [rebuild operations](#rebuild-operations) to replace the entire contents of the [data collection](https://docs.coveo.com/en/q65b0594.md). ## Prerequisites Make sure you have: * A data collection with the appropriate `dataCollectionType` value. [Create the data collection](https://docs.coveo.com/en/q65f3128.md) if you haven't created it yet. * An API key with the [privileges](https://docs.coveo.com/en/228.md) listed in the following table. See [Manage privileges](https://docs.coveo.com/en/3151.md) and [Privilege reference](https://docs.coveo.com/en/1707.md) for details. + -- | Actions | Service | Domain | Required access level | |---|---|---|---| | Create, edit, and delete data collection configurations | Content | Data collections | Edit | | Add, update, and delete items in a data collection | Organization | Organization | Edit | | Add, update, and delete items in a data collection | Content | Push items to sources | Allow for all sources | | Review data collection ingestion logs | Content | Data collections | View | | Review data collection ingestion logs | Organization | Organization | View | > **Notes** > > * The **Edit** privilege on the `Data collection` domain automatically grants the ability to create data collections. > See [Can Create ability dependence](https://docs.coveo.com/en/3151.md#can-create-ability-dependence) for more information. > > * Granting the **Edit** access level on a domain also implicitly grants **View** on that domain. -- ## Update operations Use the update operation sequence when you want to specify changes to the data collection content. You can add, partially update, or delete items. ### Step 1: Create a file container Create a temporary, private, and encrypted [Amazon S3 file container](https://docs.coveo.com/en/43.md) using the following request. Save the response body because it contains file container information that you'll use in the next steps of the update operation. **Request template** ```http POST https://api.cloud.coveo.com/push/v1/organizations//files?useVirtualHostedStyleUrl= HTTP/1.1 ​ Accept: application/json Content-Type: application/json Authorization: Bearer ``` **Request parameters**
Details In the request path: * Replace `` with your [organization ID](https://docs.coveo.com/en/148.md). In the query string: * Optionally, set `useVirtualHostedStyleUrl` to `true` if you want the service to return a virtual-hosted-style URL, such as `coveo-nprod-customerdata.s3.amazonaws.com/...`. The default value is currently `false`, which means that the service returns path-style URLs, such as `s3.amazonaws.com/coveo-nprod-customerdata/...`. + > **Important** > > The `useVirtualHostedStyleUrl` query string parameter will soon be deprecated as part of the [path-style URL deprecation](https://docs.coveo.com/en/43.md#path-style-url-deprecation). > From this point onwards, the service will only return virtual hosted-style URLs. In the `Authorization` HTTP header: * Replace `` with an [API key](https://docs.coveo.com/en/1718.md#create-a-custom-key) that has the [required privileges](#required-privileges) to push content to the data collection.
**Payload:** None **Successful response:** `201 Created` The body of a successful response contains important information about the created file container: ```json { "uploadUri": "", <1> "fileId": "", <2> "requiredHeaders": { <3> "x-amz-server-side-encryption": "AES256", "Content-Type": "application/octet-stream" } } ``` <1> The `uploadUri` property contains a pre-signed URI that you use to make a PUT request when pushing a batch of [data collection](https://docs.coveo.com/en/q65b0594.md) [items](https://docs.coveo.com/en/210.md) in step 2. + > **Notes** > > * The Amazon S3 file container applies [AES-256](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) server-side encryption to your data. > > * The file container is automatically deleted as soon as its content has been successfully forwarded to the service. > > * The `uploadUri` automatically expires after 60 minutes. > > Therefore, it's safe to upload sensitive information to the Amazon S3 file container. <2> The `fileId` property contains the unique identifier of your file container that you'll need in step 3. <3> The `requiredHeaders` property contains the required HTTP headers for sending a PUT request to the `uploadUri`. ### Step 2: Upload the update into the file container To upload the [data collection](https://docs.coveo.com/en/q65b0594.md) content update into the Amazon S3 file container you created in [step 1](#step-1-create-a-file-container), perform the following PUT request: **Request template:** ```http PUT HTTP/1.1 ​ ``` **Request parameters**
Details In the request path: * Replace `` with the value of the `uploadUri` property you received in the response when you created your file container in [step 1](#step-1-create-a-file-container). For the ``: * Enter the key-value pairs of the `requiredHeaders` object property you received in the response when you created your file container in [step 1](#step-1-create-a-file-container).
**Payload:** The payload must be a JSON of no more than 256 MB and can contain any combination of `addOrUpdate`, `partialUpdate`, and `delete` operations. ```json { "addOrUpdate": [ <1> { "itemId": "transaction-002", "timestamp": "2025-01-15T10:30:00.000Z", "currency": "USD", "transaction": { "revenue": 49.99 }, "products": [ { "product": { "productId": "SKU-1001", "price": 24.99 }, "quantity": 2 } ] }, { "itemId": "transaction-003", "timestamp": "2025-01-16T14:22:00.000Z", "currency": "USD", "transaction": { "revenue": 129.97 }, "products": [ { "product": { "productId": "SKU-2045", "price": 129.97 }, "quantity": 1 } ] } ], "partialUpdate": [ <2> { "itemId": "transaction-004", "operator": "fieldValueReplace", "field": "transaction", "value": { "revenue": 65.99 } } ], "delete": [ <3> { "itemId": "transaction-001" } ] } ``` <1> Each [item](https://docs.coveo.com/en/210.md) in the `addOrUpdate` array must adhere to the [relevant schema](#validation-schemas). <2> Each [item](https://docs.coveo.com/en/210.md) in the `partialUpdate` array must include the `itemId`, `operator`, `field`, and `value` properties. + .itemId
Details The unique identifier of the transaction to update. This must match an existing `itemId` in the [data collection](https://docs.coveo.com/en/q65b0594.md).
+ .operator
Details The partial update operator to apply. The following operators are supported: | Operator | Description | |---|---| | `fieldValueReplace` | Replaces a property value regardless of the original type. For example, updating the transaction `revenue`. To update two properties of the same item, include two separate entries in the `partialUpdate` array with the same `itemId` and different `field` values. | | `arrayAppend` | Adds elements to an array field. Can only be used on arrays of primitive values (for example, strings, numbers, Boolean values). | | `arrayRemove` | Removes elements from an array field. Can only be used on arrays of primitive values (for example, strings, numbers, Boolean values). |
+ .field
Details The name of the property to update.
+ .value
Details The value to set, add, or remove, depending on the chosen operator. For `fieldValueReplace`: provide the new value to set. For example, to update the transaction `revenue` to 65.99, set `field` to `transaction` and `value` to `{ "revenue": 65.99 }`. For `arrayAppend` or `arrayRemove`: provide an array of values to add or remove.
<3> Each [item](https://docs.coveo.com/en/210.md) in the `delete` array must specify the `itemId` property. **Successful response**: `200 OK` A successful response has no content, but indicates that the content update was successfully uploaded to the Amazon S3 file container. ### Step 3: Send the file container to update your data collection To send the file container to the [data collection](https://docs.coveo.com/en/q65b0594.md) and trigger the update processing, perform the following PUT request: **Request template:** ```http PUT https://api.cloud.coveo.com/push/v1/organizations//data-collections//stream/update?fileId= HTTP/1.1 Content-Type: application/json Authorization: Bearer ``` **Request parameters**
Details In the request path: * Replace `` with your [organization ID](https://docs.coveo.com/en/148.md). * Replace `` with the unique identifier of your data collection that you received when you [created the data collection](https://docs.coveo.com/en/q65f3128.md). In the query string: * Replace `` with the value of the `fileId` property you received in the response when you created your file container in [step 1](#step-1-create-a-file-container). The `fileId` may be used to [locate log entries](#review-ingestion-logs) related to this update in the [**Log Browser**](https://platform.cloud.coveo.com/admin/#/orgid/content/log-browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/log-browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/log-browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/log-browser/)). In the `Authorization` HTTP header: * Replace `` with an access token, such as an [API key](https://docs.coveo.com/en/1718.md) that has the [required privileges](#required-privileges) to push content to the data collection.
**Payload:** None **Successful response**: `202` A successful response indicates that the update operation was successfully queued for processing. The response body contains an `orderingId`, which represents the timestamp (in epoch milliseconds) when the operation was accepted, and a `requestId`. ```json { "orderingId": "1781207560610", "requestId": "28e37f9c-c65b-4ed3-9c09-7bf4135a5235" } ``` ## Rebuild operations Use the rebuild operation sequence when you want to replace the entire contents of a [data collection](https://docs.coveo.com/en/q65b0594.md). Any [item](https://docs.coveo.com/en/210.md) in the data collection not received during the rebuild will be deleted when the rebuild stream is closed. This is useful when you need to guarantee full data integrity. This involves a three-step process: ### Step 1: Open a stream To open a stream targeting the [data collection](https://docs.coveo.com/en/q65b0594.md), perform the following POST request: **Request template:** ```http POST https://api.cloud.coveo.com/push/v1/organizations//data-collections//stream/open HTTP/1.1 Content-Type: application/json Authorization: Bearer ``` **Request parameters**
Details In the request path: * Replace `` with your [organization ID](https://docs.coveo.com/en/148.md). * Replace `` with the unique identifier of your data collection that you received when you [created the data collection](https://docs.coveo.com/en/q65f3128.md). In the `Authorization` HTTP header: * Replace `` with an access token, such as an [API key](https://docs.coveo.com/en/1718.md) that has the [required privileges](#required-privileges) to push content to the data collection.
**Payload:** None **Successful response**: `201` A successful response indicates that the stream was successfully opened. The response body contains a `streamId` to identify the stream and an `uploadUri` for uploading items in step 2. ```json { "uploadUri": "https://coveo-nprod-customerdata.s3.amazonaws.com/stream/barcateamjqcy9k1a/284da6bc-6150-44fe-8821-c4ad0172a49e/07739df9-dfcf-46ee-80d0-b075d9a49c9f?X-Amz-Security-Token=IQoJb3JpZ2luX2VjED0aCXVzLWVhc3QtMSJGMEQCIDydGUFoV2choYYac%2FivQ5nP4czb5gjH8ysv1V9JX5s5AiBw9pkqMdDIE1P91liS9FuoOm0bF62yyZjTvYb0yK5oMirsAwgGEAAaDDA2NDc5MDE1NzE1NCIMBF4xEzSFuDPRVaudKskDGnY0VhCWNMY0%2FdYJ9yQSz6vvUwad7U0Z9Stbq41Cm3%2Fb0w8z%2FnxbvNF9Yr%2BzE3GlM7kM8i9riICfd5a7CUFE%2FgQ1ruVm3nokj2RZ%2ByaiJwTWij2tb2d%2FmA7gZFOz4jE%2FJBvBoe31hUbpf0iHbC2tGij3w%2B1%2BLasEpfFwjdQ2TA%2BCjlWQqVk1UJugcGUBFQulAPI61tiy28ekb2Lc6g1oItUjB9Tb2P26bMTgfhvkf0uBDx8J4SsyrnT%2BGUG1FU8I%2F5LG6CR2sAmNM3N06k2TSinHaISAB08rwIZF%2Fczh1XFKSUCntoDSQE4A7fa2oVRPqNhmI1a9RpuocK%2FUJ5eYnTvXWAZt71ZC0QSp%2FxSGJoIcaeL3zEu3l3yvNbp49qn7nxsWRXraVTbHzYORo7l3Z%2FSDfZeTZPNhfwBphqM3WuBbAHQ44rRokgfeUBLOAOniMienXzQjZ7IBqMSPSuzk5claaqEl7hRhqZQeip16PXY8Rv49OErThcJIfDiLQzrqTbbWbMeioWEJKQzH6r%2F8FHszeP9%2Fkj3P7aH49%2FyZ1%2BgRjVyV66AfwLgi443CMTqmXggirXEO8xYtvR%2B05uhP4%2FFvyDGUvyvPNTCBtazRBjqiAV33d26Owcls5Vv8LBVlRGgXodMmWr0RrvsXEhrEhur2u8%2Fc4i7Cub0vL9ZWkuIWy%2FG2ASCIuW6p%2FSJPrueOhuWKJ%2FgDPwT%2B8sCHEafh86DULrRfN5XWioopcwFifaMrXQonG56mC%2FzdOpFagiSaq9K4kFFfMkm90pqJUBgtw%2Bsf2GpX8yvLziOg%2FmbUDH%2Bos%2FUPMHLizV2zZc3mHrXy3bfkaA%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20260611T204759Z&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-server-side-encryption&X-Amz-Credential=ASIAQ6FOLK5RLGF4NLAF%2F20260611%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Expires=3600&X-Amz-Signature=6f23e14dad833e220e2fcbb8c8d89a97886c111286a8d35c71a1e74037299ce3", "fileId": "07739df9-dfcf-46ee-80d0-b075d9a49c9f", "requiredHeaders": { "x-amz-server-side-encryption": "AES256", "Content-Type": "application/octet-stream" }, "streamId": "284da6bc-6150-44fe-8821-c4ad0172a49e" } ``` ### Step 2: Upload items into the stream To upload the items into the stream you opened in [step 1](#step-1-open-a-stream), perform the following PUT request: **Request template:** ```http PUT HTTP/1.1 Content-Type: application/json x-amz-server-side-encryption: AES256 ``` **Request parameters**
Details In the request path: * Replace `` with the value of the `uploadUri` property you received in the response when you opened your stream in [step 1](#step-1-open-a-stream).
**Payload:** The payload must be a JSON of no more than 256 MB and can only contain an `addOrUpdate` array. ```json { "addOrUpdate": [ <1> { "itemId": "transaction-001", "timestamp": "2025-01-15T10:30:00.000Z", "currency": "USD", "transaction": { "revenue": 49.99 }, "products": [ { "product": { "productId": "SKU-1001", "price": 24.99 }, "quantity": 2 } ] }, { "itemId": "transaction-002", "timestamp": "2025-01-16T14:22:00.000Z", "currency": "USD", "transaction": { "revenue": 129.97 }, "products": [ { "product": { "productId": "SKU-2045", "price": 129.97 }, "quantity": 1 } ] } ] } ``` <1> Each [item](https://docs.coveo.com/en/210.md) in the `addOrUpdate` array must adhere to the [relevant schema](#validation-schemas). **Successful response**: `200 OK` A successful response has no content, but indicates that the items were successfully uploaded to the stream. ### Step 3: Close the stream To close the stream and synchronize the [data collection](https://docs.coveo.com/en/q65b0594.md), perform the following POST request: **Request template:** ```http POST https://api.cloud.coveo.com/push/v1/organizations//data-collections//stream//close HTTP/1.1 Content-Type: application/json Authorization: Bearer ``` **Request parameters**
Details In the request path: * Replace `` with your [organization ID](https://docs.coveo.com/en/148.md). * Replace `` with the unique identifier of your data collection that you received when you [created the data collection](https://docs.coveo.com/en/q65f3128.md). * Replace `` with the value of the `streamId` property you received in the response when you opened your stream in [step 1](#step-1-open-a-stream). The `streamId` may be used to [locate log entries](#review-ingestion-logs) related to this rebuild in the [**Log Browser**](https://platform.cloud.coveo.com/admin/#/orgid/content/log-browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/log-browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/log-browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/log-browser/)). In the `Authorization` HTTP header: * Replace `` with an access token, such as an [API key](https://docs.coveo.com/en/1718.md) that has the [required privileges](#required-privileges) to push content to the data collection.
**Payload:** None **Successful response**: `202` A successful response indicates that the stream was successfully closed and the rebuild operation was queued for processing. Any item not received during the rebuild will be removed from the [data collection](https://docs.coveo.com/en/q65b0594.md). ```json { "orderingId": "1781207560610", "requestId": "f8a5c2d1-3b4e-4c5f-8d9e-1a2b3c4d5e6f" } ``` ## Validation schemas To be ingested, [items](https://docs.coveo.com/en/210.md) in a data collection must adhere to the predefined schema for the collection's `dataCollectionType`. For each property, the schema indicates whether the property is required, its type, and any property value constraints. The `additionalProperties` property indicates whether properties not explicitly defined in the schema are allowed in the current object. The default value is `true`. **Offline purchases v1 schema**
Details ```json { "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://api.cloud.coveo.com/v1/data-collections/schemas/offline-purchases-schema-v1.json", "title": "Offline Purchases Schema", "description": "This schema is used to validate the format of Offline Purchases sent to Coveo through ingestion APIs. For more information, refer to the Data Collection documentation.", "type": "object", "required": [ "itemId", "timestamp", "transaction", "products" ], "additionalProperties": false, "properties": { "itemId": { "type": "string", "description": "Unique identifier for the transaction. When a transaction with an existing id is received, the previous record is overwritten.", "minLength": 1, "maxLength": 255 }, "currency": { "type": "string", "pattern": "^[A-Z]{3}$", "default": "USD", "description": "Currency code in upper-case ISO 4217 format." }, "timestamp": { "type": "string", "format": "date-time", "description": "Timestamp of when the transaction occurred, in ISO 8601 format." }, "transaction": { "type": "object", "required": ["revenue"], "additionalProperties": false, "properties": { "revenue": { "type": "number", "minimum": 0, "description": "Total revenue contained in the transaction. Currency must remain consistent within a transaction." } } }, "products": { "type": "array", "minItems": 1, "items": { "type": "object", "required": ["product", "quantity"], "additionalProperties": false, "properties": { "product": { "type": "object", "required": ["productId", "price"], "additionalProperties": false, "properties": { "productId": { "type": "string", "description": "Unique identifier of the product purchased." }, "price": { "type": "number", "minimum": 0, "description": "Price paid by the customer per unit of quantity." } } }, "quantity": { "type": "number", "minimum": 0, "description": "Quantity of the item purchased. The total revenue for the product in a transaction should be obtained by multiplying price and quantity." } } } } } } ```
## Review ingestion logs The [**Log Browser**](https://platform.cloud.coveo.com/admin/#/orgid/content/log-browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/log-browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/log-browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/log-browser/)) provides a unified interface to review ingestion logs for all your data collections. Use it to monitor the ingestion process and review any validation errors or other issues with your ingested data. ![Log Browser page showing data collection ingestion logs | Coveo Administration Console](https://docs.coveo.com/en/assets/images/data-collections/log-browser-for-data-collection-logs.png) | 1 | Operation timestamp | |---|---| | 2 | Operation type | | 3 | [Operation result](#operation-result) | | 4 | [Collection ID](https://docs.coveo.com/en/q65f3128.md#collection-id) | | 5 | The `Item URI` serves to identify specific items or batches: * For item-level logs, the `Item URI` represents the `itemId`. * For batch-level logs, the `Item URI` represents the `streamId` for rebuild operations or the `fileId` for update operations. * For the `Delete older than` operation that Coveo automatically performs when [closing a stream](#step-3-close-the-stream), the `Item URI` represents the `orderingId` property in the response.   Use the item URI filter (8) to search for specific items or batches | | 6 | [Log stage](#log-stage) | | 7 | Log entry expansion button | | 8 | Item URI filter to search for specific items or batches. | | 9 | Date filter | Logs are grouped into three stages: | Stage | Operation target | Description | | |---|---|---|---| | `Streaming` | Batch | This is the initial log entry recorded when a batch of items is received through the Stream API. | | | `Applying streaming extension` | Batch \ | item | This is the second processing stage during which the batch payload is chunked into `addOrUpdate`, `partialUpdate`, and `delete` objects, the contents of which are analyzed for conformance. | | `Validation` | Item | This is the final processing stage during which an added or updated item is validated against the `dataCollectionType` schema. | | The operation result indicates the outcome of the ingestion operation: icon:dot-green[alt=green,width=30,role=status-dot] Completed:: The operation was successful. icon:triangle-orange[alt=triangle-orange,width=30,role=warning-status] Warning:: The operation was unsuccessful or only partly successful, but the ingestion process was able to continue. Click icon:chevron-down[alt=chevron-down,width=16] near the right edge of the operation log entry for an error description and details about the information that wasn't ingested. icon:square-red[alt=square-red,width=30,role=error-status] Error:: The operation was unsuccessful and the ingestion process was stopped. Click icon:chevron-down[alt=chevron-down,width=16] near the right edge of the operation log entry for more details.