--- title: Full catalog data updates slug: p4eb0129 canonical_url: https://docs.coveo.com/en/p4eb0129/ collection: coveo-for-commerce source_format: adoc --- # Full catalog data updates To fully update your [catalog data](https://docs.coveo.com/en/obcf0333/) in your [source](https://docs.coveo.com/en/246/) (typically a [Catalog source](https://docs.coveo.com/en/l5if0244/)), you have to interact with the [Coveo Stream API](https://docs.coveo.com/en/12#tag/Stream). It supports two types of operations to perform full updates on your catalog data, each suited to specific use cases: * [Update operations](#update-operations): This operation is typically used to update individual [items](https://docs.coveo.com/en/pa8f6515/) in your source, but it can also be used to update the entire catalog data. It performs full document replacements—any fields not included in the update payload for an item will be removed from that item in the index. However, items not included in the payload are left unchanged and remain in the source. * [Load operations](#load-operations): Also known as open and close **Stream**, this operation overwrites the entire catalog data in your [source](https://docs.coveo.com/en/246/) with the provided data. This means that if you don't include in the payload an item that was previously indexed, it will be automatically removed from the source. ## Prerequisites To perform the operations listed in this article, you must have: * Created a [Catalog source](https://docs.coveo.com/en/n8of0593/) or [equivalent stream-enabled source](https://docs.coveo.com/en/3448#alternatives-to-the-catalog-source). * Created and configured a [catalog entity](https://docs.coveo.com/en/3139/) that uses the source containing your [catalog data](https://docs.coveo.com/en/obcf0333/). ## Leading practices * [Update operations](#update-operations) should be favored over [load operations](#load-operations) to push or update your [catalog data](https://docs.coveo.com/en/obcf0333/) in your source. They provide the same benefits as load operations but are more efficient and have fewer limitations. Load operations require uploading and processing all file containers at once, which is resource-intensive and delays data availability until the entire load completes. In contrast, update operations process each container as soon as it's ready, allowing for faster indexing and more up-to-date catalog data throughout the update process. * Any update to your [catalog data](https://docs.coveo.com/en/obcf0333/) should be done using either [update operations](#update-operations) or [partial item updates operations](https://docs.coveo.com/en/p4eb0515#partial-item-updates). ## Catalog data structure To perform full catalog data updates, you need to prepare a JSON file containing your [catalog data](https://docs.coveo.com/en/obcf0333/). This structure can vary in many ways depending on your use case. This is typically a combination of the [catalog objects](https://docs.coveo.com/en/ncig0154/): products, [variants](https://docs.coveo.com/en/mc7f0326/), and [availabilities](https://docs.coveo.com/en/mc7e9096/). This structure is then used to create a [catalog configuration](https://docs.coveo.com/en/l5if0520/) in the [Coveo Administration Console](https://docs.coveo.com/en/183/). > **Important** > > * The `objecttype` [source item type](https://docs.coveo.com/en/1965/) is crucial when defining your product data structure. > It's essential for categorizing items into product, variant, or availability [catalog objects](https://docs.coveo.com/en/ncig0154/). > > * If the file size exceeds 256 MB, you must split the content into multiple files. > See [Uploading large catalog data files](#uploading-large-catalog-data-files) for instructions. > > * Availability data can be sent to a separate source, meaning that your setup may not require availability data in the same source as your product and variant data. The JSON file must contain an object for each item (product, variant, or availability) that you want to index in your source. For instructions on how to configure items for the different catalog object types, see: ** [Product data](https://docs.coveo.com/en/m53g7119/) ** [Variant data](https://docs.coveo.com/en/m53g0506/) ** [Availability data](https://docs.coveo.com/en/m53g0124/) **Example** The following [catalog data](https://docs.coveo.com/en/obcf0333/) (structured in JSON) contains objects that represent products, variants, and availabilities: ```json { "addOrUpdate": [ { "documentId": "product://001-red", "FileExtension": ".html", "ec_name": "Coveo Soccer Shoes - Red", "model": "Authentic", "ec_brand": ["Coveo"], "ec_description": "

The astonishing, the original, and always relevant Coveo style.

", "color": ["Red"], "ec_item_group_id": "001", "ec_product_id": "001-red", "ec_images": ["https://myimagegallery?productid"], "gender": "Men", "ec_price": 28.00, "ec_category": "Soccer Shoes", "objecttype": "Product" }, { "documentId": "variant://001-red-8_wide", "FileExtension": ".html", "ec_name": "Coveo Soccer Shoes - Red / Size 8 - Wide", "ec_variant_id": "001-red-8_wide", "productsize": "8", "width": "wide", "ec_product_id": "001-red", "objecttype": "Variant" }, { "documentId": "store://s000002", "title": "Montreal Store", "lat": 45.4975, "long": -73.5687, "ec_available_items": ["001-red-8_wide","001-red-9_wide","001-red-10_wide","001-red-11_wide", "001-blue-8_wide"], "ec_availability_id": "s000002", "objecttype": "Availability" } ] } ``` ## Update operations Update operations let you build and update your entire catalog data. It doesn't overwrite the entire catalog data in your [source](https://docs.coveo.com/en/246/), meaning that if you don't include in the payload an item that was previously indexed, it will remain in the source. If certain metadata exists in the source but is missing from the payload, it will be removed meaning the item is fully replaced by the new version. This is ideal when you need to update all fields of an item, rather than just a subset, without affecting other items in the source. > **Leading practice** > > If you only need to update certain metadata in an item (for example, updating a product price), you should use one of the [partial catalog data update mechanisms](https://docs.coveo.com/en/p4eb0515/) instead. To perform a full [item](https://docs.coveo.com/en/pa8f6515/) update, you must interact with the [Coveo Stream API](https://docs.coveo.com/en/12#tag/Stream). This section guides you through the different actions that must be taken to update your [catalog data](https://docs.coveo.com/en/obcf0333/). Refer to the [Stream API reference](https://docs.coveo.com/en/12#tag/Stream) for a comprehensive list of required parameters. ### Step 1: Create a file container (Update operation) > **Important** > > Make sure that you meet the [prerequisites](#prerequisites) before performing this operation. To perform a full document update, you must first [create an Amazon S3 file container](https://docs.coveo.com/en/43/). Use the [Create a file container](https://docs.coveo.com/en/12/api-reference/push-api#tag/File-Container/paths/~1organizations~1%7BorganizationId%7D~1files/post) operation to create an Amazon S3 file container for a specific [Coveo organization](https://docs.coveo.com/en/185/): **Request template** ```http POST https://api.cloud.coveo.com/push/v1/organizations//files?useVirtualHostedStyleUrl= HTTP/1.1 ​ Accept: application/json Content-Type: application/json Authorization: Bearer ``` In the request path: * Replace `` with your [organization ID](https://docs.coveo.com/en/148/). In the query string: * Optionally, set `useVirtualHostedStyleUrl` to `true` if you want the service to return a virtual hosted-style URL, such as `+coveo-nprod-customerdata.s3.amazonaws.com/...+`. The default value is currently `false`, which means that the service returns path-style URLs, such as `+s3.amazonaws.com/coveo-nprod-customerdata/...+`. > **Important** > > The `useVirtualHostedStyleUrl` query string parameter will soon be deprecated as part of the [path-style URL deprecation](https://docs.coveo.com/en/43#path-style-url-deprecation). > From this point onwards, the service will only return virtual hosted-style URLs. In the `Authorization` HTTP header: * Replace `` with an access token, such as an [API key](https://docs.coveo.com/en/1718/) that has the [required privileges](#required-privileges) to push content to the source. **Payload**: None The body of a successful response contains important information about the temporary, private, and encrypted Amazon S3 file container that you just created: ```json { "uploadUri": "", <1> "fileId": "", <2> "requiredHeaders": { <3> "x-amz-server-side-encryption": "AES256", "Content-Type": "application/octet-stream" } } ``` <1> The `uploadUri` property contains a pre-signed URI to use in the PUT request of [step 2](#step-2-upload-the-full-item-content-into-the-file-container). > **Notes** > > * The Amazon S3 file container applies [AES-256](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) server-side encryption to your data. > > * The file container is automatically deleted as soon as its content has been successfully forwarded to the service. > > * The `uploadUri` automatically expires after 60 minutes. > > Therefore, it's safe to upload sensitive information into the Amazon S3 file container. <2> The `fileId` property contains the unique identifier of your file container. You must use this value to send the file container to the source in step 3. <3> The `requiredHeaders` property contains the required HTTP headers for sending in the PUT request of [step 2](#step-2-upload-the-full-item-content-into-the-file-container). ### Step 2: Upload the full item content into the file container To upload the content to update into the Amazon S3 file container you got from [step 1](#step-1-create-a-file-container-update-operation), perform the following PUT request: **Request template** ```http PUT HTTP/1.1 ​ ``` Where you replace: * `` with the value of the `uploadUri` property you received the response when you created your file container in [step 1](#step-1-create-a-file-container-update-operation). * `` with the key-value pairs of the `requiredHeaders` object property you received in the response when you created your file container in [step 1](#step-1-create-a-file-container-update-operation). You can now upload your update data in the body of the request. You must structure the data in JSON format and upload it as either a JSON file or a file compressed with the ZLib algorithm. For example, the following update data is structured in JSON and has [items](https://docs.coveo.com/en/pa8f6515/) that must be updated and an [item](https://docs.coveo.com/en/pa8f6515/) that must be deleted: **Payload example** ```json { "addOrUpdate": [ <1> { "objecttype": "Product", "documentId": "product://010", "ec_name": "Sneaker 010", "ec_product_id": "010", "ec_category": "Sneakers", "gender": "Unisex", "departement": "Shoes" }, { "objecttype": "Product", "documentId": "product://011", "ec_name": "Sneaker 011", "ec_product_id": "011", "ec_category": "Sneakers", "gender": "Unisex", "departement": "Shoes" }, { "objecttype": "Variant", "documentId": "variant://010-blue", "ec_name": "Sneaker 010 Royal Blue", "ec_product_id": "010", "ec_variant_id": "010-blue", "width": "wide", "productSize": "9" } ], "delete": [ <2> { "documentId": "store://s000001" } ] } ``` In the [request body](https://docs.coveo.com/en/78#batchdocumentbody-model): <1> For each [item](https://docs.coveo.com/en/pa8f6515/) you include in the [`addOrUpdate`](https://docs.coveo.com/en/78#addorupdate-array) array, specifying a unique `documentId` value is mandatory. Therefore, you should make sure that all of your [items](https://docs.coveo.com/en/pa8f6515/) contain a `documentId` for which the value is a URI that uniquely identifies the [item](https://docs.coveo.com/en/pa8f6515/). This value must be a valid URL with a proper URI prefix, such as `product://`, or any other scheme that fits your [catalog data](https://docs.coveo.com/en/obcf0333/). <2> For each [item](https://docs.coveo.com/en/pa8f6515/) you include in the [`delete`](https://docs.coveo.com/en/78#delete-array) array, specifying a unique `documentId` value is mandatory. This value must be a valid URL with a proper URI prefix, such as `product://`, or any other scheme that fits your [catalog data](https://docs.coveo.com/en/obcf0333/). A successful response has no content, but indicates that the content update was successfully uploaded to the Amazon S3 file container, as in the following example: **200 OK** ```json {} ``` > **Important** > > When the payload exceeds 256 MB, it must be chunked into 256 MB parts. > See [Uploading large catalog data files](#uploading-large-catalog-data-files) for instructions. ### Step 3: Send the file container to update your source (Update operation) To push the Amazon S3 file container into your [source](https://docs.coveo.com/en/246/), use the [Update a catalog stream source](https://platform.cloud.coveo.com/docs?urls.primaryName=PushAPI#/Stream/put_organizations%5F%5ForganizationId%5F%5Fsources%5F%5FsourceId%5F%5Fstream_update) operation as follows: **Request template** ```http PUT https://api.cloud.coveo.com/push/v1/organizations//sources//stream/update?fileId= HTTP/1.1 ​ Content-Type: application/json Authorization: Bearer ``` **Payload** ```json {} ``` Where you replace: * `` with the ID of the target [Coveo organization](https://docs.coveo.com/en/185/) (see [Retrieve the organization ID](https://docs.coveo.com/en/148/)). * `` with the [ID](https://docs.coveo.com/en/3390#copy-a-source-name-or-id) of the [source](https://docs.coveo.com/en/246/) which contains the [catalog data](https://docs.coveo.com/en/obcf0333/) that you want to update. * `` with the `fileId` you got from step 1. * `` with an access token, such as an [API key](https://docs.coveo.com/en/1718/) that has the [required privileges](#required-privileges) to push content to the source. A successful response (`202`) indicates that the operation was successfully forwarded to the service and that the batch of [items](https://docs.coveo.com/en/pa8f6515/) is now enqueued to be processed by the [Coveo indexing pipeline](https://docs.coveo.com/en/184/). For example: **202 Accepted** ```text { "orderingId": 1716387965000, <1> "requestId": "498ef728-1dc2-4b01-be5f-e8f8f1154a99" <2> } ``` Where: <1> [`orderingId`](https://docs.coveo.com/en/147/) indicates the time your request was received. You must use this value if you want to [delete items that were present in the source before the update](#step-4-delete-old-items). <2> `requestId` is the unique identifier for your request. > **Tip** > > The contents of a file container can be pushed to multiple sources in the same [Coveo organization](https://docs.coveo.com/en/185/). > Just update the target `sourceId` and Authorization HTTP header access token in your other Stream API update or [merge requests](https://docs.coveo.com/en/p4eb0515#shallow-merge-operations). > > The file container remains available for 4 days. ### Step 4: Delete old items When performing a full item update, you're either adding catalog data to your source for the first time, or replacing your whole catalog data with newer data. To make sure old items that were previously indexed are removed from your source, you must delete them. [subs="-quotes,-specialcharacters"] The [Delete old documents](https://docs.coveo.com/en/12/api-reference/push-api#tag/Stream/paths/~1organizations~1%7BorganizationId%7D~1sources~1%7BsourceId%7D~1stream~1deleteolderthan~1%7BepochMilliseconds%7D/post) operation of the Stream API deletes items that are older than a specified date. To delete old items, you must perform the following POST request: **Request template** ```http POST https://api.cloud.coveo.com/push/v1/organizations//sources//stream/deleteolderthan/ HTTP/1.1 Content-Type: application/json Authorization: Bearer ``` Where you replace: * `` with the unique identifier of your organization (see [Find your organization ID](https://docs.coveo.com/en/n1ce5273/)). * `` with the [unique identifier of the source](https://docs.coveo.com/en/3390#copy-a-source-name-or-id) to which you want to push content. * `` with the value of the `orderingId` you received when you sent the file container to update your source in [step 3](#step-3-send-the-file-container-to-update-your-source-update-operation). If you have to push multiple file containers, you must use the `orderingId` of the first file container you sent to update your source. * `` with an access token, such as an [API key](https://docs.coveo.com/en/1718/) that has the [required privileges](#required-privileges) to push content to the source. A successful response will produce the HTTP response code **201 Created** without any content. ## Load operations The load operation (also known as stream) overwrites the entire [catalog data](https://docs.coveo.com/en/obcf0333/) in your [source](https://docs.coveo.com/en/246/). A load operation uses the catalog data you send in the request to completely replace the existing data in the source. > **Important** > > * Load operations should be used with caution due to their [limitations](#limitations). > > * Load operations aren't recommended for sources that contain more than 1,000,000 items. Performing a load operation via the Stream API involves the following steps: . [Open a stream](#step-1-open-a-stream) . [Upload your catalog data into the stream](#step-2-upload-your-catalog-data-into-the-stream) . [Close the stream](#step-3-close-the-stream) ### Limitations [Update operations](#update-operations) should be favored over the load operation for both pushing and performing full updates on your [catalog data](https://docs.coveo.com/en/obcf0333/). The load operation has limitations that can affect the performance and reliability of your catalog data updates: * **Content deletion**: When using load operations, indexed items that aren't sent in the request will be automatically removed from your source. To prevent the accidental deletion of a substantial number of items from a source, the delete operation is skipped during the process if all of the existing items were to be deleted. Perform an [update operation](#update-operations) to intentionally delete indexed items. > **Important** > > When your source isn't used with a [catalog configuration](https://docs.coveo.com/en/l5if0520/), and you open and close a stream with an empty JSON file, all of the content from your source will be deleted. * **Delayed data ingestion**: When using the load operation to push or fully update your [catalog data](https://docs.coveo.com/en/obcf0333/), the index waits until the entire catalog data is uploaded before starting the ingestion process. This means that there's a delay in the availability of the updated data, causing a mismatch between the data in your system and the data in the Coveo index. * **Lack of batch processing**: When using the load operation to update your [catalog data](https://docs.coveo.com/en/obcf0333/), you must push your entire catalog data every time you want to update it. To avoid these limitations, consider using [update operations](#update-operations) instead. ### Step 1: Open a stream > **Important** > > Make sure that you meet the [prerequisites](#prerequisites) before performing this operation. The first step is to open a stream using the Stream API. To achieve this, you must perform the following POST request: ```http POST https://api.cloud.coveo.com/push/v1/organizations/{organizationId}/sources/{sourceId}/stream/open HTTP/1.1 Content-Type: application/json Accept: application/json Authorization: Bearer ``` Where you replace: * `{organizationId}` with the unique identifier of your organization (see [Find your organization ID](https://docs.coveo.com/en/n1ce5273/)). * `{sourceId}` with the [unique identifier of the source](https://docs.coveo.com/en/3390#copy-a-source-name-or-id) to which you want to push content. * `` with an access token, such as an [API key](https://docs.coveo.com/en/1718/) that has the [required privileges](https://docs.coveo.com/en/n8of0593#required-privileges) to push content to the source. If your request is successful, you'll get the HTTP response code 201. This will get you a response that looks like this: ```json { "streamId": "1234-5678-9101-1121", "uploadUri": "link:https://coveo-nprod-customerdata.s3.amazonaws.com/[...]", "fileId": "b5e8767e-8f0d-4a89-9095-1127915c89c7", "requiredHeaders": { "x-amz-server-side-encryption": "AES256", "Content-Type": "application/octet-stream" } } ``` > **Important** > > * Take note of the generated `streamId` and `uploadUri` values, as you'll need them in the next steps. > > * The `uploadUri` is valid for one hour. ### Step 2: Upload your catalog data into the stream To upload your [catalog data](https://docs.coveo.com/en/obcf0333/) into the stream, you must structure the data in JSON format and upload it as either a JSON file or a file compressed with the ZLib algorithm. Use the following Stream API PUT request: ```http PUT {uploadUri} HTTP/1.1 x-amz-server-side-encryption: AES256 Content-Type: application/octet-stream ``` * Where you replace `{uploadUri}` with the `uploadUri` you received when you opened the stream in [step 1](#step-1-open-a-stream). * The `x-amz-server-side-encryption` and `Content-Type` parameters are authentication headers and so should be included in the request headers section instead of the body of the request. You can now upload your [catalog data](https://docs.coveo.com/en/obcf0333/). **Example** The following [catalog data](https://docs.coveo.com/en/obcf0333/) (structured in JSON) contains objects that represent products, variants, and availabilities: ```json { "addOrUpdate": [ { "documentId": "product://001-red", "FileExtension": ".html", "ec_name": "Coveo Soccer Shoes - Red", "model": "Authentic", "ec_brand": ["Coveo"], "ec_description": "

The astonishing, the original, and always relevant Coveo style.

", "color": ["Red"], "ec_item_group_id": "001", "ec_product_id": "001-red", "ec_images": ["https://myimagegallery?productid"], "gender": "Men", "ec_price": 28.00, "ec_category": "Soccer Shoes", "objecttype": "Product" }, { "documentId": "variant://001-red-8_wide", "FileExtension": ".html", "ec_name": "Coveo Soccer Shoes - Red / Size 8 - Wide", "ec_variant_id": "001-red-8_wide", "productsize": "8", "width": "wide", "ec_product_id": "001-red", "objecttype": "Variant" }, { "documentId": "store://s000002", "title": "Montreal Store", "lat": 45.4975, "long": -73.5687, "ec_available_items": ["001-red-8_wide","001-red-9_wide","001-red-10_wide","001-red-11_wide", "001-blue-8_wide"], "ec_availability_id": "s000002", "objecttype": "Availability" } ] } ``` > **Important** > > When the payload exceeds 256 MB, it must be chunked into 256 MB parts. > See [Uploading large catalog data files](#uploading-large-catalog-data-files) for instructions. > **Leading practice** > > * Make sure that your [catalog data](https://docs.coveo.com/en/obcf0333/) contains information to fill the [commerce standard fields](https://docs.coveo.com/en/n73f0502#standard-commerce-fields). > > * To validate that the parsing of the file is successful, test a subset of your [catalog data](https://docs.coveo.com/en/obcf0333/) before uploading all of it. ### Step 3: Close the stream Once you uploaded all your [catalog data](https://docs.coveo.com/en/obcf0333/), you must close the stream. To achieve this, you must perform the following POST request: ```http POST https://api.cloud.coveo.com/push/v1/organizations/{organizationId}/sources/{sourceId}/stream/{streamId}/close HTTP/1.1 Authorization: Bearer ``` Where you replace: * `{organizationId}` with the ID of your organization (see [Find your organization ID](https://docs.coveo.com/en/n1ce5273/)). * `{sourceId}` with the [unique identifier of the source](https://docs.coveo.com/en/3390#copy-a-source-name-or-id) to which you want to push content. * `{streamId}` with the ID of your stream (see [step 1](#step-1-open-a-stream)). * `` with an access token, such as an [API key](https://docs.coveo.com/en/1718/) that has the [required privileges](https://docs.coveo.com/en/n8of0593#required-privileges) to push content to the source. If the request to close your items is successful, you'll get the HTTP response code **200 Created**. The response body contains an [`orderingId`](https://docs.coveo.com/en/147/) that indicates the time your request was received, as well as the `requestId` which is the unique identifier for your request. **Example** **200 Created** ```json { "orderingId": 1716387965000, "requestId": "498ef728-1dc2-4b01-be5f-e8f8f1154a99" } ``` Given that your request is successful, when you upload [catalog data](https://docs.coveo.com/en/obcf0333/) into a source, it will completely replace the previous content of the source. Expect a 15-minute delay for the removal of the old items from the index. After you've uploaded all your items, check the [**Log Browser**](https://platform.cloud.coveo.com/admin/#/orgid/logs/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/logs/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/logs/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/logs/browser/)) to ensure that the streaming of products has been successful. For more information see [Use the Log Browser to review indexing logs](https://docs.coveo.com/en/1864/). ## Stream API limits The Stream API enforces certain limits on request size and frequency. These limits differ depending on whether the [organization](https://docs.coveo.com/en/185/) to which data is pushed is a production or [non-production](https://docs.coveo.com/en/2959/) [organization](https://docs.coveo.com/en/185/). The following table indicates the Stream API limits depending on your [organization](https://docs.coveo.com/en/185/) type: [%header,cols="3"] |=== |API limits |Production |Non-production |Stream API calls per day |15,000 |10,000 |Burst limit (Stream API calls per 5 minutes) |250 |150 |Stream API loads per day |96 |96 |Maximum file size |256 MB |256 MB |Maximum [item](https://docs.coveo.com/en/210/) size |3 MB |3 MB |Total size of indexed items per day[.footnote]^[[1](#limit-note1)]^ |50 GB |50 GB |[Standard commerce field](https://docs.coveo.com/en/n73f0502#standard-commerce-fields) data allowance per item |1 MB |1 MB |Maximum items per [source](https://docs.coveo.com/en/246/) |2,000,000 |2,000,000 |Partial updates per day |12,000,000 |12,000,000 |Partial updates per 5 minutes[.footnote]^[[1](#limit-note1)]^ |1,000,000 |1,000,000 |Full item updates per 5 minutes[.footnote]^[[1](#limit-note1)]^ |600,000 |600,000 |=== -- 1. This limit will be applied starting January 31, 2026. -- > **Important** > > These limits could change at any time without prior notice. > You can review your Push API usage and limits on the [**System Performance**](https://platform.cloud.coveo.com/admin/#/orgid/organization/system-performance/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/organization/system-performance/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/organization/system-performance/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/organization/system-performance/)) page of the [Coveo Administration Console](https://docs.coveo.com/en/183/). > > To modify these limits, contact your Coveo representative. ## Stream API error codes If a request to the Stream API fails because one of the limits has been exceeded, the API will trigger one of the following response status codes: [%header,cols="1,5"] |=== |Status code |Triggered when |[`413`](https://docs.coveo.com/en/95#413-request-entity-too-large) |The total Stream API request size exceeds 256 MB when pushing a large file container. See [Uploading large catalog files](#uploading-large-catalog-data-files). .4+|[`429`](https://docs.coveo.com/en/95#429) |The amount of total Stream API (upload and update) requests exceeds 15,000 per day (10,000 for [non-production organizations](https://docs.coveo.com/en/2959/)). The quota is reset at midnight [UTC](https://en.wikipedia.org/wiki/Coordinated_Universal_Time). |The amount of total Stream API upload requests exceeds 96 per day (4 per hour). The quota is reset at midnight [UTC](https://en.wikipedia.org/wiki/Coordinated_Universal_Time). |The amount of total Stream API requests exceeds 250 (150 for [non-production organizations](https://docs.coveo.com/en/2959/)) within a 5 minute period. The [`retry-after`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After) header indicates how long the user agent should wait before making another request. |[Coveo declined your request due to a reduced indexing capacity](https://docs.coveo.com/en/n91c0225/). |=== You can review your Push API usage and limits on the [**System Performance**](https://platform.cloud.coveo.com/admin/#/orgid/organization/system-performance/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/organization/system-performance/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/organization/system-performance/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/organization/system-performance/)) page of the [Coveo Administration Console](https://docs.coveo.com/en/183/). :leveloffset!: ## Uploading large catalog data files The Stream API limits the size of your [catalog data](https://docs.coveo.com/en/obcf0333/) JSON file to 256 MB. If your catalog data file exceeds the limit, you must upload multiple JSON files. To upload multiple JSON files: When you initially open the stream, you receive an `uploadUri`. This URI is used to upload your first set of metadata (JSON file). . After uploading the first file, make a POST request to the following endpoint to get a new `uploadUri`: ```http POST https://api.cloud.coveo.com/push/v1/organizations/{organizationId}/sources/{sourceId}/stream/{streamId}/chunk HTTP/1.1 Content-Type: application/json Accept: application/json Authorization: Bearer ``` This request returns a new `uploadUri` that you can use for the next step. . Make a PUT request using the `uploadUri` your received in the previous step. The body of the request must contain the [catalog data](https://docs.coveo.com/en/obcf0333/) chunk (maximum 256 MB) that you want to upload. ```http PUT {uploadUri} HTTP/1.1 x-amz-server-side-encryption: AES256 Content-Type: application/octet-stream ``` If your request to upload the catalog data is successful, you'll receive a `200` HTTP response code. . If you have more [catalog data](https://docs.coveo.com/en/obcf0333/) files to upload, repeat this process until all of your [catalog data](https://docs.coveo.com/en/obcf0333/) has been uploaded. For each file, first obtain a new `uploadUri`, then upload the file. ## Troubleshooting ## `NO_VALID_OPERATION_IN_BATCH` error When using the [Coveo Stream API](https://docs.coveo.com/en/12#tag/Stream) to push content or perform updates, you may encounter the `NO_VALID_OPERATION_IN_BATCH` error in the [**Log Browser**](https://platform.cloud.coveo.com/admin/#/orgid/logs/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/logs/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/logs/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/logs/browser/)). This error indicates that the payload sent to the file container doesn't contain any valid operations. ### Common causes This error typically occurs when the request payload contains empty `addOrUpdate` or `delete` arrays. For example: ```json { "addOrUpdate": [] } ``` or ```json { "delete": [] } ``` The Stream API expects at least one valid operation in the payload. If the arrays are empty, the request will be rejected with this error. ### Solution **Validate your payload**: Ensure that your request includes at least one valid operation (for example, `addOrUpdate` or `delete`) with valid content before sending your request to the Stream API. :leveloffset!: ## Required privileges The following table indicates the [privileges](https://docs.coveo.com/en/228/) required for your [organization](https://docs.coveo.com/en/185/)’s [groups](https://docs.coveo.com/en/2867/) to view or edit elements of the [**Catalogs**](https://platform.cloud.coveo.com/admin/#/orgid/commerce/catalogs/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/commerce/catalogs/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/commerce/catalogs/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/commerce/catalogs/)) page and its associated panels (see [Manage privileges](https://docs.coveo.com/en/3151/) and [Privilege reference](https://docs.coveo.com/en/1707/)). The **Commerce** domain is only available to [organizations](https://docs.coveo.com/en/185/) in which Coveo for Commerce features are enabled. [cols="3",options="header"] |=== |Action |Service - Domain |Required access level |View catalogs |Commerce - Catalogs Content - Sources Content - Fields Organization - Organization |View .3+.^|Edit catalogs |Content - Fields Content - Sources Organization - Organization |View |Commerce - Catalogs |Edit |Search - Execute Query |Allowed |===