How to Stream Your Catalog Data to Your Source

Ecommerce Developer Product Documentation

To send your catalog data to your catalog source, you must use the Stream API.

Important

This process acts as a rebuild, therefore any currently indexed items not contained in the payload will be deleted.

This process consists of three steps:

  1. Open a stream

  2. Upload your catalog data into the stream

  3. Close the stream

When your catalog requires an update to a subset of products, see How to Update Your Catalog.

Note

Stream API operations are also available in a C# Platform SDK.

Required Parameters

Refer to the Push API (V1) reference for a comprehensive list of required parameters.

Step 1: Open a Stream

POST https://api.cloud.coveo.com/push/v1/organizations/{organizationId}/sources/{sourceId}/stream/open

Content-Type: application/json
Accept: application/json
Authorization: Bearer <MY_ACCESS_TOKEN>

You will get a response like this one:

{
    "streamId": "1234-5678-9101-1121",
    "uploadUri": "link:https://coveo-nprod-customerdata.s3.amazonaws.com/[...]",
    "fileId": "b5e8767e-8f0d-4a89-9095-1127915c89c7",
    "requiredHeaders": {
      "x-amz-server-side-encryption": "AES256",
      "Content-Type": "application/octet-stream"
  }
}

Step 2: Upload Your Catalog Data Into the Stream

Using the uploadUri you received, make the following call:

PUT {uploadUri}

x-amz-server-side-encryption: AES256
Content-Type: application/octet-stream

You can now upload your catalog data (JSON file). The following is an example of content payload that contains products, variants, and availabilities in the body of the request:

{
    "AddOrUpdate": [
    {
     "DocumentId": "product://001-red",
     "FileExtension": ".html",
     "ec_name": "Coveo Soccer Shoes - Red",
     "model": "Authentic",
     "ec_brand": ["Coveo"],
     "ec_description": "<p>The astonishing, the original, and always relevant Coveo style.</p>",
     "color": ["Red"],
     "ec_item_group_id": "001",
     "productid": "001-red",
     "ec_images": ["https://myimagegallery?productid"],
     "gender": "Men",
     "ec_price": 28.00,
     "ec_category": "Soccer Shoes",
     "objecttype": "Product"
   },
   {
     "DocumentId": "variant://001-red-8_wide",
     "FileExtension": ".html",
     "ec_name": "Coveo Soccer Shoes - Red / Size 8 - Wide",
     "sku": "001-red-8_wide",
     "productsize": "8",
     "width": "wide",
     "productid": "001-red",
     "objecttype": "Variant"
   },
   {
      "DocumentId": "store://s000002",
      "title": "Montreal Store",
      "lat": 45.4975,
      "long": -73.5687,
      "availableskus": ["001-red-8_wide","001-red-9_wide","001-red-10_wide","001-red-11_wide", "001-blue-8_wide"],
      "availabilityid": "s000002",
      "objecttype": "Availability"
    },
   // ...
    ]
}
Notes
Warning
WARNING

When your catalog source is used in a catalog configuration, currently indexed items not contained in the payload will be automatically deleted. To prevent users from inadvertently deleting a large amount of items in a catalog source, the delete operation is skipped during the stream (rebuild) process if more than 75% of existing items were to be deleted. To delete indexed items, users should instead perform a Full Document Update.

When your catalog source isn’t used with a catalog configuration, and you open and close a stream with no content payload, all content will be deleted from your source.

Catalog Payload Exceeds 256 MB

When a single catalog metadata payload exceeds 256 MB, you will need to send the additional payload in multiple document uploads or chunks.

Tip
Leading practice

To validate the parsing of the file is successful, we recommend that you test a subset of your catalog data before uploading the entire catalog.

When you initially open the stream, you get an uploadUri and a streamId. After you have sent the first payload, you need to send the next content payload in the body of the request:

POST https://api.cloud.coveo.com/push/v1/organizations/{organizationId}/sources/{sourceId}/stream/{streamId}/chunk

Content-Type: application/json
Accept: application/json
Authorization: Bearer <MY_ACCESS_TOKEN>

Payload

{}

You then receive a second uploadUri and you must repeat the process until you have uploaded all metadata to your catalog.

Step 3: Close the Stream

POST https://api.cloud.coveo.com/push/v1/organizations/{organizationId}/sources/{sourceId}/stream/{streamId}/close

Authorization: Bearer <MY_ACCESS_TOKEN>

When you upload a catalog into a source, it will replace the previous content of the source completely. Expect a 15-minute delay for the removal of the old items from the index.

Stream API Limits

The Stream API enforces certain limits on request size and frequency.

Important

These limits could change at any time without prior notice. Contact us to modify these limits.

Stream API Error codes

  • 413: If the amount of total Stream API calls exceed 50K per day.

  • 429: If your total Stream API request size exceeds 256 MB when pushing a large file container. See Catalog Payload Exceeds 256 MB

What’s Next?

Once your initial catalog data upload is complete, you can make updates to the catalog content by performing a full document update or by making smaller adjustments to information on single products with a partial document update.