How to Stream Your Catalog Data to Your Source
How to Stream Your Catalog Data to Your Source
To send your catalog data to your catalog source, you must use the Stream API.
|
This process acts as a rebuild, therefore any currently indexed items not contained in the payload will be deleted. |
This process consists of three steps:
When your catalog requires an update to a subset of products, see How to Update Your Catalog.
|
Note
Stream API operations are also available in a C# Platform SDK. |
Required Parameters
Refer to the Push API (V1) reference for a comprehensive list of required parameters.
Step 1: Open a Stream
POST https://api.cloud.coveo.com/push/v1/organizations/{organizationId}/sources/{sourceId}/stream/open
Content-Type: application/json
Accept: application/json
Authorization: Bearer <MY_ACCESS_TOKEN>
You will get a response like this one:
{
"streamId": "1234-5678-9101-1121",
"uploadUri": "link:https://coveo-nprod-customerdata.s3.amazonaws.com/[...]",
"fileId": "b5e8767e-8f0d-4a89-9095-1127915c89c7",
"requiredHeaders": {
"x-amz-server-side-encryption": "AES256",
"Content-Type": "application/octet-stream"
}
}
Step 2: Upload Your Catalog Data Into the Stream
Using the uploadUri
you received, make the following call:
PUT {uploadUri}
x-amz-server-side-encryption: AES256
Content-Type: application/octet-stream
You can now upload your catalog data (JSON file). The following is an example of content payload that contains products, variants, and availabilities in the body of the request:
{
"AddOrUpdate": [
{
"DocumentId": "product://001-red",
"FileExtension": ".html",
"ec_name": "Coveo Soccer Shoes - Red",
"model": "Authentic",
"ec_brand": ["Coveo"],
"ec_description": "<p>The astonishing, the original, and always relevant Coveo style.</p>",
"color": ["Red"],
"ec_item_group_id": "001",
"productid": "001-red",
"ec_images": ["https://myimagegallery?productid"],
"gender": "Men",
"ec_price": 28.00,
"ec_category": "Soccer Shoes",
"objecttype": "Product"
},
{
"DocumentId": "variant://001-red-8_wide",
"FileExtension": ".html",
"ec_name": "Coveo Soccer Shoes - Red / Size 8 - Wide",
"sku": "001-red-8_wide",
"productsize": "8",
"width": "wide",
"productid": "001-red",
"objecttype": "Variant"
},
{
"DocumentId": "store://s000002",
"title": "Montreal Store",
"lat": 45.4975,
"long": 73.5687,
"availableskus": ["001-red-8_wide","001-red-9_wide","001-red-10_wide","001-red-11_wide", "001-blue-8_wide"],
"availabilityid": "s000002",
"objecttype": "Availability"
},
// ...
]
}
|
Notes
|
|
WARNING
Currently indexed items not contained in the payload will be automatically deleted. To prevent users from inadvertently deleting a large amount of documents in a catalog source, the delete operation is skipped during the stream (rebuild) process if more than 75% of existing items were to be deleted. To delete indexed items, users should instead perform a Full Document Update. |
Catalog Payload Exceeds 256 MB
When a single catalog metadata payload exceeds 256 MB, you will need to send the additional payload in multiple document uploads or chunks.
|
To validate the parsing of the file is successful, we recommend that you test a subset of your catalog data before uploading the entire catalog. |
When you initially open the stream, you get an uploadUri
and a streamId
.
After you have sent the first payload, you need to send the next content payload in the body of the request:
POST https://api.cloud.coveo.com/push/v1/organizations/{organizationId}/sources/{sourceId}/stream/{streamId}/chunk
Content-Type: application/json
Accept: application/json
Authorization: Bearer <MY_ACCESS_TOKEN>
Payload
{}
You then receive a second uploadUri
and you must repeat the process until you have uploaded all metadata to your catalog.
Step 3: Close the Stream
POST https://api.cloud.coveo.com/push/v1/organizations/{organizationId}/sources/{sourceId}/stream/{streamId}/close
Authorization: Bearer <MY_ACCESS_TOKEN>
When you upload a catalog into a source, it will replace the previous content of the source completely. Expect a 15-minute delay for the removal of the old items from the index.
What’s Next?
Once your initial catalog data upload is complete, you can make updates to the catalog content by performing a full document update or by making smaller adjustments to information on single products with a partial document update.