Uploading a Data Stream for an Extension Test
Uploading a Data Stream for an Extension Test
When using the Extension API /test endpoint to test an indexing pipeline extension (IPE), you can pass data streams in the mock documents of your request bodies.
If those data streams are very large, it can be impractical to pass them as inline content.
This is why the Extension API exposes the /test/datastream endpoint, which allows you to retrieve URIs where you can upload data streams.
You can then refer to those uploaded data streams by their id in the mock document of your /test request body.
This article provides more detailed explanations of those steps.
Step 1 (Optional): Request a Data Stream
The Search API exposes endpoints to retrieve the thumbnails, text bodies, and HTML bodies of indexed items[1]. Be aware that those data streams may not match exactly the ones which would actually be available when executing an IPE. Nonetheless, they’re typically similar enough to carry out meaningful tests. You can alternatively craft your own data streams.
Authenticate the request using an access token granting the Search - Execute queries - Allowed privilege (see Manage API Keys).
Retrieve and store the body of a successful response.
Using the Search API to retrieve the HTML body of a specific YouTube item:
GET https://platform.cloud.coveo.com/rest/search/v2/html?organizationId=myorganization&uniqueId=42.1294$https://youtube.com/User:TheLumineers/Channel:UCB7P9Hr5BYB5Mkxau6t3Sgw/Video:zND5Qc7LzdM HTTP/1.1
Authorization: Bearer **********-****-****-****-************200 OK:
<meta http-equiv='Content-Type' content='text/html; charset=UTF-16'>
<base target="_blank" href="https://youtube.com/User:TheLumineers/Channel:UCB7P9Hr5BYB5Mkxau6t3Sgw/Video:zND5Qc7LzdM">
<html>
<body>
<div style="margin: 20px auto; width: 640px">
<div>
<!--Embedded Youtube video player-->
<iframe type='text/html' src='https://www.youtube.com/embed/zND5Qc7LzdM' width='640' height='360'
frameborder='0' allowfullscreen='true'></iframe>
</div>
<div style="border: 1px solid #CCC; padding:10px; font-family: Helvetica">
<p style="text-align: justify;">The Lumineers performing "Stubborn Love" at the Tractor Tavern in Seattle -
9.27.2011<br /><br />Video taken by Gerad Simon.<br /><br />http://www.thelumineers.com/<br />http://www.facebook.com/TheLumineers<br />http://www.myspace.com/thelumineers<br /><br />June 04 -- Cincinnati, OH @ Riverbend Music Center*<br />June 06 -- Clarkston, MI @ DTE Energy Music Theatre*%<br />June 07 -- Canandaugia, NY @ CMAC*%<br />June 08 -- Mountain Hunter, NY @ Mountain Jam Festival<br />June 09 -- New York, NY @ Governor's Ball<br />June 12 -- Portsmouth, VA @ nTelos Wireless Pavilion*%<br />June 13 -- Cary, NC @ Koka Booth Amphitheater*% (Presale: April 10; On-sale April 12)<br />June 14 -- Charleston, SC @ Family Circle Pavilion* (Presale: April 10; On-sale April 12)<br />June 15 -- Manchester, TN @ Bonnaroo Music Festival<br />June 23 -- Dover, DE @ Firefly Music Festival<br />June 28 -- Worthy Farm, Pilton @ Glastonbury<br />June 29 -- Paris @ L'Olympia<br />June 30 -- St. Gallen @ Open Air St. Gallen<br />July 05 -- Werchter @ Rock Werchter 2013<br />July 08 -- Montreux @ Montreux Jazz Festival<br />July 09 -- Milan @ Un Altro Music Festival<br />July 10 -- Barcelona @ Razzmatazz<br />July 13 -- Balado, Kinnross-Shire, Scotland @ T in the Park<br />July 25 -- Floyd, VA @ FloydFest
<br />July 26 -- Columbia, MD @ Merriweather Post Pavilion<br />July 27 -- Camden, NJ @ XPoNential Music Festival<br />July 28 -- Newport, RI @ Newport Folk Festival<br />July 31 -- Toronto, ON @ Edgefest '13<br />Aug 02 -- Chicago, IL @ Lollapalooza<br />Aug 04 -- Montreal, QC @ Osheaga Music & Arts Festival<br />Aug 16 -- Hamburg @ Dockville Festival<br />Aug 17 -- Biddinghuizen @ Lowlands<br />Aug 18 -- Leipzig @ Highfield Festival<br />Aug 19 -- Hanau @ Amphitheater Hanau
<br />Aug 22 -- Belfast @ Belsonic 2013 // Custom House Square<br />Aug 23 -- Leeds @ Reading Festival<br />Sept 14 -- Colorado @ Red Rocks Amphitheatre<br />Sept 15 -- Colorado @ Red Rocks Amphitheatre<br />Nov 17 -- Paris @ Zenith<br />Nov 18 -- Amsterdam @ Heineken Music Hall<br />Nov 21 -- Dublin @ 02 Arena<br />Dec 01 -- Brussels @ Cirque Royal<br />Dec 02 -- Winterthur @Eishalle Deutweg<br />Dec 03 -- Bolgna @ Estragon<br />Dec 06 -- Munich @ Zenith<br />Dec 07 -- Berlin @ Columbiahalle<br />Dec 08 -- Cologne @ Palladium
<br />Dec 10 -- Copenhagen @ Vega<br />Dec 11 -- Oslo @ Sentrum Scene<br />Dec 12 -- Stockholm @ Annexet<br /><br />*with Cold War Kids<br />%with J Roddy Walston</p>
</div>
</div>
</body>
</html>Step 2: Request an Upload URI
Use the POST /rest/organizations/{organizationId}/extensions/{extensionId}/test/datastream operation to request an upload URI, specifying your organizationId and extensionId path parameters.
Authenticate the request using an access token granting the Content - Extension - Edit privilege on the target extension (see Manage API Keys).
In the body of a successful response, retrieve and store the streamId, the uri, and the requiredHeaders properties.
POST https://platform.cloud.coveo.com/rest/organizations/myorganization/extensions/myorganization-vwlyqfbtjkotxqkmrxqjcbcpoy/test/datastream HTTP/1.1
Authorization: Bearer **********-****-****-****-************200 OK
{
"streamId": "73e67b38-c307-429e-ac89-147083f31ca2",
"uri": "link:https://coveo-nprod-customerdata.s3.amazonaws.com/proda/blobstore/myorganization/73e67b38-c307-429e-ac89-147083f31ca2?X-Amz-Security-Token[...]",
"requiredHeaders": {
"x-amz-server-side-encryption": "AES256",
"Content-Type": "application/octet-stream"
}
}Step 3: Upload a Data Stream
Make a PUT request to the uri obtained in step 1, using the requiredHeaders also obtained in step 1, and where the body of your call is your data stream, in raw format.
If you requested a data stream from the Search API in Step 1, this is where you use it.
|
Note
If you want to use your uploaded data stream as the |
PUT link:https://coveo-nprod-customerdata.s3.amazonaws.com/proda/blobstore/myorganization/73e67b38-c307-429e-ac89-147083f31ca2?X-Amz-Security-Token[...] HTTP/1.1
x-amz-server-side-encryption: AES256
Content-Type: application/octet-streamRequest body:
<meta http-equiv='Content-Type' content='text/html; charset=UTF-16'>
<base target="_blank" href="https://youtube.com/User:TheLumineers/Channel:UCB7P9Hr5BYB5Mkxau6t3Sgw/Video:zND5Qc7LzdM">
<html>
<body>
<div style="margin: 20px auto; width: 640px">
<div>
<!--Embedded Youtube video player-->
<iframe type='text/html' src='https://www.youtube.com/embed/zND5Qc7LzdM' width='640' height='360'
frameborder='0' allowfullscreen='true'></iframe>
</div>
<div style="border: 1px solid #CCC; padding:10px; font-family: Helvetica">
<p style="text-align: justify;">The Lumineers performing "Stubborn Love" at the Tractor Tavern in Seattle -
9.27.2011<br /><br />Video taken by Gerad Simon.<br /><br />http://www.thelumineers.com/<br />http://www.facebook.com/TheLumineers<br />http://www.myspace.com/thelumineers<br /><br />June 04 -- Cincinnati, OH @ Riverbend Music Center*<br />June 06 -- Clarkston, MI @ DTE Energy Music Theatre*%<br />June 07 -- Canandaugia, NY @ CMAC*%<br />June 08 -- Mountain Hunter, NY @ Mountain Jam Festival<br />June 09 -- New York, NY @ Governor's Ball<br />June 12 -- Portsmouth, VA @ nTelos Wireless Pavilion*%<br />June 13 -- Cary, NC @ Koka Booth Amphitheater*% (Presale: April 10; On-sale April 12)<br />June 14 -- Charleston, SC @ Family Circle Pavilion* (Presale: April 10; On-sale April 12)<br />June 15 -- Manchester, TN @ Bonnaroo Music Festival<br />June 23 -- Dover, DE @ Firefly Music Festival<br />June 28 -- Worthy Farm, Pilton @ Glastonbury<br />June 29 -- Paris @ L'Olympia<br />June 30 -- St. Gallen @ Open Air St. Gallen<br />July 05 -- Werchter @ Rock Werchter 2013<br />July 08 -- Montreux @ Montreux Jazz Festival<br />July 09 -- Milan @ Un Altro Music Festival<br />July 10 -- Barcelona @ Razzmatazz<br />July 13 -- Balado, Kinnross-Shire, Scotland @ T in the Park<br />July 25 -- Floyd, VA @ FloydFest
<br />July 26 -- Columbia, MD @ Merriweather Post Pavilion<br />July 27 -- Camden, NJ @ XPoNential Music Festival<br />July 28 -- Newport, RI @ Newport Folk Festival<br />July 31 -- Toronto, ON @ Edgefest '13<br />Aug 02 -- Chicago, IL @ Lollapalooza<br />Aug 04 -- Montreal, QC @ Osheaga Music & Arts Festival<br />Aug 16 -- Hamburg @ Dockville Festival<br />Aug 17 -- Biddinghuizen @ Lowlands<br />Aug 18 -- Leipzig @ Highfield Festival<br />Aug 19 -- Hanau @ Amphitheater Hanau
<br />Aug 22 -- Belfast @ Belsonic 2013 // Custom House Square<br />Aug 23 -- Leeds @ Reading Festival<br />Sept 14 -- Colorado @ Red Rocks Amphitheatre<br />Sept 15 -- Colorado @ Red Rocks Amphitheatre<br />Nov 17 -- Paris @ Zenith<br />Nov 18 -- Amsterdam @ Heineken Music Hall<br />Nov 21 -- Dublin @ 02 Arena<br />Dec 01 -- Brussels @ Cirque Royal<br />Dec 02 -- Winterthur @Eishalle Deutweg<br />Dec 03 -- Bolgna @ Estragon<br />Dec 06 -- Munich @ Zenith<br />Dec 07 -- Berlin @ Columbiahalle<br />Dec 08 -- Cologne @ Palladium
<br />Dec 10 -- Copenhagen @ Vega<br />Dec 11 -- Oslo @ Sentrum Scene<br />Dec 12 -- Stockholm @ Annexet<br /><br />*with Cold War Kids<br />%with J Roddy Walston</p>
</div>
</div>
</body>
</html>Step 4: Use the Data Stream in an Extension Test
Use the POST /rest/organizations/{organizationId}/extensions/{extensionId}/test operation to test your extension, specifying your organizationId and extensionId path parameters.
In the request body, set the id of the target data stream to the streamId value retrieved in step 2.
The possible data stream values are DOCUMENT_DATA, BODY_HTML, BODY_TEXT, and THUMBNAIL.
Authenticate the request using an access token granting the Content - Extension - Edit privilege on the target extension (see Manage API Keys).
You have a simple extension which logs the HTML body of documents.
myDataStream = document.get_data_stream('body_html').read()
log(myDataStream)You use the data stream uploaded in step 3 to test your extension.
POST https://platform.cloud.coveo.com/rest/organizations/myorganization/extensions/myorganization-vwlyqfbtjkotxqkmrxqjcbcpoy/test HTTP/1.1
Authorization: Bearer **********-****-****-****-************Request body:
{
"document":{
"dataStreams":[
{
"values":{
"BODY_HTML":{
"id":"73e67b38-c307-429e-ac89-147083f31ca2"
}
}
}
]
}
}200 OK response body (excerpt):
{
...
"logs": [
{
"severity": "NORMAL",
"comment": "<meta http-equiv='Content-Type' content='text/html; charset=UTF-16'>\n<base target=\"_blank\" href=\"https://youtube.com/User:TheLumineers/Channel:UCB7P9Hr5BYB5Mkxau6t3Sgw/Video:zND5Qc7LzdM\">\n<html>\n\n<body>\n\t<div style=\"margin: 20px auto; width: 640px\">\n\t\t<div>\n\t\t\t<!--Embedded Youtube video player-->\n\t\t\t<iframe type='text/html' src='https://www.youtube.com/embed/zND5Qc7LzdM' width='640' height='360'\n\t\t\t\tframeborder='0' allowfullscreen='true'></iframe>\n\t\t</div>\n\t\t<div style=\"border: 1px solid #CCC; padding:10px; font-family: Helvetica\">\n\t\t\t<p style=\"text-align: justify;\">The Lumineers performing \"Stubborn Love\" at the Tractor Tavern in Seattle -\n\t\t\t\t9.27.2011<br /><br />Video taken by Gerad Simon.<br /><br />http://www.thelumineers.com/<br />http://www.facebook.com/TheLumineers<br />http://www.myspace.com/thelumineers<br /><br />June 04 -- Cincinnati, OH @ Riverbend Music Center*<br />June 06 -- Clarkston, MI @ DTE Energy Music Theatre*%<br />June 07 -- Canandaugia, NY @ CMAC*%<br />June 08 -- Mountain Hunter, NY @ Mountain Jam Festival<br />June 09 -- New York, NY @ Governor's Ball<br />June 12 -- Portsmouth, VA @ nTelos Wireless Pavilion*%<br />June 13 -- Cary, NC @ Koka Booth Amphitheater*% (Presale: April 10; On-sale April 12)<br />June 14 -- Charleston, SC @ Family Circle Pavilion* (Presale: April 10; On-sale April 12)<br />June 15 -- Manchester, TN @ Bonnaroo Music Festival<br />June 23 -- Dover, DE @ Firefly Music Festival<br />June 28 -- Worthy Farm, Pilton @ Glastonbury<br />June 29 -- Paris @ L'Olympia<br />June 30 -- St. Gallen @ Open Air St. Gallen<br />July 05 -- Werchter @ Rock Werchter 2013<br />July 08 -- Montreux @ Montreux Jazz Festival<br />July 09 -- Milan @ Un Altro Music Festival<br />July 10 -- Barcelona @ Razzmatazz<br />July 13 -- Balado, Kinnross-Shire, Scotland @ T in the Park<br />July 25 -- Floyd, VA @ FloydFest\n\t\t\t\t<br />July 26 -- Columbia, MD @ Merriweather Post Pavilion<br />July 27 -- Camden, NJ @ XPoNential Music Festival<br />July 28 -- Newport, RI @ Newport Folk Festival<br />July 31 -- Toronto, ON @ Edgefest '13<br />Aug 02 -- Chicago, IL @ Lollapalooza<br />Aug 04 -- Montreal, QC @ Osheaga Music & Arts Festival<br />Aug 16 -- Hamburg @ Dockville Festival<br />Aug 17 -- Biddinghuizen @ Lowlands<br />Aug 18 -- Leipzig @ Highfield Festival<br />Aug 19 -- Hanau @ Amphitheater Hanau\n\t\t\t\t<br />Aug 22 -- Belfast @ Belsonic 2013 // Custom House Square<br />Aug 23 -- Leeds @ Reading Festival<br />Sept 14 -- Colorado @ Red Rocks Amphitheatre<br />Sept 15 -- Colorado @ Red Rocks Amphitheatre<br />Nov 17 -- Paris @ Zenith<br />Nov 18 -- Amsterdam @ Heineken Music Hall<br />Nov 21 -- Dublin @ 02 Arena<br />Dec 01 -- Brussels @ Cirque Royal<br />Dec 02 -- Winterthur @Eishalle Deutweg<br />Dec 03 -- Bolgna @ Estragon<br />Dec 06 -- Munich @ Zenith<br />Dec 07 -- Berlin @ Columbiahalle<br />Dec 08 -- Cologne @ Palladium\n\t\t\t\t<br />Dec 10 -- Copenhagen @ Vega<br />Dec 11 -- Oslo @ Sentrum Scene<br />Dec 12 -- Stockholm @ Annexet<br /><br />*with Cold War Kids<br />%with J Roddy Walston</p>\n\t\t</div>\n\t</div>\n</body>\n\n</html>"
}
],
...
"duration": 0.103328537,
"result": {
"status": "SUCCESS"
}
}In the above response body, you can see that the comment property is indeed the content of your uploaded data stream, confirming your extension works as expected.
documentdata stream, you can use the original item itself.
Very useful
Not really
Very useful
Not really