Build the Products catalog

This is for:

Developer

In this article

Product coverage
Locales
Limitations
QProtocol setup
Google product feed setup
Catalog field mapping
Next steps

In this article, we’ll discuss the product catalog that feeds product recommendations, how it’s built, and what you can do to get greater product coverage.

You can use the Products catalog for product recommendations in Coveo Merchandising Hub (CMH). See Product recommendations in CMH.

Product coverage

In our experience, few clients have obtained complete product coverage by only collecting QProtocol events. For this reason, we recommend providing an alternative source of product data such as a Google product feed.

For an alternative source to function correctly, the product IDs used must match the IDs that we collect from the ecProduct event.

The product catalog from QProtocol is produced from events collected over the last 14 days (rolling window). Products that don’t appear in the last fourteen days are automatically removed. For alternative sources, products disappear from the catalog immediately after they have been removed from the source. We refresh any source every 4 hours.

Where the available products are populated by an alternative source, data from QProtocol is automatically mixed with data from the alternative source. For example, if name is not provided in the alternative source but is provided within QProtocol, that field will be available in the catalog. However, it will only be available for products that appeared within the last 14 days and hence has to be treated as optional.

Note

Marking a product availability: out of stock inside a Google product feed will set its stock to 0, even if a different stock is provided in QProtocol.

When an alternative source URL is configured, the target server has to respond within 60s (time to first byte) and complete a read within 60s (time to read after first byte). We support the following protocols: HTTP, HTTPS, SFTP, FTP, and FTPS. Simple authentication via username and password can be provided.

Locales

A product catalog can exist in multiple languages and currencies. We build a product catalog for every combination of ecView.language and ecProduct.product.price.currency, sent to us via QProtocol.

Refer to the following restrictions:

Language must be an IETF tag, for example, en (ISO 639) or en-us (ISO 639 - ISO 3166-1)
Currency must be an ISO 4217 currency code (for example, GBP, USD)
The locale must be in the format <language>-<currency> (for example, en-GB-GBP)

When QProtocol is the only source of product catalog data, we only consider additional locales that have a product catalog coverage of 25% in comparison to the largest locale on the site.

For example, if a website with a locale “en-US-USD” has 2000 products, we would need to see 500 products over the past two weeks before the additional locale is available.

Note

If a locale has not reached sufficient product coverage, it will not be available for selection when adding a new product feed.

Note

Coverage can change over time based on your user’s behavior. We therefore recommend providing an alternative source of product information. You can provide up to 10 different sources for locales.

Once a locale is available, it can be utilized in a call to the Recommendations API. When using the Qubit recommendation package, this is automatically done by introspecting the last ecView event.

https://recs.qubit.com/vc/recommend/2.1/TID?strategy=XXX&locale=en-USD

For sites without multiple languages and currencies, or where a client does not wish to show recommendations to other locales, it’s not necessary to provide a locale. The default behavior when no locale is provided is to return the most common language/currency combination.

Limitations

A product within the catalog needs to conform to certain rules:

A product can’t be part of more than 10 category hierarchies
A product can’t be part of more than 25 categories irrespective of the hierarchy (for example, [“Sale > Shoes”, “Fall Promotion”] is 3 categories)
A product image URL can’t exceed 4000 characters
A product ID can’t exceed 60 characters
The product data overall can’t exceed 5KB - this means you can’t embed images

QProtocol setup

In this setup, Recommendations will automatically get product information from the QProtocol events generated on your site.

Refer to the following table for details of which events and event fields are required to run recommendations in the ecommerce vertical. We also identify fields that we recommend you implement for data completeness but are optional.

ecView

Field (implementation) Description

Field (implementation)	Description
type (required)	Reports the page type (for example, home, category, search, product, basket, checkout, confirmation, help, contact, registration, content, account, or other)
subtypes (recommended)	An unordered list of subtypes to describe the view
language (recommended)	The language used to render the page in this view, which must be an IETF language tag
country (recommended)	The selected country for the view, which must be an ISO 3166-1 alpha-2 code (for example, GB, FR, US)
currency (recommended)	The ISO 4217 currency for the user (for example, `GBP`, `USD`)

type (required)

Reports the page type (for example, home, category, search, product, basket, checkout, confirmation, help, contact, registration, content, account, or other)

subtypes (recommended)

An unordered list of subtypes to describe the view

language (recommended)

The language used to render the page in this view, which must be an IETF language tag

country (recommended)

The selected country for the view, which must be an ISO 3166-1 alpha-2 code (for example, GB, FR, US)

currency (recommended)

The ISO 4217 currency for the user (for example, GBP, USD)

ecProduct

Field (implementation) Description

Field (implementation)	Description
eventType (required)	The type of product event (for example, ‘listing’, ‘detail’, or ‘linked_product’)
product.sku (required)	Unique product identifier
product.productId (required)	A unique ID to identify a product and all of its size, color, pattern, material, age, group, gender variants
product.name (required)	The product’s name, which should match the name shown on the product page
product.stock (recommended)	The number of available units in stock
product.color (recommended)	The product’s color
product.size (recommended)	The product’s size
product.price.value (required)	The price of the product after discounts, promotions, etc, rounded to 2 decimal places
product.price.currency (required)	The ISO 4217 currency code (for example, `GBP`, `USD`)
product.url (required)	The URL of the product’s page
product.description (recommended)	An accurate description of the product, which should match the description on the product page
product.categories (required)	A list of one or more product categories the product belongs to. Each category is a full category path, with each level separated by >
product.category (required)	An array of categories, highest level first, defining the product. Recommended to not exceed 4 items long
product.images (required)	An array containing the URLs of the product’s images. The main image you want to use to display the product must be the first element in the array
product.originalPrice.value (required)	The price the product was originally sold at before any promotions or discounts
product.originalPrice.currency (required)	The ISO 4217 currency code (for example, `GBP`, `USD`)
product.manufacturer (recommended)	The product’s manufacturer

eventType (required)

The type of product event (for example, ‘listing’, ‘detail’, or ‘linked_product’)

product.sku (required)

Unique product identifier

product.productId (required)

A unique ID to identify a product and all of its size, color, pattern, material, age, group, gender variants

product.name (required)

The product’s name, which should match the name shown on the product page

product.stock (recommended)

The number of available units in stock

product.color (recommended)

The product’s color

product.size (recommended)

The product’s size

product.price.value (required)

The price of the product after discounts, promotions, etc, rounded to 2 decimal places

product.price.currency (required)

The ISO 4217 currency code (for example, GBP, USD)

product.url (required)

The URL of the product’s page

product.description (recommended)

An accurate description of the product, which should match the description on the product page

product.categories (required)

A list of one or more product categories the product belongs to. Each category is a full category path, with each level separated by >

product.category (required)

An array of categories, highest level first, defining the product. Recommended to not exceed 4 items long

product.images (required)

An array containing the URLs of the product’s images. The main image you want to use to display the product must be the first element in the array

product.originalPrice.value (required)

The price the product was originally sold at before any promotions or discounts

product.originalPrice.currency (required)

The ISO 4217 currency code (for example, GBP, USD)

product.manufacturer (recommended)

The product’s manufacturer

ecBasketTransactionSummary

Field (implementation) Description

Field (implementation)	Description
transaction.id (required)	A unique transaction ID
basket.subtotalIncludingTax.value (recommended)	The basket subtotal, including tax, but before the application of discounts, promotions, shipping costs, etc, rounded to 2 decimal places
basket.subtotalIncludingTax.currency (recommended)	The ISO 4217 currency code (for example, `GBP`, `USD`)
basket.total.value (required)	The basket total after the application of discounts, promotions, shipping costs, etc, rounded to 2 decimal places
basket.total.currency (required)	The ISO 4217 currency code (for example, `GBP`, `USD`)

transaction.id (required)

A unique transaction ID

basket.subtotalIncludingTax.value (recommended)

The basket subtotal, including tax, but before the application of discounts, promotions, shipping costs, etc, rounded to 2 decimal places

basket.subtotalIncludingTax.currency (recommended)

The ISO 4217 currency code (for example, GBP, USD)

basket.total.value (required)

The basket total after the application of discounts, promotions, shipping costs, etc, rounded to 2 decimal places

basket.total.currency (required)

The ISO 4217 currency code (for example, GBP, USD)

ecBasketItemTransaction

Field (implementation) Description

Field (implementation)	Description
transaction.id (required)	A unique transaction ID
basket.subtotalIncludingTax.value (recommended)	The basket subtotal, including tax, but before the application of discounts, promotions, shipping costs, etc, rounded to 2 decimal places
basket.subtotalIncludingTax.currency (recommended)	The ISO 4217 currency code (for example, `GBP`, `USD`)
basket.total.value (recommended)	The basket total after the application of discounts, promotions, shipping costs, etc, rounded to 2 decimal places
basket.total.currency (recommended)	The ISO 4217 currency code (for example, `GBP`, `USD`)
product.sku (required)	Unique product identifier
product.productId (required)	A unique ID to identify a product and all of its size, color, pattern, material, age, group, gender variants
product.name (recommended)	The product’s name, which should match the name shown on the product page
product.stock (recommended)	The number of available units in stock
product.price.value (recommended)	The price of the product after discounts, promotions, etc, rounded to 2 decimal places
product.price.currency (recommended)	The ISO 4217 currency code (for example, `GBP`, `USD`)
product.url (recommended)	The URL of the product’s landing page
product.description (recommended)	An accurate description of the product, which should match the description on the product page
product.categories (recommended)	A list of one or more product categories the product belongs to. Each category is a full category path, with each level separated by >
product.category (recommended)	An array of categories, highest level first, defining the product. Recommended to not exceed 4 items long
product.images (recommended)	An array containing the URLs of the product’s images
quantity (recommended)	The number of products described by the line item
subtotal.value (recommended)	The subtotal for the current items taking into account the current price and quantity, rounded to 2 decimal places
subtotal.currency (recommended)	The ISO 4217 currency code (for example, `GBP`, `USD`)

transaction.id (required)

A unique transaction ID

basket.subtotalIncludingTax.value (recommended)

The basket subtotal, including tax, but before the application of discounts, promotions, shipping costs, etc, rounded to 2 decimal places

basket.subtotalIncludingTax.currency (recommended)

The ISO 4217 currency code (for example, GBP, USD)

basket.total.value (recommended)

The basket total after the application of discounts, promotions, shipping costs, etc, rounded to 2 decimal places

basket.total.currency (recommended)

The ISO 4217 currency code (for example, GBP, USD)

product.sku (required)

Unique product identifier

product.productId (required)

A unique ID to identify a product and all of its size, color, pattern, material, age, group, gender variants

product.name (recommended)

The product’s name, which should match the name shown on the product page

product.stock (recommended)

The number of available units in stock

product.price.value (recommended)

The price of the product after discounts, promotions, etc, rounded to 2 decimal places

product.price.currency (recommended)

The ISO 4217 currency code (for example, GBP, USD)

product.url (recommended)

The URL of the product’s landing page

product.description (recommended)

An accurate description of the product, which should match the description on the product page

product.categories (recommended)

A list of one or more product categories the product belongs to. Each category is a full category path, with each level separated by >

product.category (recommended)

An array of categories, highest level first, defining the product. Recommended to not exceed 4 items long

product.images (recommended)

An array containing the URLs of the product’s images

quantity (recommended)

The number of products described by the line item

subtotal.value (recommended)

The subtotal for the current items taking into account the current price and quantity, rounded to 2 decimal places

subtotal.currency (recommended)

The ISO 4217 currency code (for example, GBP, USD)

Note

For other verticals, such as eGaming, it’s necessary to emit other events.

Google product feed setup

In this setup, Recommendations uses a server-side integration with a Google product feed to get product information. You can import a Google product feed into Qubit by following the steps outlined in importing a Google Product Feed.

The following information needs to be available in the product feed:

Attribute Description Example

Attribute	Description	Example
id	Your product’s unique identifier. Use the product’s SKU where possible	A2B4
title	Your product’s name that matches the title from your landing page	Men Pique Polo Shirt
description	Your product’s description that matches the description from your landing page	Red, 100% cotton, large men’s t-shirt
link	Your product’s landing page, starting with http or https	`http://www.example.com/asp/sp.asp?cat=12&id=1030`
image_link	The URL of your product’s main image	`http://www.example.com/image1.jpg`
availability	Your product’s availability	in stock
price	Your product’s price in the format `<NUMBER> <CURRENCY>`	15.00 USD
product_type	The full path of the category or the numerical category ID	Apparel & Accessories > Clothing > Outerwear > Coats & Jackets or 371

Your product’s unique identifier. Use the product’s SKU where possible

A2B4

title

Your product’s name that matches the title from your landing page

Men Pique Polo Shirt

description

Your product’s description that matches the description from your landing page

Red, 100% cotton, large men’s t-shirt

link

Your product’s landing page, starting with http or https

http://www.example.com/asp/sp.asp?cat=12&id=1030

image_link

The URL of your product’s main image

http://www.example.com/image1.jpg

availability

Your product’s availability

in stock

price

Your product’s price in the format <NUMBER> <CURRENCY>

15.00 USD

product_type

The full path of the category or the numerical category ID

Apparel & Accessories > Clothing > Outerwear > Coats & Jackets or 371

Refer to the Products data specification for details of optional attributes and full specifications.

Refer to the following examples:

<?xml version="1.0"?>
<rss xmlns:g="http://base.google.com/ns/1.0" version="2.0">
  <channel>
    <title>Dummy Client - Online Store</title>
    <link>`+http://dummy-client.qubitproducts.com+`</link>
    <description>This is a sample feed containing the required and recommended attributes for a variety of different products</description>

    <item>
      <g:id>UNIQUE_ID_00001</g:id>
      <g:title>Two tone dress</g:title>
      <g:description>Two tone dress</g:description>
      <g:link>`+http://dummt-client.qubitproducts.com/dresses/two-tone-dress.html+`</g:link>
      <g:image_link>`+http://dummy-client.qubitproducts.com/media/catalog/product/cache/1/image/800x800/9df78eab33525d08d6e5fb8d27136e95/d/r/dress_1.jpg+`</g:image_link>
      <g:availability>in stock</g:availability>
      <g:price>159.00 USD</g:price>
      <g:product_type>Women's clothing > Dresses</g:product_type>
    </item>

    <item>
      <g:id>UNIQUE_ID_00002</g:id>
      <g:title>Two tone shirt</g:title>
      <g:description>Two tone shirt</g:description>
      <g:link>`+http://dummy-client.qubitproducts.com/shirts/two-tone-shirt.html+`</g:link>
      <g:image_link>`+http://dummy-client.qubitproducts.com/media/catalog/product/cache/1/image/800x800/9df78eab33525d08d6e5fb8d27136e95/d/r/shirt.jpg+`</g:image_link>
      <g:availability>in stock</g:availability>
      <g:price>169.00 USD</g:price>
      <g:product_type>Men's clothing > Dresses</g:product_type>
    </item>
  </channel>
</rss>

It’s possible to add custom values such as exclude_from_recommendations. It’s important to follow the guidelines in Custom attributes in XML data feeds. An updated example is shown below:

<?xml version="1.0"?>
<rss xmlns:g="http://base.google.com/ns/1.0" xmlns:c="http://base.google.com/cns/1.0" version="2.0">
  <channel>
    <title>Dummy Client - Online Store</title>
    <link>`+http://dummy-client.qubitproducts.com+`</link>
    <description>This is a sample feed containing the required and recommended attributes for a variety of different products</description>

    <item>
      <g:id>UNIQUE_ID_00001</g:id>
      <g:title>Two tone dress</g:title>
      <g:description>Two tone dress</g:description>
      <g:link>`+http://dummy-client.qubitproducts.com/dresses/two-tone-dress.html+`</g:link>
      <g:image_link>`+http://dummy-client.qubitproducts.com/media/catalog/product/cache/1/image/800x800/9df78eab33525d08d6e5fb8d27136e95/d/r/dress_1.jpg+`</g:image_link>
      <g:availability>in stock</g:availability>
      <g:price>159.00 USD</g:price>
      <g:product_type>Women's clothing > Dresses</g:product_type>
      <g:sale_price>150.00 USD</g:sale_price>
      <c:exclude_from_recommendations type="boolean">false</c:exclude_from_recommendations>
    </item>

    <item>
      <g:id>UNIQUE_ID_00002</g:id>
      <g:title>Two tone shirt</g:title>
      <g:description>Two tone shirt</g:description>
      <g:link>`+http://dummy-client.qubitproducts.com/shirts/two-tone-shirt.html+`</g:link>
      <g:image_link>`+http://fashion-demo.qubitproducts.com/media/catalog/product/cache/1/image/800x800/9df78eab33525d08d6e5fb8d27136e95/d/r/shirt.jpg+`</g:image_link>
      <g:availability>in stock</g:availability>
      <g:price>169.00 USD</g:price>
      <g:product_type>Men's clothing > Dresses</g:product_type>
      <g:sale_price>160.00 USD</g:sale_price>
      <c:exclude_from_recommendations type="boolean">true</c:exclude_from_recommendations>
    </item>
  </channel>
</rss>

Catalog field mapping

Catalog fields can be mapped from:

a Google feed.

Fields are mapped "as is", except for ones in the table below.
the ecProduct events.

Fields are mapped as specified in the table below.

Other fields are taken from the ecProduct.product object with:
- underscores instead of dots
- the product_ prefix dropped.
  
  Example
  
  product.manufacturer becomes manufacturer.
both sources. In that case, the Google feed is prioritized.

Catalog field Merged from QP events Renamed during catalog creation Derived during catalog creation Notes

base_currency

Renamed from product_price_baseCurrency in the ecProduct events.

categories

Categories are extracted from product_type field in the Google feed field. That field supports multiple comma-separated categories.

category

The first value from the categories array.

currency

Derived from the locale chosen when uploading this feed into Coveo Experimentation Hub.

id

Checks which of the following fields from the Google feed has the best match rate with product_id in the ecProduct events:

id
mpn
gtin
product_id
item_group_id
sku

Once the best match is found, it’s mapped to id.

image_url

Renamed from image_link in the Google feed. If multiple images are provided through a comma-separated list, only the first image link is used.

images

Derived by creating an array where first value is image_link and other values are from the additional_image_link value.

language

Derived from the locale chosen when uploading this feed into Coveo Experimentation Hub.

locale

Derived from the locale chosen when uploading this feed into Coveo Experimentation Hub.

name

Renamed from title in the Google feed.

product_stock

Populated using the value of the product_stock field in ecProduct events.

sku_code

Checks which of the following fields exist in the Google feed, in that order:

mpn
id
gtin

The first found field is mapped to sku_code, the rest (if any) are not mapped.

stock

If stock is contained in the Google feed, this value is used.

Otherwise, the stock integer value is retrieved from the following fallback logic:

Check for quantity in the feed.
Check for sku_stock_level in the feed.
Check if availability == out of stock (or out_of_stock), then set to `0 `.
If all checks fail, this field is not generated.

unit_base_price

Mapped from product_originalPrice_baseValue in the ecProduct events.

unit_price

Mapped from price in the Google feed. Otherwise, product_originalPrice_value is used from the ecProduct events.

unit_sale_base_price

Mapped from product_price_baseValue in the ecProduct events.

unit_sale_price

Mapped from sale_price in the Google feed. Otherwise, product_price_value is used from the ecProduct events.

url

Renamed from link in the Google feed. If that doesn’t exist, falls back to product_url in the ecProduct events.

views

Count of the ecProduct events over the last 15 days.

Next steps

Once your data layer or product feed has been verified, you can add the Recommendations programmatic experience. You can do this in 2 different ways.

You can either add it as an experience in the Qubit platform or call the Recommendations API.