Catalog enrichment using image classification

This is for:

In this article

Image classification at Coveo
About fields and field mappings
What’s next?

Building product content usually involves a lot of manual work, which is time-consuming and error-prone. Traditionally, this entails meticulously inputting relevant metadata for each product, leading to potential gaps or discrepancies in product information. For example, if you’re selling a product, you may need to manually add additional product information, such as the product dimensions, color, or manufacturer. This process can be tedious, especially if you have a large inventory of products.

Image classification harnesses the power of AI to automatically enrich product content with relevant metadata using product images. This process helps accurately describe your products, improving the relevance of your search experience, product recommendations, and shopper personalization.

Among other benefits, image classification offers the ability to:

Fill missing product information that’s not currently available in your product content.
Ensure consistency in your product metadata. For example, without image classification, some products might have their color described as blue while others might have it described as navy.
Improve the quality of the search results.
Increase the number of results matching a search query.
Reduce the amount of manual work required to build your product content.
Focus on filling retailer-specific information, such as pricing and availability.

Enriching a product catalog with image classification

Image classification at Coveo

To enrich product content through image classification, you’ll need to rely on a third-party image classification service.

Generic image classification services, such as Google Cloud Vision or Amazon Rekognition, can be used to classify a wide range of images, but they might not be as accurate as specialized services, which are designed to classify images in specific product verticals, such as fashion retail or home furniture.

For example, if you operate a fashion retail business, you might want to use a specialized image classification service that’s designed to classify clothing items.

Prerequisites

Before you can start using image classification to enrich your product content and push the metadata to the Coveo Platform, you need:

Access to an image classification service, such as Google Cloud Vision or Amazon Rekognition.
A source that indexes your product content, such as a Catalog source.

How to enrich your product content with image classification

In your own ecosystem, your content usually resides in a storage system, such as a product information management (PIM) or digital asset management (DAM). In a typical commerce scenario, you need to develop middleware that connects your storage system to the Coveo index, formatting and sending data via the Coveo Stream API.

The product content must be formatted as a JSON object which contains the metadata that’s relevant to the item object type you’re indexing.

The following diagram shows the typical flow of indexing product content to a Coveo source when you aren’t leveraging image classification to enrich your product content:

Typical flow of indexing product content to Coveo

1	The content stored in your content storage systems is sent to a middleware.
2	The middleware formats the content in a JSON file.
3	The middleware sends the JSON file to the Coveo Stream API.
4	The Coveo Stream API pushes the content to your source in the Coveo index.

When you’re enriching your product content with image classification, you’ll need to add the metadata returned by the image classification service to your product content.

Your product content usually includes images, which are formatted as URLs that give public access to the image files, typically stored in a cloud storage service.

Most image classification services provide an API that you can use to send an image URL and get back the classification results. You’ll need to use the response from the image classification service to enrich your product content with the relevant metadata.

For example, by sending the kayak image to an image classification service, you might get back the following response:

{
  "kayak_type": "single seat",
  "color": "Yellow",
  "accessories": ["Water proof front storage", "Back storage"],
  [...]
}

Send enriched content to Coveo

Once you’ve received the metadata from the image classification service, you need to add it to your product content and push it to your Coveo index.

The process of pushing the enriched product content to the Coveo Platform is the same, whether you push the entire product inventory or only the updated products. In both cases, Coveo recommends that the enrichment procedure is done at the content storage system level, before the content is pushed to the Coveo Platform.

Once your product content includes the metadata returned by the image classification service, you can push it to your Coveo index using the Coveo Stream API. If your source already contains your product content, you can’t push only the updated metadata to the Coveo Platform. To update your source with the new metadata provided by the image classification service, you’ll need to push your whole product inventory to the Coveo Platform again using the Coveo Stream API.

The following diagram shows the flow of enriching your product content with image classification and sending it to your source in the Coveo Platform:

1	URLs of the product images stored in your content storage system are sent to the image classification service.
2	The image classification service processes the images.
3	The image classification service sends back the metadata to your content storage system, where the classification results are added to the product content.
4	The content stored in your content storage systems is sent to a middleware.
5	The middleware formats the content to match Coveo’s catalog schemas in a JSON file (see Commerce fields).
6	The middleware sends the JSON file to the Coveo Stream API.
7	The Coveo Stream API pushes the content to your source in the Coveo index.

About fields and field mappings

When you’re enriching your catalog data with image classifications, you need to ensure that the metadata you’re adding is correctly mapped to the fields in your Coveo index.

Example

If the image classification service returns a kayak_type metadata value, and that you want this information to be used in the Coveo index, you need to:

Create a kayak_type field in your Coveo index. Ensure that these fields have the Free text search and Stemming options enabled. See Add or edit a field for instructions on how to configure fields using the Coveo Administration Console.
Map the kayak_type metadata to the kayak_type field in your Coveo index.

What’s next?

When you’ve enriched your product content with image classification and pushed the metadata to your Coveo index, you can use this metadata to improve the relevance of your search results, recommendations, and shopper personalization.

To ensure that your product content now includes the metadata returned by the image classification service, you can use the Content Browser to inspect the content of your source.
When you’ve enriched your product content with image classification, you can leverage the classifications to implement image-based search on your Coveo-powered search interfaces.