Image search

This is for:

System Administrator Developer
In this article

Allowing shoppers to search using images is a great way to help them find similar products quickly.

In a digital commerce scenario, it can help visitors find products in a new way that doesn’t rely on text search. For example, in a B2C scenario, a visitor might have an image of a specific kayak paddle that they want to buy, but they don’t know the brand or model name. Conversely, in a B2B scenario, a visitor might have a specific part they need to replace, but they don’t know the part number. In either scenario, by allowing them to upload the image, they can find the product they’re looking for.

This article explains how you can leverage a third-party API to enable visual search on a Coveo-powered storefront.

Image search at Coveo

To enable image search on your Coveo-powered storefront, you’ll need to rely on a third-party image recognition API.

Generic image recognition services, such as Google Cloud Vision or Amazon Rekognition, can be used to recognize a wide range of images, but they might not be as accurate as specialized services, which are designed to recognize images in specific product verticals, such as fashion retail or home furniture.

For example, if you operate a fashion retail business, you might want to use a specialized image recognition service that’s designed to recognize clothing items.

Prerequisites

Before you start leveraging visual search, you need:

  • Access to an image classification service.

  • Enriched product content with the same third-party image classification service. For example, if you used Google Cloud Vision to enrich your product content, you need to use the same service to classify the images uploaded by your visitors.

In a traditional search scenario, visitors enter a query in a search box and receive a list of results.

By integrating visual search, you allow your visitors to upload an image instead of entering a query. This image is then sent to a third-party image recognition service, which returns a list of labels that describe the image. These labels are then transformed into a query that the Coveo Platform can understand.

This means that you need to:

  1. Implement the ability for visitors to upload an image on your search interface. Coveo doesn’t provide this UI component out of the box, so you’ll need to build it yourself.

  2. Implement the logic that sends the image to the third-party image recognition service.

  3. Implement the logic that transforms the labels returned by the image recognition service into a query that Coveo can understand.

  4. Send the query to Coveo and display the results.

Step 1: Implement the ability for visitors to upload an image

To enable image uploads from your visitors, you need to add an upload feature to your search interface. This component should allow visitors to upload an image from their device or provide a link to an image.

If you let visitors upload images from their devices, you need to manage the upload and convert the image into a URL format for the image recognition service.

Image search component on Barca | Coveo

Step 2: Send the image to the image recognition service

Once the visitor has uploaded an image, you need to send it to the image recognition service.

To achieve this, you’ll need to implement a function that sends the image URL to the image recognition service API. The API will return a list of labels that describe the image.

Step 3: Transform the labels into a Coveo query

By sending an image to the image recognition service, you get back a list of labels that describe the image.

For example, by sending the kayak image to an image classification service, you might get back the following response:

[...]
"tags": [
{
    "tag": "category:kayak",
    "score": 0.9984000325202942
},
{
    "tag": "color:yellow",
    "score": 0.9795398712158203
},
{
    "tag": "accessories:water proof front storage",
    "score": 0.8808417320251465
},
[...]
]

To perform a search in Coveo based on these labels, you’ll need to implement a function that transforms the labels returned by the image recognition service into a query that Coveo can understand.

One way to do this is to use the label values to format a basic query expression that combines the returned label values and sends it to Coveo. Based on the previous response, the query could look like this:

kayak yellow water proof front storage

Tip
Leading practice

Most image recognition services return a list of labels with a confidence score. To improve the accuracy of your search, you can filter out labels with a low confidence score.

For example, using the previous response, you could filter out labels with a score below 0.9. This means that the accessories:water proof front storage label would be excluded from the query. The query would then look like this:

kayak yellow

Step 4: Send the query to Coveo and display the results

Once you have the query, you need to send it to the Coveo platform and display the results on a search results page.

To achieve this, you’ll need to implement a mechanism that takes the query expression obtained from step 3 and sends it to Coveo.

This can be done by using the query property in the body of a request to the Search endpoint of the Commerce API.