About Multi-Modal Catalog Encoder

This is for:

System Administrator
Important

The Multi-Modal Catalog Encoder is a premium capability offered under a separate entitlement from the Catalog Semantic Encoder (CSE). Contact your Coveo representative to enable it in your Coveo organization.

The Coveo Machine Learning Multi-Modal Catalog Encoder is a semantic search capability that combines catalog text, structured attributes, and product images to power product discovery. It lets shoppers find products by describing how those products look, not only by the words in your catalog data. For example, a shopper who searches for round modern black handle can find the right product even when your catalog calls it a knob and never mentions its shape or style.

The Multi-Modal Catalog Encoder builds on the Catalog Semantic Encoder (CSE) by generating product embeddings that incorporate visual signals from product images alongside titles, descriptions, and attributes. Because product images exist for nearly every SKU, they help fill the gaps left by incomplete or inconsistent catalog data, without manual enrichment or rule-based attribute extraction.

Shoppers continue to submit text queries. The encoder interprets those queries against both textual and visual signals, so it improves relevance for descriptive queries while preserving the behavior of navigational, SKU-based, and exact-match queries.

Prerequisites

Before you contact your Coveo representative to enable the Multi-Modal Catalog Encoder in your Coveo organization, make sure that you:

  • Have a Coveo license that includes the Multi-Modal Catalog Encoder entitlement.

  • Have configured a catalog entity, with the standard commerce fields populated and mapped, including the field that references product images (ec_images).

  • Have your catalog data indexed in your Coveo organization.

  • Make product images accessible to Coveo through the image URLs in your catalog data.

  • Have sufficient image coverage in your catalog. The encoder supports catalogs with up to 50% of products missing images.

What you can do with the Multi-Modal Catalog Encoder

Most catalogs don’t describe products the way shoppers search for them. Descriptions can be sparse, inconsistent, or focused on what a product does rather than what it is, and marketplace items often arrive with distributor-entered data that doesn’t meet your content standards. Text-only semantic search can perform poorly when the text it relies on is incomplete.

The Multi-Modal Catalog Encoder addresses this by treating product images as a relevance signal:

  • Search by look: Surface products based on visual characteristics such as color, shape, pattern, material, and style, even when those characteristics don’t appear in a product’s text.

  • Closed content gaps: Use visual signals to complement or substitute for missing textual attributes, reducing your reliance on catalog data completeness.

  • Relevance from the first query: Get strong semantic performance in cold-start scenarios, without prior search usage or behavioral data.

  • Less manual tuning: Lower the need for thesaurus entries and attribute enrichment work that compensate for content gaps.

The following table shows examples of queries where visual signals improve relevance:

Scenario Example query How images help

Visual descriptor search

round modern black handle

The shape, style, and color are identified from the product image, even if the catalog entry is titled “knob”.

Pattern and texture search

checkered shirt

The pattern is visible in the image but absent from the product’s attributes.

Sparse descriptions

mid-century walnut side table

The description covers usage rather than appearance, so the image supplies the missing style and material signals.

Colloquial queries

comfy oversized reading chair

Shopper vocabulary doesn’t match the catalog taxonomy, so visual understanding bridges the gap.

Core building blocks

The Multi-Modal Catalog Encoder has three building blocks, each covered in a dedicated article:

How the Multi-Modal Catalog Encoder works

The Multi-Modal Catalog Encoder operates in two phases: model build and query time. The model builds from the catalog data that’s already indexed in your Coveo organization, so catalog ingestion is unchanged and requires no modification to support the encoder.

The following diagram shows how the encoder creates and optimizes product representations during model build, then uses the published model to support query encoding and hybrid retrieval at query time.

Flow diagram showing Multi-Modal Catalog Encoder model build and shopper query flow | Coveo for Commerce

Model build

When the model builds, Coveo processes your catalog in three stages:

  1. Image processing: Coveo retrieves the images referenced in your indexed catalog and validates them. Low-quality, placeholder, and duplicate images are detected and excluded, and previously retrieved images are cached so that only new or modified images are downloaded.

  2. Embedding generation: The encoder generates a multi-modal embedding for each product, combining its textual content and visual signals. Only newly added or modified products are processed on an ongoing basis, so full catalog reprocessing isn’t required.

  3. Serving optimization: The model pre-computes embeddings for frequent queries and automatically tunes its serving parameters for your catalog, so no manual configuration is required.

Query time

At query time, the encoder turns each shopper query into ranked results:

  1. Query encoding: The shopper’s query is encoded into the same semantic space as the product embeddings. Frequent queries are served from the pre-computed embedding cache, and new queries are encoded in real time and added to the cache.

  2. Retrieval: Products whose embeddings are semantically similar to the query are retrieved and combined with traditional lexical matching in a hybrid search.

  3. Ranking: Results are ranked using the full Coveo relevance stack, including machine learning models and behavioral signals when available.

Note

When behavioral data indicates that image-based signals hurt relevance for certain scenarios, the encoder uses that data to reduce or disable their influence.

For details on how text and image signals combine, see Multi-modal embeddings.

How it differs from the Catalog Semantic Encoder

The Multi-Modal Catalog Encoder is an enhanced version of Coveo’s semantic search offering. When the premium entitlement is enabled, multi-modal embeddings replace the core embeddings used by the Catalog Semantic Encoder, and the rest of your search configuration is unaffected.

If multi-modal embedding services are unavailable or degraded, search automatically falls back to text-only semantic embeddings, so failures in multi-modal infrastructure never affect search availability. If the entitlement is removed, multi-modal behavior is fully disabled and your Coveo organization returns to the standard Catalog Semantic Encoder. For details about how to enable the encoder, validate its impact, and understand its fallback behavior, see Manage the Multi-Modal Catalog Encoder.

Supported languages and regions

The Multi-Modal Catalog Encoder is available in all supported Coveo regions. It supports queries in English, French, German, Spanish, and Italian. Support for additional Coveo-supported languages is planned for future releases.

What the Multi-Modal Catalog Encoder isn’t

The Multi-Modal Catalog Encoder uses product images only as an internal relevance signal. It’s distinct from the following capabilities:

  • Image search: Shoppers submit text queries. The encoder doesn’t accept an uploaded image as a query, so it isn’t reverse image search or image-upload search.

  • Catalog attribute generation: The encoder doesn’t generate catalog attributes or extract text from images to populate fields such as ec_name or ec_description.

  • Catalog enrichment: The encoder doesn’t persist any generated data back to your catalog and doesn’t modify your customer-visible catalog data. Visual signals are used only to build embeddings.

The Multi-Modal Catalog Encoder also doesn’t include embedding export to third-party systems or direct visibility into the vector space.