Multi-modal embeddings

This is for:

System Administrator

Multi-modal embeddings are the product representations at the core of the Multi-Modal Catalog Encoder. Each embedding combines a product’s textual content and its visual signals into a single vector used for semantic retrieval and ranking.

Where the Catalog Semantic Encoder (CSE) generates embeddings from text alone, multi-modal embeddings combine two sources of information:

  • Textual signals: Product titles, descriptions, and structured attributes.

  • Visual signals: Characteristics identified from product images, such as color, shape, pattern, material, and style.

Because both signals live in the same semantic space as shopper queries, a query like checkered flannel shirt can match a product whose image shows a checkered pattern, even if the word checkered appears nowhere in the catalog entry.

Product embeddings

The encoder generates one embedding per product from the signals available for that product. Textual content and visual characteristics are represented together, so a single embedding captures both what a product’s data says about it and what its images show.

Products that are semantically similar sit close together in the resulting space, and dissimilar products sit far apart. This is what lets a descriptive query retrieve visually and conceptually related products, rather than only products that share exact keywords.

Flexible input handling

The encoder adapts to the data available for each product. A product can be encoded from its text and images together, from text alone when no usable image exists, or from images alone when textual content is minimal.

The following diagram shows how the encoder creates a product embedding from the usable textual and visual signals available for each product.

Diagram showing a complete product with text and image inputs

As a result, every product in your catalog receives a semantic representation, regardless of how complete its data is. Relevance degrades gracefully rather than failing when one type of input is missing or unusable.

How visual signals contribute to relevance

Visual signals complement textual relevance rather than replacing it. The encoder balances both sources:

  • Descriptive queries benefit the most from visual signals. Queries that express style, appearance, or use-case intent match against the visual characteristics of products.

  • Navigational, SKU-based, and exact-match queries behave as they do today. Visual signals don’t interfere with shoppers who know exactly what they’re looking for.

  • High-confidence textual matches retain their weight. When textual or attribute-based signals provide strong relevance confidence, image-based signals don’t dominate the ranking.

When behavioral and usage data is available, the encoder uses it to refine how much influence visual signals have. In scenarios where image-based signals are observed to hurt relevance, their influence is reduced or disabled.

Query embeddings

At query time, the shopper’s query is encoded into the same semantic space as the product embeddings. To keep search fast, the encoder uses a query embedding cache.

Precomputed queries

When the model builds, embeddings are precomputed for your catalog’s frequent queries and loaded into the cache before the model starts serving traffic. Frequent queries are then served from the cache, which keeps latency low for the vast majority of traffic.

Real-time queries

Queries that aren’t in the cache are encoded in real time, then added to the cache for subsequent shoppers. This ensures that new, long-tail queries still benefit from full semantic understanding.

Hybrid retrieval and ranking

Multi-modal embeddings power semantic matching within Coveo’s hybrid search. Products whose embeddings are sufficiently similar to the query embedding are boosted alongside traditional lexical matching, so semantic and keyword relevance work together rather than competing.

The similarity thresholds and boosting parameters that govern this behavior are tuned automatically for your catalog when the model builds. No manual tuning is required.

Because multi-modal embeddings deliver semantic relevance without requiring prior search usage or behavioral data, the encoder is effective from the first day of a deployment. This keeps relevance strong for long-tail queries and for newly added products that have no interaction history.

Embedding updates

Embeddings stay in sync with your catalog automatically through the following mechanisms:

  • Incremental updates: Only newly added or modified products are embedded on an ongoing basis, so full catalog reprocessing isn’t required.

  • Safe reprocessing: Embedding generation is safe to run multiple times for the same product without affecting relevance or ranking consistency.

  • Validation: Generated embeddings are validated before they’re published, so incomplete or empty representations never reach your search index.

Supported languages

Multi-modal embeddings support queries in English, French, German, Spanish, and Italian. Support for additional Coveo-supported languages is planned for future releases.

Compatibility

Multi-modal embeddings are compatible with the broader Coveo for Commerce relevance stack, including hybrid lexical and semantic retrieval and machine learning ranking models.