Coveo indexing pipeline

A source crawls its target repository to fetch and push the content and metadata of each repository item.

As a member of the Administrators or Content Managers built-in groups, you can add sources from the Coveo Administration Console. Some item metadata is already made available by the connector, including the URI, modification date, and more depending on the source.

This stage is the first Push API or Stream API stage. It handles the reception of items to index from an external custom process.

As a developer, you control when and what items you send to the Push API or Stream API, one by one or in batch.

The Streaming extension queue is unique to the Stream API. It holds items ready to be processed by the Streaming extension stage.

The Streaming extension is unique to the Stream API. It should be viewed as a way to enable specific ML features, such as Coveo Personalization-as-you-go, and is the only way to benefit from partial item updates.

The Push API queue holds items ready to be processed by the Consuming stage.

This is the last stage of the Push API or Stream API set. It transfers items from the Push API queue to be processed by the document processing manager (DPM). Items are sent to the DPM queue, one by one or in batch, depending on how they were pushed to the Streaming stage.

The DPM queue holds items ready to be processed by the document processing manager (DPM) set of stages.

Detecting is the initial stage of the DPM set. It checks the item content to determine its file type, such as PDF, HTML, XML, etc.

In this stage, extensions are applied before the item contents are converted into a different format in the Processing stage. See more information on how to use pre-conversion extensions.

If you’re unsure which stage best fits your extension, you can compare pre-conversion and post-conversion. A script may also be assigned to a source twice, once in each Applying extension stage, though it’s crucial to avoid making scripts too general.

When optical character recognition is enabled for a source, at this stage, Coveo extracts text from images and PDF files. The extracted text is then searchable in a Coveo-powered search interface and appears in the item Quick view. For details on this technology, see Optical character recognition.

This stage essentially converts the content and properties of each item from its native format into a common format suitable for the Indexing stage using the appropriate Coveo converter for the supported file formats.

This stage applies standard and custom source mappings to set Coveo field values with item metadata or literal text.

As an administrator:

This stage is similar to Applying extension (pre-conversion). However, extensions are applied to items after they’ve been converted and mapped in the Processing and Mapping stages. Typically, more extensions are applied in post-conversion than pre-conversion. See more information on how to use post-conversion extensions.

If you’re unsure which stage best fits your extension, you can compare pre-conversion and post-conversion. A script may also be assigned to a source twice, once in each Applying extension stage, though it’s crucial to avoid making scripts too general.

This stage further processes item fields and metadata to optimize index efficiency.

This stage holds items ready to be processed by the Indexing stage.

This stage puts the item extracted content and properties into the Coveo unified index to make it available for user queries. Temporary files containing the extracted item content and properties are then deleted.