Coveo Crawling Module

This is for:

Developer System Administrator

You can use the Coveo Crawling Module software to index on-premises content and make it searchable in a Coveo-powered search interface. This overview explains what the Crawling Module is, when to use it, what it can index, and how it works.

What’s the Coveo Crawling Module?

The Coveo Crawling Module is software installed on your server that retrieves content from your on-premises systems. The Crawling Module sends this content to the Coveo Platform, which then indexes your data.

The Crawling Module is a Coveo Enterprise edition license add-on. Your license must also include the Crawling Module sources you want to use. See the Crawling Module requirements for details.

When to use the Coveo Crawling Module?

Use the Crawling Module when both of these statements apply:

  • Your content is on premises.

  • Your firewall doesn’t allow a Coveo-hosted source to open an inbound connection.

For example, you can deploy it on a Windows server on your premises or in Azure when your security policies allow outbound communication but not inbound access.

If you can open a port in your firewall to let a cloud-hosted source access your on-premises content, you don’t need to install the Crawling Module. Instead, you can create on-premises sources in the Coveo Administration Console. Similarly, if you want to index cloud content, you must use the dedicated source. See the source documentation for detailed instructions.

What can the Crawling Module index?

When your Coveo license allows it, the Crawling Module can index the following content:

How does the Coveo Crawling Module work?

The Crawling Module connects your on-premises content to the Coveo Platform without requiring inbound access to your secured enterprise systems. After you deploy the Crawling Module on a Windows server on your premises or in Azure, all communications are outbound.

Components

The Coveo Crawling Module has three components:

  • Maestro, a software managing and monitoring your local workers and the State Store.

  • One or more workers, which are responsible for executing content update tasks requested by the Coveo Platform. Each worker can only handle one task at a time, so the content you index can require more than one worker. See Number of workers for details.

  • The State Store, which stores information regarding the last update operations, such as the source state and the URI of indexed items. As a result, the workers know what has been indexed during the last update operation, and therefore what needs to be indexed next time.

Workflow

The Coveo Crawling Module indexing workflow is the following:

Crawling Module Workflow

  1. The Crawling Module workers periodically poll the Coveo Platform for source update tasks. When an update is due, it is assigned to the next available content worker.

  2. The State Store provides the worker with information regarding the last source update operation.

  3. Based on the information provided by the State Store, the worker determines which pieces of content changed since the last source update operation and crawls them.

  4. The worker provides the State Store with information on the task it just completed.

  5. If applicable, the worker applies a pre-push extension to the crawled content.

  6. The worker provides the Push API with the items to index. The worker authenticates with the API key Maestro received at the end of its installation process.

  7. The Push API relays the received items so that the content in your Coveo-powered search interface reflects your actual on-premises data. See Coveo indexing pipeline for details on the indexing process.

  8. If you make changes to the Crawling Module configuration, Maestro applies them to the workers and the State Store.

If you have a source that indexes permissions, your security worker follows a similar workflow when indexing permissions.

What’s next?