---
title: Coveo Crawling Module
slug: '3260'
canonical_url: https://docs.coveo.com/en/3260/
collection: index-content
source_format: adoc
---
# Coveo Crawling Module
The Coveo Crawling Module allows you to [index](https://docs.coveo.com/en/204/) [on-premises content](#supported-content) in order to make it searchable in a Coveo-powered [search interface](https://docs.coveo.com/en/2741/).
Customers who can't open an inbound port in their firewall for the Coveo-hosted [connectors](https://docs.coveo.com/en/2734/) to access their on-premises content are typical Crawling Module users.

> **Note**
>
> If you can open a port in your firewall to let a cloud-hosted connector access your on-premises content, you don't need to install the Crawling Module.
> Instead, you can create [on-premises sources](https://docs.coveo.com/en/1612/) in the [Coveo Administration Console](https://docs.coveo.com/en/183/).
> Similarly, if you want to index cloud content, you must use the dedicated source.
> See the [connector documentation](https://docs.coveo.com/en/1702/) for detailed instructions.

The hosted Coveo connectors pull content from cloud and on-premises secured enterprise systems to make it searchable.
The Coveo Crawling Module runs outside of the Coveo Platform.
It pulls your content from your on-premises systems, and then sends it to a Coveo push-type source, which serves as an intermediate to index your data.

Once the Coveo Crawling Module is [deployed on a Windows server](https://docs.coveo.com/en/3261/) (either on your premises or running in Azure), all communications are outbound, and no inbound ports to your secured enterprise system are required.
Nevertheless, you can [manage](https://docs.coveo.com/en/3390/) a Crawling Module source just like you would manage a cloud source.

## Supported content

When your Coveo [license](https://docs.coveo.com/en/3262#coveo-license) allows it, the Crawling Module can index the following content:

* [Confluence Data Center](https://docs.coveo.com/en/1822/)

* [On-premises database](https://docs.coveo.com/en/1885/)

* [File system](https://docs.coveo.com/en/1766/)

* [Jira Software Data Center](https://docs.coveo.com/en/1772/)

* [SharePoint server](https://docs.coveo.com/en/2061/)

* [Sitemap](https://docs.coveo.com/en/1967/)

* [Web](https://docs.coveo.com/en/malf0160/)

* Any other application exposing its data through a [REST API](https://docs.coveo.com/en/1896/) or a [GraphQL API](https://docs.coveo.com/en/n6gh2329/)

## Components

The Coveo Crawling Module has three components:

* Maestro, a software managing and monitoring your local workers and the State Store.

* One or more workers, which are responsible for executing [content update tasks](https://docs.coveo.com/en/2039/) requested by the Coveo Platform.
Each worker can only handle one task at a time, so you may need more than one, depending on the content to index.
See [Number of workers](https://docs.coveo.com/en/3278/) for details.

* The State Store, which stores information regarding the last update operations, such as the source state and the URI of indexed items.
As a result, the workers know what has been indexed during the last update operation, and therefore what needs to be indexed next time.

## Workflow

The Coveo Crawling Module indexing workflow is the following:

![Crawling Module Workflow](https://docs.coveo.com/en/assets/images/crawling-module/dockerless-crawling-module.png)

. The Crawling Module workers periodically poll the Coveo Platform for [source update tasks](https://docs.coveo.com/en/2039/).
When an update is due, it is assigned to the [next available content worker](https://docs.coveo.com/en/3278/).

. The State Store provides the worker with information regarding the last source update operation.

. Based on the information provided by the State Store, the worker crawls the source content.

. The worker provides the State Store with information on the task it just completed.

. If applicable, the worker applies a [pre-push extension](https://docs.coveo.com/en/3269/) to the crawled content.

. The worker provides the Push API with the items that have changed since the last source update operation.
The worker authenticates with the API key Maestro received at the end of its installation process.

. The Push API relays the received items so that the content in your Coveo-powered search interface reflects your actual on-premises data.
See [Coveo indexing pipeline](https://docs.coveo.com/en/1893/) for details on the indexing process.

. If you make changes to the Crawling Module configuration, Maestro applies them to the workers and the State Store.

If you have [a source that indexes permissions](https://docs.coveo.com/en/1779/), your [security worker](https://docs.coveo.com/en/3278#security-workers) follow a similar workflow when indexing [permissions](https://docs.coveo.com/en/223/).

## What's next?

[Deploy the Crawling Module](https://docs.coveo.com/en/3261/).