Crawling Module Requirements
Crawling Module Requirements
Your Coveo license must also include the connectors you want to use. See Supported Content for details on what the Crawling Module can index. These connectors must allow the Crawling Module as a content retrieval method.
To check whether your Coveo license includes the desired Crawling Module connectors:
In the navigation menu, select Sources.
In the Add a Source of Supported Content panel, select the desired source. If the source has more than one content retrieval method, a drop-down menu with the available connectors appears, and you must select the Crawling Module option. If the Crawling Module connector is grayed out, your license doesn’t include it. You must then contact the Coveo Sales team to upgrade your license.
We recommend that you install the Coveo On-Premises Crawling Module on a server running Windows Server 2019. Windows Server 2016 is also supported.
Your server must also host the repository to index or have access to the server on which this repository is located.
By default, after its deployment, the Crawling Module has one content worker and one security worker.
CPU and RAM
CPU and RAM requirements are based on the number of workers you need. As a guideline, consider that a Crawling Module instance with 4 to 6 workers typically requires 4 CPUs and 16 GB of RAM.
If your server CPU or RAM is insufficient, you could experience unresponsiveness or crashes preventing update operations from completing.
When you install the Crawling Module on your server, you must select the disk on which you want to deploy it. Coveo recommends starting with a disk space of at least 100 GB.
Your actual required disk space might be higher depending on the number of items to index, as well as their type and size. As a rule of thumb, consider that 10 million items require at least 10 GB of State Store storage.
If your environment restricts outgoing communications, ensure to allow the IP addresses that the Crawling Module uses.
Moreover, ensure that outbound communication from the Crawling Module server is not blocked or slowed down by SSL decryption, antivirus scans, firewalls, Splunk, endpoint protection, etc.
Since the Crawling Module uses executable files (
.exe extension), its proper operation is susceptible to be blocked by antivirus software. You should therefore ensure that the antivirus software of the server on which you intend to install the Crawling Module doesn’t hinder Crawling Module activities.
Furthermore, to ensure that your antivirus software doesn’t flag Crawling Module executable files as potential threats, you should add the following to the antivirus software exclusion list:
- Your data folder (
- Your installation folder (
C:\Program Files\Coveo\Maestroby default)
Once you have ensured your environment meets all the above requirements, you can proceed with the Crawling Module deployment.