Number of Workers

Workers are the Coveo On-Premises Crawling Module components responsible for executing content update tasks requested by the Coveo Cloud platform (see Workflow and Refresh, Rescan, and Rebuild).

A worker executes a single content update operation at once. When you have more than one worker, they can work simultaneously. For instance, with two workers, a Jira Software source refresh and a File System source rescan can happen at the same time, while a single worker would execute these tasks one after the other (see Supported Content).

Operations involving a significant number of items are not split between the available workers, so there is no advantage in having extra workers if there are no operations to execute simultaneously. In fact, having several unused workers may negatively impact performance.

When the Coveo Cloud platform requests an On-Premises Crawling Module operation while all workers are busy, this new task is delayed until one of the workers becomes available. Content update operations repeatedly executed several hours behind schedule indicate that you have too few workers (see Review Your Source Status). As a result, search results in your Coveo-powered search page may not reflect your actual data. You should therefore increase your number of workers to better fit the content you want to index.

In practical terms, if you use the Crawling Module to retrieve the content of a single on-premises repository, you do not need more than one worker.

If you have more than one source of content, a single worker may be enough if simultaneous operations can be avoided, for instance when:

  • Your sources do not have many items to retrieve or update, so update operations typically do not take long.
  • You can schedule the update operations so that one finishes before the other is due.

However, sources for which updates take a significant amount of time should usually have one worker each, so that the timely update of other sources is not hindered.

In short, since size varies considerably from one source to another, there is no rule of thumb regarding the number of workers you should have based on your number of Crawling Module sources. If you have more than one source, Coveo suggests to start with two workers, monitor their update operations, and then adjust the number of workers as needed.

What’s Next?

  • Knowing the numbers of workers you need allows you to adjust the hardware of the server on which you will deploy the Crawling Module (see Hardware).
  • Once you have deployed the Crawling Module, you can edit its configuration to create or delete workers.
Recommended Articles