Number of workers
Number of workers
There are two types of workers:
Content workers, which accomplish content update tasks.
A worker executes one content update operation at a time. When you have more than one content worker, each works on its own task. For example, with two content workers, a Jira Software source refresh and a File System source rescan can happen simultaneously, while a single content worker would execute these tasks one after the other.
When all content workers are busy, any new content update task is delayed until one of them becomes available. Content update operations repeatedly executed behind schedule indicate that you have too few content workers. Consequently, the search results in your Coveo-powered search interface may not reflect your actual data.
By default, after its deployment, the Crawling Module has four content workers.
As a rule of thumb, we recommend starting with one content worker between two sources associated with this Crawling Module. So, if you have four Crawling Module sources, two content workers should typically suffice.
Each Crawling Module source that indexes permissions created with the new connector version comes with at least one security identity provider that provides Coveo with the permission model of each retrieved item. Coveo can then replicate this model in the search interfaces it powers so that each end user can see only the content they’re allowed to access in your original repository. The Security Identities (platform-ca | platform-eu | platform-au) Administration Console page shows a list of your security identity providers.
Security workers are responsible for crawling the content permissions and feeding security identity providers this information. They also extract members from group security identities and associate users with their email identity. For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.
Similarly to a content worker, a security worker can execute only one update operation at a time. It’s therefore crucial to have an adequate number to ensure that Coveo search results reflect your actual content permissions.
By default, after its deployment, the Crawling Module has four security workers.
As a rule of thumb, we recommend starting with one security worker between two sources that index permissions associated with this Crawling Module.
These are sources that have the Same users and groups as in your content system content security option selected, or
"sourceVisibility": "SECURED" if the source was created via API.
So, if you have four Crawling Module sources, and only two of them index permissions, a single security worker should typically suffice.
Knowing the numbers of workers you need allows you to adjust the hardware of the server on which you will deploy the Crawling Module.
You can also monitor your workers from the Crawling Module component dashboard.