Analyze the Rebuild Process

Rebuilding is the action of crawling a set of documents and pushing them into the index. At the end of the process, the search index is expected to contain only the crawled documents.

The Coveo Command Center provides a high-level monitoring view of the indexing process (see Rebuilding a Search Index Manually).

Indexing Details in the Sitecore Logs

The Sitecore logs show precise log traces to help monitor rebuild tasks. The rebuild task is divided into several phases. The beginning and end of each phases are clearly indicated in the logs. You can then see if one specific phase is taking longer than usual to execute.

Also, every trace contains the name of the source that’s being rebuilt, making it easy to untangle traces when many search indexes are rebuilt at the same time.

Here is an example of the logged traces during a rebuild task:

ManagedPoolThread #8 10:44:24 INFO  Job started: Index_Update_IndexName=Coveo_web_index
ManagedPoolThread #8 10:44:24 WARN  The index Coveo_web_index has already been initialized.
...
ManagedPoolThread #8 10:44:49 INFO  [Rebuilding source "YOUR COVEO SOURCE"] Rebuild started.
ManagedPoolThread #8 10:44:49 INFO  [YOUR COVEO SOURCE] Synchronizing source...
ManagedPoolThread #8 10:44:49 INFO  Total Field Count for Coveo_web_index: 202, Actual Field Count: 182
...
ManagedPoolThread #8 10:45:18 INFO  [YOUR COVEO SOURCE] Source synchronized.
...
ManagedPoolThread #8 10:45:19 INFO  [Permissions synchronization "Expanded Sitecore Security Provider for YOUR SITECORE INSTANCE"] Starting to send the permissions...
...
ManagedPoolThread #8 10:45:39 INFO  [Rebuilding source "YOUR COVEO SOURCE"] Crawling Sitecore items...
...
ManagedPoolThread #8 10:46:01 INFO  [Rebuilding source "YOUR COVEO SOURCE"] Sitecore items crawled.
ManagedPoolThread #8 10:46:01 INFO  [Rebuilding source "YOUR COVEO SOURCE"] Finalizing rebuild...
ManagedPoolThread #8 10:46:01 INFO  [Rebuilding source "YOUR COVEO SOURCE"] Waiting for items to be uploaded to Coveo Cloud...
...
ManagedPoolThread #8 10:46:04 INFO  [Rebuilding source "YOUR COVEO SOURCE"] items are uploaded.
ManagedPoolThread #8 10:46:04 INFO  [Rebuilding source "YOUR COVEO SOURCE"] Waiting for organization to be provisioned...
ManagedPoolThread #8 10:46:04 INFO  [Rebuilding source "YOUR COVEO SOURCE"] Organization is provisioned.
ManagedPoolThread #8 10:46:05 INFO  [Rebuilding source "YOUR COVEO SOURCE"] Waiting for items to be searchable...
...
ManagedPoolThread #8 10:46:15 INFO  [Rebuilding source "YOUR COVEO SOURCE"] Committed items: 1023 / 2041
...
ManagedPoolThread #8 10:46:50 INFO  [Rebuilding source "YOUR COVEO SOURCE"] Committed items: 2041 / 2041
ManagedPoolThread #8 10:46:50 INFO  [Rebuilding source "YOUR COVEO SOURCE"] Items are searchable.
ManagedPoolThread #8 10:46:51 INFO  [Rebuilding source "YOUR COVEO SOURCE"] Removing old items...
...
ManagedPoolThread #8 10:47:07 INFO  [Rebuilding source "YOUR COVEO SOURCE"] 538 remaining items.
...
ManagedPoolThread #8 10:47:15 INFO  [Rebuilding source "YOUR COVEO SOURCE"] 0 remaining items.
ManagedPoolThread #8 10:47:15 INFO  [Rebuilding source "YOUR COVEO SOURCE"] Old items removed.
ManagedPoolThread #8 10:47:16 INFO  [Rebuilding source "YOUR COVEO SOURCE"] Rebuild finished.
ManagedPoolThread #8 15:47:18 INFO  Job ended: Index_Update_IndexName=Coveo_web_index (units processed: 2041)

Here are some details regarding each phase of the rebuild process.

Synchronizing Source

This is when the Sitecore configuration is compared to the resources in the Coveo Platform. The sources, fields, or security providers are modified in the Coveo Platform to match Sitecore configurations.

Sending Permissions

All Sitecore permissions are sent to Coveo Cloud. This way, Coveo Cloud doesn’t have to contact the Sitecore instance at query time.

Crawling Sitecore Items

Using the crawlers configured on the search index, Sitecore iterates over a set of items and passes them to the search index using the Search Provider framework.

Waiting for Items to Be Uploaded to Coveo Cloud

Since Coveo Cloud is an online service, the items have to be uploaded in order to be indexed. The time required to upload the items varies depending on the number of items, the size of each item, and the bandwidth of your Internet connection.

Waiting for Organization to Be Provisioned

Upon creation, a Coveo organization takes a few minutes to be up and running. In other words, the Coveo Cloud service has to deploy some components before the organization is fully functional. At this point, the organization must be ready before the rebuild process can continue.

Waiting for Items to Be Searchable

This step validates that all items sent to the Coveo Platform are being committed and are searchable. It displays the number of committed items and the number of expected items. This process times out after 1 hour of inactivity.

Removing Old Items

A rebuild operation replaces the source content with a new set of items. However, the old items have to be removed. This step monitors the old items and ensures they’re removed. This process times out after 5 minutes of inactivity.

Recommended Articles