Analyze the rebuild process

This is for:

Developer System Administrator

Rebuilding is the action of crawling a set of items and pushing them into your Coveo organization source. At the end of the process, the source is expected to contain only the crawled items. A Coveo index rebuild in Sitecore, whether launched through the Sitecore user interface or the Coveo Command Center, includes many steps. Items are first handled by Sitecore and then handed over to Coveo for Sitecore. The best way to understand which Sitecore items are indexed, and how they’re indexed, is to monitor activities using the Coveo rebuild logs.

This article breaks down Coveo index rebuild log messages and explains what Coveo for Sitecore is actually doing as these messages are being recorded. This article also shows how you can isolate Coveo rebuild logs in a separate file and change the log level to help you diagnose rebuild issues.

Coveo rebuild phases

Coveo rebuild logs are divided into four phases, the same phases you see in the rebuild status bar during a Command Center rebuild.

Image of indexing phases in the Command Center during a rebuild | Coveo for Sitecore 5

One Setup Requirements phase

During this phase, the Sitecore configuration is compared to the resources in the Coveo Platform. The sources, fields, and security providers are modified in Coveo to match Sitecore configurations.

Two Sending Permissions phase

During this phase, all Sitecore users and roles are sent to Coveo. This way, Coveo doesn’t have to contact the Sitecore instance at query time.

Three Sending Documents phase

Using the crawlers configured on the search index, Sitecore iterates over a set of items and passes them to Coveo for Sitecore. In most cases, when a document isn’t indexed, the reason is that Coveo for Sitecore hasn’t received the item from Sitecore.

Coveo for Sitecore creates batches of 10 documents that are uploaded in Coveo. The time required to upload all items depends on the number of items, the time to render the page, and the bandwidth of your Internet connection.

Four Documents Validation phase

The Waiting for Items to Be Searchable step validates that all items sent to Coveo are being committed and are searchable. It displays the number of committed items and the number of expected items.

A rebuild operation replaces the source content with a new set of items. However, the old items have to be removed. The Removing Old Items step monitors the old items and ensures they’re removed. This process times out after 5 minutes of inactivity.

Note

The Documents Validation phase waits for all steps in previous phases to complete. For example:

  • The Documents Validation phase pauses while document uploading steps are still in progress.

  • When you create a Coveo organization, the Coveo service has to deploy some components before the organization is fully functional. This process is called organization provisioning. Organization provisioning must be completed before steps in the Documents Validation phase can begin.

Coveo rebuild logs

By default, the Coveo rebuild logs are recorded in log.<DATE>.txt and log.<DATE>.<TIME>.txt files in the <SITECORE_INSTANCE_ROOT>\Data\logs folder, with INFO as the log level.

Tip
Leading practice

The Coveo Diagnostic Page provides a convenient log viewer. It reads the .txt files from the <SITECORE_INSTANCE_ROOT>\Data\logs folder and color codes messages based on their severity.

Coveo rebuild log messages contain the name of the related source, which reduces confusion when many search indexes are rebuilt at the same time. Coveo logs messages to mark the beginning and end of each rebuild phase.

Image of Coveo rebuild logs | Coveo for Sitecore 5

Color coded example of Coveo rebuild logs.
One Setup Requirements phase
One Sending Permissions phase
One Sending Documents phase
One Documents Validation phase

Configuring custom Coveo logs

Coveo for Sitecore makes it easy to customize its rebuild logs by providing a log4net custom log configuration that you can enable and adjust. This configuration lets you:

  • Record Coveo logs in separate files (for example, in Coveo.Search.<DATE>.txt files when using the configuration as is).

  • Lower the log level to DEBUG to increase logging granularity.

    Image of custom Coveo rebuild logs on the Diagnostic Page log viewer
    Figure 1. Setting the log level to DEBUG gives you the ability to search for a Sitecore Item ID and better follow how the item is handled during the rebuild.

To record Coveo logs in a separate file and set the log level to DEBUG

  1. Open file Coveo.SearchProvider.Custom.config in a text editor.

    Note

    The Coveo.SearchProvider.Custom.config file is located in the <SITECORE_INSTANCE_ROOT>\App_Config\Include\Coveo\ folder.

  2. Locate the <log4net> element. It should be near the end of the file and commented out.

    <!--
  <log4net>
    <appender name="CoveoLogger" type="log4net.Appender.RollingFileAppender, Sitecore.Logging">
      <file value="$(dataFolder)/logs/Coveo.Search" />
      <appendToFile value="true" />
      <rollingStyle value="Composite" />
      <datePattern value=".yyyyMMdd'.txt'" />
      <staticLogFileName value="false" />
      <maxSizeRollBackups value="20" />
      <maximumFileSize value="250MB" />
      <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="%4t %d{ABSOLUTE} %l %-5p %m%n" />
      </layout>
    </appender>
    <logger name="Coveo" additivity="false">
      <level value="INFO" />
      <appender-ref ref="CoveoLogger" />
    </logger>
  </log4net>
-->

  3. Uncomment the <log4net> element.

  4. Set the log level to DEBUG.

    <level value="DEBUG" />
    Important

    Setting the log level to DEBUG has an impact on server performance and the size of log files. Don’t forget to set the log level back to INFO or to comment out the <log4net> element when you no longer need the added granularity.

  5. Save your changes.