Change the crawling root of an index

This is for:

Developer System Administrator

By default, Coveo for Sitecore indexes all items under /sitecore/content, as well as all items under /sitecore/media library/Files. You may want to change the crawling root of your indexes, most often to prevent indexing undesirable items in specific indexes. This can be particularly useful in multi-site installations, or if you have indexes dedicated to certain content.

By changing the crawling root, you can focus the indexing on an item and its content tree, which can accelerate your indexing time.

Modifying a crawling root

To change the crawling root of a crawler

  1. Open the Indexes page of the Command Center. You can access it at https://<INSTANCE_HOSTNAME>/coveo/command-center/index.html#indexes/.

  2. Expand the index you want to reconfigure by clicking anywhere on its row.

  3. At the end of the applicable crawler row, click Edit.

  4. In the root row, in the Value column, specify the new content tree path.

  5. Click Save.

  6. Back on the Indexes page, click Apply and Restart.

    Note

    Behind the scenes, the Coveo.SearchProvider.Custom.config file is updated.

To test your new crawler configuration

  1. In the Indexes page, expand the related index by clicking anywhere on the index row.

    CrawlerRootChangedIndexRebuild

  2. Click Rebuild.

    Note

    A crawler refresh doesn’t delete existing indexed items that don’t match the new crawler root. A rebuild deletes those old items.

  3. If your new crawler is properly configured, no error message appears and your browser developer tools show a successful rebuild API call.

    CrawlerRootChangedSuccessfulRebuildCall

  4. A short time after rebuild completion, the Command Center Indexes page displays the remaining number of indexed documents for the related index in the Content column.

    Note

    A Command Center process polls your Coveo organization on a periodic basis to get the number of items in your indexes.

  5. You can also view your remaining indexed items in the Coveo Administration Console Content Browser (platform-ca | platform-eu | platform-au).

Adding a new crawling root

You might also want to add a new crawler to an index and define its root.

To add a new crawler on an index

  1. Open your Coveo.SearchProvider.Custom.config file in a text editor. It’s located under <SITECORE_INSTANCE_ROOT>\App_config\Include\Coveo.

  2. Locate the following code block.

    <contentSearch>
  <configuration>
    <indexes hint="list:AddIndex">
      <index id="Coveo_master_index">
        <locations hint="list:AddCrawler">
          <crawler name="ContentCrawler">
            <!-- Uncomment this node if you want to change the crawling root to a smaller part of your site. -->
            <!--<root>/sitecore/content/YOUR_PATH</root> -->
          </crawler>
          <!-- Uncomment this node if you want to crawl a different section of your site. -->
          <!--
          <crawler name="SubSiteCrawler" type="Sitecore.ContentSearch.SitecoreItemCrawler, Sitecore.ContentSearch">
            <database>master</database>
            <root>/sitecore/YOUR_PATH</root>
            <stopOnError>true</stopOnError>
          </crawler>
          -->
        </locations>
      </index>
      <index id="Coveo_web_index">
        <locations hint="list:AddCrawler">
          <crawler name="ContentCrawler">
            <!-- Uncomment this node if you want to change the crawling root to a smaller part of your site. -->
            <!--<root>/sitecore/content/YOUR_PATH</root> -->
          </crawler>
          <!-- Uncomment this node if you want to crawl a different section of your site. -->
          <!--
          <crawler name="SubSiteCrawler" type="Sitecore.ContentSearch.SitecoreItemCrawler, Sitecore.ContentSearch">
            <database>web</database>
            <root>/sitecore/YOUR_PATH</root>
            <stopOnError>true</stopOnError>
          </crawler>
          -->
        </locations>
      </index>
    </indexes>
  </configuration>
</contentSearch>

  3. Locate the <index> element associated with the Sitecore database you want to target. For the sake of this example, we assume we want to add a new crawler in the Coveo_web_index index.

  4. As indicated in the configuration file, uncomment the <crawler name="SubSiteCrawler" ...> element.

    <index id="Coveo_web_index">
  <locations hint="list:AddCrawler">
    <crawler name="ContentCrawler">
      <!-- Uncomment this node if you want to change the crawling root to a smaller part of your site. -->
      <!--<root>/sitecore/content/YOUR_PATH</root> -->
    </crawler>
    <!-- Uncomment this node if you want to crawl a different section of your site. -->
    <crawler name="SubSiteCrawler" type="Sitecore.ContentSearch.SitecoreItemCrawler, Sitecore.ContentSearch">
      <database>web</database>
      <root>/sitecore/YOUR_PATH</root>
      <stopOnError>true</stopOnError>
    </crawler>
  </locations>

  5. Change the name of the crawler to one you deem appropriate.

    <index id="Coveo_web_index">
  <locations hint="list:AddCrawler">
    <crawler name="ContentCrawler">
      <!-- Uncomment this node if you want to change the crawling root to a smaller part of your site. -->
      <!--<root>/sitecore/content/YOUR_PATH</root> -->
    </crawler>
    <!-- Uncomment this node if you want to crawl a different section of your site. -->
    <crawler name="Staff" type="Sitecore.ContentSearch.SitecoreItemCrawler, Sitecore.ContentSearch">
      <database>web</database>
      <root>/sitecore/YOUR_PATH</root>
      <stopOnError>true</stopOnError>
    </crawler>
  </locations>
    Note

    Only use alphanumeric characters for your crawler names.

  6. Set the <root> element text content to your desired crawling root. Add a trailing slash to your path.

    <index id="Coveo_web_index">
  <locations hint="list:AddCrawler">
    <crawler name="ContentCrawler">
      <!-- Uncomment this node if you want to change the crawling root to a smaller part of your site. -->
      <!--<root>/sitecore/content/YOUR_PATH</root> -->
    </crawler>
    <!-- Uncomment this node if you want to crawl a different section of your site. -->
    <crawler name="Staff" type="Sitecore.ContentSearch.SitecoreItemCrawler, Sitecore.ContentSearch">
      <database>web</database>
      <root>/sitecore/Content/Home/Staff/</root>
      <stopOnError>true</stopOnError>
    </crawler>
  </locations>

  7. Save and close your file.

  8. The Command Center now displays your new crawler.

    AddCrawler

  9. Test your new configuration.

    1. In the Command Center, in the Indexes section, click the Refresh button associated with your new crawler.

    2. If your new crawler is properly configured, no error message appears and your browser developer tools show a successful refresh API call.

      SuccessfulRefresh