Change the Crawling Root of an Index

By default, Coveo for Sitecore indexes all items under /sitecore/content, as well as all items under /sitecore/media library/Files. You may want to change the crawling root of your indexes, most often to prevent indexing undesirable items in specific indexes. This can be particularly useful in multi-site installations, or if you have indexes dedicated to certain content.

By changing the crawling root, you can focus the indexing on an item and its content tree, which can accelerate your indexing time.

Modifying a Crawling Root

To change the crawling root of a crawler

  1. Open your Coveo.SearchProvider.Custom.config file in a text editor. It’s located under [Sitecore instance root]\Website\App_config\Include\Coveo or [Sitecore instance root]\App_config\Include\Coveo (in Sitecore 9 and 10 instances).
  2. Locate the following code block.

    <contentSearch>
      <configuration>
        <indexes hint="list:AddIndex">
          <index id="Coveo_master_index">
            <locations hint="list:AddCrawler">
              <crawler name="ContentCrawler">
                <!-- Uncomment this node if you want to change the crawling root to a smaller part of your site. -->
                <!--<root>/sitecore/content/YOUR_PATH</root> -->
              </crawler>
              <!-- Uncomment this node if you want to crawl a different section of your site. -->
              <!--
              <crawler name="SubSiteCrawler" type="Sitecore.ContentSearch.SitecoreItemCrawler, Sitecore.ContentSearch">
                <database>master</database>
                <root>/sitecore/YOUR_PATH</root>
                <stopOnError>true</stopOnError>
              </crawler>
              -->
            </locations>
          </index>
          <index id="Coveo_web_index">
            <locations hint="list:AddCrawler">
              <crawler name="ContentCrawler">
                <!-- Uncomment this node if you want to change the crawling root to a smaller part of your site. -->
                <!--<root>/sitecore/content/YOUR_PATH</root> -->
              </crawler>
              <!-- Uncomment this node if you want to crawl a different section of your site. -->
              <!--
              <crawler name="SubSiteCrawler" type="Sitecore.ContentSearch.SitecoreItemCrawler, Sitecore.ContentSearch">
                <database>web</database>
                <root>/sitecore/YOUR_PATH</root>
                <stopOnError>true</stopOnError>
              </crawler>
              -->
            </locations>
          </index>
        </indexes>
      </configuration>
    </contentSearch>
    
  3. Locate the <index> element associated with the Sitecore database you want to target. For the sake of this example, we assume we want to change the root of the ContentCrawler crawler in the Coveo_web_index index.

  4. As indicated in the configuration file, uncomment the <root> element line.

    <index id="Coveo_web_index">
      <locations hint="list:AddCrawler">
        <crawler name="ContentCrawler">
          <!-- Uncomment this node if you want to change the crawling root to a smaller part of your site. -->
          <root>/sitecore/content/YOUR_PATH</root>
        </crawler>
    
  5. Set the <root> element text content to your desired crawling root. Add a trailing slash to your path.

    <index id="Coveo_web_index">
      <locations hint="list:AddCrawler">
        <crawler name="ContentCrawler">
          <!-- Uncomment this node if you want to change the crawling root to a smaller part of your site. -->
          <root>/sitecore/content/Home/Ecommerce/</root>
        </crawler>
    
  6. Save and close your file.

  7. Test your new configuration.
    1. In the Command Center, in the Indexes section, expand the row associated with the targeted index.

    2. Click Rebuild.

    3. If your new crawler is properly configured, no error message appears and your browser developer tools show a successful rebuild API call.

    4. A few minutes after rebuild completion, the Command Center should display the updated number of indexed documents.

    5. You can also view your remaining indexed items on the Coveo Platform (see Inspect Items With the Content Browser).

Adding a New Crawling Root

You might also want to add a new crawler to an index and define its root.

To add a new crawler on an index

  1. Open your Coveo.SearchProvider.Custom.config file in a text editor. It’s located under [Sitecore instance root]\Website\App_config\Include\Coveo or [Sitecore instance root]\App_config\Include\Coveo (in Sitecore 9 and 10 instances).
  2. Locate the following code block.

    <contentSearch>
      <configuration>
        <indexes hint="list:AddIndex">
          <index id="Coveo_master_index">
            <locations hint="list:AddCrawler">
              <crawler name="ContentCrawler">
                <!-- Uncomment this node if you want to change the crawling root to a smaller part of your site. -->
                <!--<root>/sitecore/content/YOUR_PATH</root> -->
              </crawler>
              <!-- Uncomment this node if you want to crawl a different section of your site. -->
              <!--
              <crawler name="SubSiteCrawler" type="Sitecore.ContentSearch.SitecoreItemCrawler, Sitecore.ContentSearch">
                <database>master</database>
                <root>/sitecore/YOUR_PATH</root>
                <stopOnError>true</stopOnError>
              </crawler>
              -->
            </locations>
          </index>
          <index id="Coveo_web_index">
            <locations hint="list:AddCrawler">
              <crawler name="ContentCrawler">
                <!-- Uncomment this node if you want to change the crawling root to a smaller part of your site. -->
                <!--<root>/sitecore/content/YOUR_PATH</root> -->
              </crawler>
              <!-- Uncomment this node if you want to crawl a different section of your site. -->
              <!--
              <crawler name="SubSiteCrawler" type="Sitecore.ContentSearch.SitecoreItemCrawler, Sitecore.ContentSearch">
                <database>web</database>
                <root>/sitecore/YOUR_PATH</root>
                <stopOnError>true</stopOnError>
              </crawler>
              -->
            </locations>
          </index>
        </indexes>
      </configuration>
    </contentSearch>
    
  3. Locate the <index> element associated with the Sitecore database you want to target. For the sake of this example, we assume we want to add a new crawler in the Coveo_web_index index.

  4. As indicated in the configuration file, uncomment the <crawler name="SubSiteCrawler" ...> element.

    <index id="Coveo_web_index">
      <locations hint="list:AddCrawler">
        <crawler name="ContentCrawler">
          <!-- Uncomment this node if you want to change the crawling root to a smaller part of your site. -->
          <!--<root>/sitecore/content/YOUR_PATH</root> -->
        </crawler>
        <!-- Uncomment this node if you want to crawl a different section of your site. -->
        <crawler name="SubSiteCrawler" type="Sitecore.ContentSearch.SitecoreItemCrawler, Sitecore.ContentSearch">
          <database>web</database>
          <root>/sitecore/YOUR_PATH</root>
          <stopOnError>true</stopOnError>
        </crawler>
      </locations>
    
  5. Change the name of the crawler to one you deem appropriate.

    <index id="Coveo_web_index">
      <locations hint="list:AddCrawler">
        <crawler name="ContentCrawler">
          <!-- Uncomment this node if you want to change the crawling root to a smaller part of your site. -->
          <!--<root>/sitecore/content/YOUR_PATH</root> -->
        </crawler>
        <!-- Uncomment this node if you want to crawl a different section of your site. -->
        <crawler name="Staff" type="Sitecore.ContentSearch.SitecoreItemCrawler, Sitecore.ContentSearch">
          <database>web</database>
          <root>/sitecore/YOUR_PATH</root>
          <stopOnError>true</stopOnError>
        </crawler>
      </locations>
    

    You should limit the name of your crawler to alphanumeric characters.

  6. Set the <root> element text content to your desired crawling root. Add a trailing slash to your path.

    <index id="Coveo_web_index">
      <locations hint="list:AddCrawler">
        <crawler name="ContentCrawler">
          <!-- Uncomment this node if you want to change the crawling root to a smaller part of your site. -->
          <!--<root>/sitecore/content/YOUR_PATH</root> -->
        </crawler>
        <!-- Uncomment this node if you want to crawl a different section of your site. -->
        <crawler name="Staff" type="Sitecore.ContentSearch.SitecoreItemCrawler, Sitecore.ContentSearch">
          <database>web</database>
          <root>/sitecore/Content/Home/Staff/</root>
          <stopOnError>true</stopOnError>
        </crawler>
      </locations>
    
  7. Save and close your file.
  8. The Command Center now displays your new crawler.

  9. Test your new configuration.

    1. In the Command Center, in the Indexes section, click the Refresh button associated with your new crawler.

    2. If your new crawler is properly configured, no error message appears and your browser developer tools show a successful refresh API call.

Recommended Articles