Exclude search interfaces from external search engines

This is for:

Developer

You may want to exclude Coveo-powered search interfaces from external search engines (SEs).

External SEs make requests to the pages they index, increasing traffic on these pages. In the context of Coveo-powered search interfaces, this can be problematic as it impacts the query per month (QPM) count for a given Coveo organization.

If you don’t want some Coveo-powered search interfaces to be indexed by external SEs, you can implement the following mechanisms:

robots.txt files

External SE crawlers look for, download, and parse a website’s robots.txt file to know which URLs the crawlers can access before beginning the indexing process. robots.txt files can also be used to direct crawlers to your sitemaps.

Note

This approach is used to avoid overloading your site with requests and isn’t a mechanism to avoid keeping your pages out of external SEs.

For more information on robots.txt files, and on how to configure them, see Introduction to robots.txt.

noindex directive

Coveo-powered search interfaces are excluded from external SE search results when they include a noindex HTML meta tag or when they include a noindex header in their HTTP response.

For more information on the noindex directive, and on how to configure it, see Block Search indexing with 'noindex'.