Exclude Search Interfaces From External Search Engines
You may want to exclude Coveo-powered search interfaces from external search engines (SEs).
External SEs make requests to the pages they index, increasing traffic on these pages. In the context of Coveo-powered search interfaces, this can be problematic as it impacts the query per month (QPM) count for a given Coveo organization.
If you don’t want some Coveo-powered search interfaces to be indexed by external SEs, you can implement the following mechanisms:
External SE crawlers look for, download, and parse a website’s
robots.txt file to know which URLs the crawlers can access before beginning the indexing process.
robots.txt files can also be used to direct crawlers to your sitemaps.
This approach is used to avoid overloading your site with requests and isn’t a mechanism to avoid keeping your pages out of external SEs.
For more information on
robots.txt files, and on how to configure them, see Introduction to robots.txt.
Coveo-powered search interfaces are excluded from external SE search results when they include a
noindex HTML meta tag or when they include a
noindex header in their HTTP response.
For more information on the
noindex directive, and on how to configure it, see Block Search indexing with 'noindex'.