Remove Bot Traffic in Usage Analytics Reports

When your goal is to focus solely on the users visiting and interacting with your search interfaces, you have to identify and remove bot traffic from your usage analytics reports.

If the META tag of your web pages allows robots to either index public web pages or follow the links they contain (i.e., <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">), you can consider that web robots actions are included in your reports. Moreover, when you index your sites using Coveo, the Coveobot is also producing human-like events in your analytics data (e.g., visiting pages and clicking links).

The Coveo’s Customer Success Team (CSM) made a filter that covers the most popular and active bots. Using this filter will remove most bot traffic from your reports.

To remove bot traffic in usage analytics reports

1. On the Named Filters page, add the following filter (see Create Named Filters):

   Browser does not contain WormlyBot, bingbot, Googlebot, YandexBot, SMTBot/1.0, Yandex.Browser, DuckDuckBot, Baiduspider or Coveobot

   When creating the filter above, you must add each value (split with commas) separately. Otherwise, the filter will be ineffective.

   

2. In the report you want to remove bot traffic, apply the named filter you just created (see Add Named Filters to Reports).

3. Using the filter as a stating point, consider improving its efficiency with the following guidelines:

   • Look for suspicious number of visits coming from a city or region in which your company or client base is not located, review the user agents of those visits, and consider excluding the most recurrent ones with another filter.

   • If there are a lot of visits on your search pages with unknown browsers, since it is possible to not pass the user agent used to identify the user’s browser, consider adding unknown to the filter values.