- Report Management
- Creating Dashboards
- Creating Explorers
- Shared Report Articles
- Reviewing Trends From the Summary Dashboard
- Reviewing Your Company Self-Service Usage
- Exporting Data From a Card
- Finding Search Implementation Successes and Issues With Metrics
- Remove Bot Traffic in Usage Analytics Reports
Remove Bot Traffic in Usage Analytics Reports
When your goal is to focus solely on the users visiting and interacting with your search interfaces, you have to identify and remove bot traffic from your usage analytics reports.
If the META tag of your web pages allows robots to either index public web pages or follow the links they contain (i.e.,
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">), you can consider that web robots actions are included in your reports. Moreover, when you index your sites using Coveo, the
Coveobot is also producing human-like events in your analytics data (e.g., visiting pages and clicking links).
The Coveo’s Customer Success Team (CSM) made a filter that covers the most popular and active bots. Using this filter will remove most bot traffic from your reports.
Other robots can visit your websites for testing, diagnostic, and monitoring purposes.
To remove bot traffic in usage analytics reports
Browser does not contain WormlyBot, bingbot, Googlebot, YandexBot, SMTBot/1.0, Yandex.Browser, DuckDuckBot, Baiduspider or Coveobot
When creating the filter above, you must add each value (split with commas) separately. Otherwise, the filter will be ineffective.
In the report you want to remove bot traffic, apply the named filter you just created (see Add Named Filters to Reports).
Using the filter as a stating point, consider improving its efficiency with the following guidelines:
Look for suspicious number of visits coming from a city or region in which your company or client base is not located, review the user agents of those visits, and consider excluding the most recurrent ones with another filter.
If there are a lot of visits on your search pages with unknown browsers, since it is possible to not pass the user agent used to identify the user’s browser, consider adding
unknownto the filter values.