Analytics - Section
- Nomenclature Change for Queries
- Use Cases
- Getting Started
- Reviewing User Visits With the Visit Browser
- Retrieving End-Users Information Pushed to Coveo Usage Analytics
- Leveraging Usage Analytics
- Data Export Management
- Dimension Management
- Named Filter Management
- Permission Filter Management
- Metric Alerts Review
- Incoherent Events Review
- Setting the Period to Review Search Usage Data
- Adding Global Dimension Filters
- Logging Coveo Events From Google Tag Manager
- About UA Service Query Suggestions
- About Usage Analytics Limits
Reviewing Usage Analytics Metric Alerts
Metric alerts are currently in beta.
The Coveo Cloud platform actively monitors around 30 key metrics allowing you to review and assess the health of your Coveo solution. These metrics are automatically gathered and analyzed to detect any deviation from their usual trend, thus giving you an indication of whether something changed in the user behavior or a technical problem occurred.
An alert is generated when at least one monitored metric deviates from its expected path for at least three hours, so no alert is raised for temporary system outages (e.g., during 20 minutes the Platform stops receiving UA events from your application due to an update). Short outages where everything is getting immediately back to normal are assumed to be detected by other systems before the Platform. Metric alerts are created for persistent issues that you should investigate to ensure nothing is broken.
When the deployment of a new version of the Coveo application results in sending all events five times, an alert appears on the Metric Alerts page.
Metric alerts are not enabled on Coveo Cloud organizations with Sandbox and Trial licenses.
Since raised alerts are specific to your application or deployment, alerts are not generated in case of a Coveo outage. Coveo outages are announced on https://status.cloud.coveo.com/ where you can follow incidents and their resolution. You can also request a root cause analysis (RCA) for each of them.
An alert is not about a slow trend change over time, an alert is really about something deviating from past behaviors (statistical deviation), where some metrics spike or fall outside of typical past trends. A metric can trigger an alert if its trend variation is either an increase or a decrease depending on the metric type.
The Click Events Count Overall metric will raise an alert only if an abnormal decreasing trend occurs. Since, it can be expected that a portal experiences a higher number of visits than usual, an increasing trend is considered normal.
A metric alert can contain many anomalies since when something unusual happens, it often affects more than one metric. All the metrics in a state of inconsistency are bundled together to form one distinct alert.
An alert, which is computed hourly, lasts from the time the first metric anomaly was detected until four hours after the last metric anomaly was detected. If no new anomalies are found after those consecutive hours, the alert is considered over. If an anomaly is detected during the fifth hour, a new alert is created for this anomaly.
How Metric Alerts Work
Completely automated via machine learning using predictive analytics (algorithm from the forecasting family), the system behind the metric alerts learns from time series data, past patterns, by building a model from it. The system requires a couple of weeks of data before building a model that can generate alerts.
In the chart above, the rounded dots are actual data points. The blue line over the dots represents the expected value from the built model. The gray band defines the high and low confidence of the model. As the number of events grows and the pattern stabilizes, the difference between the high and the low confidence narrows down. Lack of data and end-user behavior volatility may cause the band to expand.
Each new event generates a new dot over the time series. The system then uses the generated model to predict the next value. The system then compares if the new actual value is matching the next predicted value. If the actual value is within the high and low confidence of the predicted value, then everything is fine. As the actual value deviates outside of the confidence range, it becomes an anomaly. An anomaly becomes significant based on its level of deviation and the fact it repeats over time.
On the chart where the rounded dots stop and the line continues, the model forecasts the next values over time.
Inspect Metric Alerts
Click an alert row to see all available information on the alert such as a graphical representation of each metric anomaly in the alert and the alert ID. When an alert involves more than one metric, on top of the graph, click the drop-down menu, and then select the metric that you want to review. The area in which the anomaly is detected is shown in a color that changes depending on the severity level (yellow for low, orange for medium, and red for high) and the blue line indicates the expected value.
Use the value of the drop-down menus to filter alerts to the ones you want to review.
At the bottom-left of the table, click 20 or 30 to show more alerts on the page (default is 10).
At the bottom-right of the table, click the left and right arrow icons, or a page number to navigate through pages and see older or newer alerts.
Dismiss an Alert
On the Metric Alerts page, when an alert is false or when you solved the issue that caused it, click the desired alert, and then in the Action bar, click Dismiss.
After fixing the issue behind an alert, metric values will trend normally again and no more anomalies will be detected, thus ending the alert.
The alert disappears from the page table.
You can reactivate dismissed alerts (see Reactivate an Alert).
Reactivate an Alert
On the Metric Alerts page, click the Visibility drop-down menu, and then select Dismissed.
Click the alert that you dismissed by error, and then in the Action bar, click Reactivate.
Metric Anomaly Reference
The following table shows what is considered an anomaly () for each monitored metric:
|Metric||Value higher than expected||Value lower than expected|
|Anonymous Search % Overall|
|Average Click Rank Overall|
|Average Response Time Overall|
|Click Events Count Overall|
|Click Through Ratio Overall|
|Content Gap Ratio Overall|
|Custom Events Count Overall|
|Distinct Queries Overall
Distinct Visits Overall
|Empty Query % Overall|
|Events Count Overall|
|ML Boosted Docs % Overall|
|Search Events Count Overall|
|Used QS % Overall|
|View Events Count Overall|
|Visit Click Through Ratio Overall|
The body of the Metric Alerts page is essentially a table listing all the alerts matching specific criteria.
Date column indicates the date when the alert was triggered.
Anomaly column indicates the number of metrics involved in the alert and whether those metric values were higher or lower than expected based on their usual trend.
Severity column indicates the level of severity of the alert which can be Low, Medium, or High.
The severity level of an alert is derived from its metric anomaly with the highest severity value. Only anomalies that are responsible for the raised alerts, which are of Medium and High severity, appear on the page.
For a given alert, if a metric anomaly becomes (or a new anomaly is) more important than any anomalies detected before, the severity of the alert is updated accordingly.
The severity of an anomaly does not change over time, however, there can be a new anomaly (e.g., an hour later) for the same metric. An anomaly corresponds to one metric at one moment in time (always on an hourly basis).
Metrics column indicates the name of the metrics involves in the alert.
When expanding an alert row, more information regarding the alert is shown:
ID: The unique identifier of the alert.
Graph: When hovering a graph point (representing an hour or a day), you see the Recorded value and the Expected value for the involved metric.
|Action||Service - Domain||Required access level|
|View metric alerts||
Analytics - Metric alerts
|Dismiss and reactivate metric alerts||
Analytics - Metric alerts