Reviewing Usage Analytics Metric Alerts

Administrators can use the Metric Alerts page to review the list of metric anomalies detected in your Coveo Cloud organization.

Metric alerts are currently in beta.

Metric Alerts

The Coveo Cloud platform actively monitors around 30 key metrics allowing you to review and assess the health of your Coveo solution. These metrics are automatically gathered and analyzed to detect any deviation from their usual trend, thus giving you an indication of whether something changed in the user behavior or a technical problem occurred.

An alert is generated when at least one monitored metric deviates from its expected path for at least three hours, so no alert is raised for temporary system outages (e.g., during 20 minutes the Platform stops receiving UA events from your application due to an update). Short outages where everything is getting immediately back to normal are assumed to be detected by other systems before the Platform. Metric alerts are created for persistent issues that you should investigate to ensure nothing is broken.

Metric Alerts Animation

When the deployment of a new version of the Coveo application results in sending all events five times, an alert appears on the Metric Alerts page.‌

  • Metric alerts are not enabled on Coveo Cloud organizations with Sandbox and Trial licenses.

  • Since raised alerts are specific to your application or deployment, alerts are not generated in case of a Coveo outage. Coveo outages are announced on https://status.cloud.coveo.com/ where you can follow incidents and their resolution. You can also request a root cause analysis (RCA) for each of them.

  • An alert is not about a slow trend change over time, an alert is really about something deviating from past behaviors (statistical deviation), where some metrics spike or fall outside of typical past trends. A metric can trigger an alert if its trend variation is either an increase or a decrease depending on the metric type.

    The Click Events Count Overall metric will raise an alert only if an abnormal decreasing trend occurs. Since, it can be expected that a portal experiences a higher number of visits than usual, an increasing trend is considered normal.

  • A metric alert can contain multiple anomalies since when something unusual happens, it often affects more than one metric. All the metrics in a state of inconsistency are bundled together to form one distinct alert.

  • An alert, which is computed hourly, lasts from the time the first metric anomaly was detected until four hours after the last metric anomaly was detected. If no new anomalies are found after those consecutive hours, the alert is considered over. If an anomaly is detected during the fifth hour, a new alert is created for this anomaly.

How Metric Alerts Work

Completely automated via machine learning using predictive analytics (algorithm from the forecasting family), the system behind the metric alerts learns from time series data, past patterns, by building a model from it. The system requires a couple of weeks of data before building a model that can generate alerts.

Behind Metric Alerts

In the chart above, the rounded dots are actual data points. The blue line over the dots represents the expected value from the built model. The gray band defines the high and low confidence of the model. As the number of events grows and the pattern stabilizes, the difference between the high and the low confidence narrows down. Lack of data and end-user behavior volatility may cause the band to expand.

Each new event generates a new dot over the time series. The system then uses the generated model to predict the next value. The system then compares if the new actual value is matching the next predicted value. If the actual value is within the high and low confidence of the predicted value, then everything is fine. As the actual value deviates outside of the confidence range, it becomes an anomaly. An anomaly becomes significant based on its level of deviation and the fact it repeats over time.

On the chart where the rounded dots stop and the line continues, the model forecasts the next values over time.

Access the “Metric Alerts” Page

  1. If not already done, log in to the Coveo Cloud platform as a member of a group with the required privileges.

  2. In the main menu on the left, under Analytics, select Metric Alerts.

Inspect Metric Alerts

On the Metric Alerts page, in the table, you can review the history of alerts within your organization analytics data that occurred in the last 365 days.

  • Click an alert row to see all available information on the alert such as a graphical representation of each metric anomaly in the alert and the alert ID. When an alert involves more than one metric, on top of the graph, click the drop-down menu, and then select the metric that you want to review. The area in which the anomaly is detected is shown in a color that changes depending on the severity level (yellow for low, orange for medium, and red for high) and the blue line indicates the expected value.

  • Use the value of the drop-down menus to filter alerts to the ones you want to review (see Filter Alerts By Visibility and Filter Alerts by Metric).

  • At the bottom-left of the table, click 20 or 30 to show more alerts on the page (default is 10).

  • At the bottom-right of the table, click the left and right arrow icons, or a page number to navigate through pages and see older or newer alerts.

Filter Alerts by Visibility

On the Metric Alerts page, in the right section of the Action bar, click the Visibility drop-down menu, and then select to show only the Active and Dismissed alerts in the table.

Filter Alerts by Metric

On the Metric Alerts page, in the right section of the Action bar, click the Metric drop-down menu, and then select to show only the alerts related to All or one of the following metrics in the table:

  • Anonymous Search % Overall
  • Average Click Rank Overall
  • Average Click Rank per Hub
  • Average Events per Visit per Hub
  • Average Number of Recommendations Overall
  • Average Number of Results Overall
  • Average Response Time Overall
  • Click Events Count Overall
  • Click Events per Hub
  • Click Events per Source
  • Click Through Ratio Overall
  • Click Through Ratio per Hub
  • Custom Events Count Overall
  • Custom Events per Hub
  • Distinct Queries Overall
  • Distinct Visits Overall
  • Empty Query % Overall
  • Events Count Overall
  • ML Boosted Docs % Overall
  • ML Boosted Docs % per Pipeline
  • Search Events Count Overall
  • Search Events per Hub
  • Search Events per Language
  • Search Events per Origin Level

  • Used QS % Overall
  • Used QS % per Hub
  • View Events Count Overall
  • View Events per Hub

Dismiss an Alert

On the Metric Alerts page, when an alert is false or when you solved the issue that caused it, click the desired alert, and then in the Action bar, click Dismiss.

After fixing the issue behind an alert, metric values will trend normally again and no more anomalies will be detected, thus ending the alert.

The alert disappears from the page table.

You can reactivate dismissed alerts (see Reactivate an Alert).

Reactivate an Alert

  1. On the Metric Alerts page, click the Visibility drop-down menu, and then select Dismissed.

  2. Click the alert that you dismissed by error, and then in the Action bar, click Reactivate.

Metric Anomaly Reference

The following table shows what is considered an anomaly () for each monitored metric:

Metric Value higher than expected Value lower than expected
Anonymous Search % Overall
Average Click Rank Overall
Average Click Rank per Hub
Average Events per Visit per Hub
Events Count Overall
Average Number of Recommendations Overall
Average Number of Results Overall
Average Response Time Overall
Click Events Count Overall
Click Events per Hub
Click Events per Source
Click Through Ratio Overall
Click Through Ratio per Hub
Custom Events Count Overall
Custom Events per Hub
Distinct Queries Overall
Distinct Visits Overall
Empty Query % Overall
ML Boosted Docs % Overall
ML Boosted Docs % per Pipeline
Search Events Count Overall
Search Events per Hub
Search Events per Language
Search Events per Origin Level
Used QS % Overall
Used QS % per Hub
View Events Count Overall
View Events per Hub
  • : Anomaly

  • : Good

Page Reference

The body of the Metric Alerts page is essentially a table listing all the alerts matching specific criteria.

  • Date column indicates the date when the alert was triggered.

  • Anomaly column indicates the number of metrics involved in the alert and whether those metric values were higher or lower than expected based on their usual trend.

  • Severity column indicates the level of severity of the alert which can be Low, Medium, or High.

  • The severity level of an alert is derived from its metric anomaly with the highest severity value. Only anomalies that are responsible for the raised alerts, which are of Medium and High severity, appear on the page.

  • For a given alert, if a metric anomaly becomes (or a new anomaly is) more important than any previously detected anomalies, the severity of the alert is updated accordingly.

  • The severity of an anomaly does not change over time, however, there can be a new anomaly (e.g., an hour later) for the same metric. An anomaly corresponds to one metric at one moment in time (always on an hourly basis).

  • Metrics column indicates the name of the metrics involves in the alert.

  • When expanding an alert row, more information regarding the alert is shown:

    • ID: The unique identifier of the alert.

    • Graph: When hovering a graph point (representing an hour or a day), you see the Recorded value and the Expected value for the involved metric.

Required Privileges

The following table indicates the privileges required to view elements of the Metric Alerts page (see Privilege Management and Privilege Reference).

Action Service - Privilege Required access level
View metric alerts

Analytics - Metric alerts

Organization - Organization

View
Dismiss and reactivate metric alerts

Analytics - Metric alerts

Edit