Reviewing Usage Analytics Metric Alerts

Members of the Administrators built-in group can use the Metric Alerts page to review the list of metric anomalies detected in your Coveo Cloud organization.

Metric alerts are currently in beta.

Metric Alerts

The Coveo Cloud Platform actively monitors around 30 key metrics allowing you to review and assess the health of your Coveo solution. These metrics are automatically gathered and analyzed to detect any deviation from their usual trend, thus giving you an indication of whether something changed in the user behavior or a technical problem occurred.

An alert is generated when at least one monitored metric deviates from its expected path for at least three hours, so no alert is raised for temporary system outages (e.g., during 20 minutes the Platform stops receiving UA events from your application due to an update). Short outages where everything is getting immediately back to normal are assumed to be detected by other systems before the Platform. Metric alerts are created for persistent issues that you should investigate to ensure nothing is broken.

Metric Alerts Animation

When the deployment of a new version of the Coveo application results in sending all events five times, an alert appears on the Metric Alerts page.‌

  • Metric alerts aren’t enabled on Coveo Cloud organizations with Sandbox and Trial licenses.

  • Since raised alerts are specific to your application or deployment, alerts aren’t generated in case of a Coveo outage. Coveo outages are announced on https://status.cloud.coveo.com/ where you can follow incidents and their resolution. You can also request a root cause analysis (RCA) for each of them.

  • An alert isn’t about a slow trend change over time, an alert is really about something deviating from past behaviors (statistical deviation), where some metrics spike or fall outside of typical past trends. A metric can trigger an alert if its trend variation is either an increase or a decrease depending on the metric type.

    The Click Event Count Overall metric will raise an alert only if an abnormal decreasing trend occurs. Since, it can be expected that a portal experiences a higher number of visits than usual, an increasing trend is considered normal.

  • A metric alert can contain many anomalies since when something unusual happens, it often affects more than one metric. All the metrics in a state of inconsistency are bundled together to form one distinct alert.

  • An alert, which is computed hourly, lasts from the time the first metric anomaly was detected until four hours after the last metric anomaly was detected. If no new anomalies are found after those consecutive hours, the alert is considered over. If an anomaly is detected during the fifth hour, a new alert is created for this anomaly.

How Metric Alerts Work

Completely automated via machine learning using predictive analytics (algorithm from the forecasting family), the system behind the metric alerts learns from time series data, past patterns, by building a model from it. The system requires a couple of weeks of data before building a model that can generate alerts.

Behind Metric Alerts

In the chart above, the rounded dots are actual data points. The blue line over the dots represents the expected value from the built model. The gray band defines the high and low confidence of the model. As the number of events grows and the pattern stabilizes, the difference between the high and the low confidence narrows down. Lack of data and end-user behavior volatility may cause the band to expand.

Each new event generates a new dot over the time series. The system then uses the generated model to predict the next value. The system then compares if the new actual value is matching the next predicted value. If the actual value is within the high and low confidence of the predicted value, then everything is fine. As the actual value deviates outside of the confidence range, it becomes an anomaly. An anomaly becomes significant based on its level of deviation and the fact it repeats over time.

On the chart where the rounded dots stop and the line continues, the model forecasts the next values over time.

Inspect Metric Alerts

On the Metric Alerts page, in the table, you can review the history of alerts within your organization analytics data that occurred in the last 365 days.

  • Click an alert row to see all available information on the alert such as a graphical representation of each metric anomaly in the alert and the alert ID. When an alert involves more than one metric, on top of the graph, click the drop-down menu, and then select the metric that you want to review. The area in which the anomaly is detected is shown in a color that changes depending on the severity level (yellow for low, orange for medium, and red for high) and the blue line indicates the expected value.

  • Use the value of the drop-down menus to filter alerts to the ones you want to review.

  • At the bottom-left of the table, click 20 or 30 to show more alerts on the page (default is 10).

  • At the bottom-right of the table, click the left and right arrow icons, or a page number to navigate through pages and see older or newer alerts.

Dismiss an Alert

On the Metric Alerts page, when an alert is false or when you solved the issue that caused it, click the desired alert, and then in the Action bar, click Dismiss.

After fixing the issue behind an alert, metric values will trend normally again and no more anomalies will be detected, thus ending the alert.

The alert disappears from the page table.

You can reactivate dismissed alerts (see Reactivate an Alert).

Reactivate an Alert

  1. On the Metric Alerts page, click the Visibility drop-down menu, and then select Dismissed.

  2. Click the alert that you dismissed by error, and then in the Action bar, click Reactivate.

Metric Anomaly Reference

The following table shows what’s considered an anomaly () for each monitored metric:

Metric Value higher than expected Value lower than expected
Anonymous Search % Overall
Average Click Rank Overall
Average Response Time Overall
Click Event Count Overall
Click Through Ratio Overall
Content Gap Ratio Overall
Custom Event Count Overall
Distinct Queries Overall
Distinct Visits Overall
Empty Query % Overall
Event Count Overall
ML Boosted Docs % Overall
Search Event Count Overall
Used QS % Overall
View Event Count Overall
Visit Click Through Ratio Overall
  • : Anomaly

  • : Good

Page Reference

The body of the Metric Alerts page is essentially a table listing all the alerts matching specific criteria.

  • Date column indicates the date when the alert was triggered.

  • Anomaly column indicates the number of metrics involved in the alert and whether those metric values were higher or lower than expected based on their usual trend.

  • Severity column indicates the level of severity of the alert which can be Low, Medium, or High.

  • The severity level of an alert is derived from its metric anomaly with the highest severity value. Only anomalies that are responsible for the raised alerts, which are of Medium and High severity, appear on the page.

  • For a given alert, if a metric anomaly becomes (or a new anomaly is) more important than any anomalies detected before, the severity of the alert is updated accordingly.

  • The severity of an anomaly doesn’t change over time, however, there can be a new anomaly (e.g., an hour later) for the same metric. An anomaly corresponds to one metric at one moment in time (always on an hourly basis).

  • Metrics column indicates the name of the metrics involves in the alert.

  • When expanding an alert row, more information regarding the alert is shown:

    • ID: The unique identifier of the alert.

    • Graph: When hovering a graph point (representing an hour or a day), you see the Recorded value and the Expected value for the involved metric.

Required Privileges

The following table indicates the privileges required to view elements of the Metric Alerts page (see Privilege Management and Privilege Reference).

Action Service - Domain Required access level
View metric alerts

Analytics - Metric alerts

View
Dismiss and reactivate metric alerts

Analytics - Metric alerts

Edit
Recommended Articles