Alerts are notifications from AIOps monitoring tools that indicate that there is an anomaly. IT teams get these alerts on their monitoring dashboard via emails or enterprise collaboration tools such as Slack or Teams. Service level agreements expect IT teams to analyze every alert within a specific timeframe and take appropriate action.

As the technology landscape grows and monitoring tools become more complex, the number of alerts also rises exponentially. IDC finds that 37% of IT organizations now face 10,000 alerts or more per month. Moreover, a vast majority of professionals admit that they cannot handle the “near-constant barrage of alerts” or what’s now being called alert storms. It is, therefore, not uncommon for SREs and ITOps personnel to filter monitoring alerts and notifications into the junk folder or, at the very least, ignore them.

Alert storms affect the performance of IT teams in three specific ways:

1. Most of the alerts tend to be false alarms. When IT teams pay the same level of attention to every alert, they are overwhelmed, and their productivity is affected.
2. While addressing alerts in chronological order, they might miss out on more severe problems, as they are involved in analyzing a false positive.
3. Without a clear structure to predict and prevent concerns, they continue to manually resolve issues every time they occur, resulting in astronomical resolution times.

Given the rapid digitization of businesses globally, IT teams need a proactive and preventative approach to monitoring and alerts.

How to Clear Up Alert Storms?

#1 Do not treat all anomalies as issues

Although monitoring systems may throw a lot of alerts, not all of them are real issues. In reality, only about 5-10% of the anomalies may be worth investigating. One way to reduce the volume of alerts is to have an anomaly score based on the severity of the anomaly. Then, you can set up an SLA to prioritize those alerts that cross a severity threshold.

#2 Perform temporal and topological alert grouping

To further reduce alerts, you can use temporal correlation or correlation based on time. That is, if the same issue causes multiple alerts during a specified time interval, consolidate all the alerts into one. For instance, if a network issue brings down the CRM, HRMS, and the website simultaneously, you should get one alert, not four.

You can also correlate alerts based on topology. If multiple upstream services connect to a downstream service, and the downstream service goes down, all upstream services will start alerting. Using the knowledge of topology, we can correlate all these alerts into a single one. For instance, if net banking, check clearance, etc., all connect to a database service, which goes down, we should get one consolidated alert.

#3 Correlate outages to workloads

Anomalies in application metrics are often the result of a business event. Therefore, in addition to the univariate and multivariate outlier detection, it is imperative to also gather workload data and perform a thorough workload-behavior correlation.

In one of our client engagements, we leveraged HEAL’s proprietary workload-behavior correlation, adaptive learning algorithms, causal dependency graphs, inference techniques, and topology to identify the 26,000 relevant anomalies from over 200,000+ outliers — reducing alert storms by over 87%. This also reduces false positives to 1% or less while keeping precision above 90%, an industry-leading standard.

#3 Set up systems to minimize false negatives

Even if you’re able to suppress a vast majority of irrelevant alerts, there might still be the problem of missed alerts. To reduce this, you can use a robust continuous machine learning pipeline. Choose a machine learning engine (MLE) that learns from previous incidents and optimizes its ability to identify the anomalies resulting in issues.

#4 Reduce alerts over time by identifying future choke points

The final step is to leverage all these learnings towards better planning for the future. While the future chokepoint analysis might not reduce alerts in the present, it will play a significant role in optimizing alerts in the future.

HEAL’s ensemble-based regression models offer the opportunity to forecast accurately, with the capability to capture non-linear relationships as well. Unlike most machine learning solutions, which are great only at interpolation, these models can extrapolate to hypothetical workloads and minimize projection errors. Using these techniques on an already identified set of significant metrics, you can predict potential problems well in advance, sometimes 2-3 months in advance. Armed with this information, you can provision more accurately to prevent outages.

The most significant advantages of leveraging machine learning to address alert storms are:

• It dramatically reduces mean time to resolve (MTTR), ensuring higher uptime for your applications
• It saves manual effort spent in investigating a large number of alerts
• It prevents your teams from missing important alerts
• It also optimizes the anomaly detection, alerting, and automation mechanisms for better performance in the future

To see how HEAL’s preventing healing paradigm can transform your business, speak to a consultant today.