alerting rules

28 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Forecast alert use cases: capacity planning and business metrics

Forecast-based alerts have two main use cases. Capacity planning detects when resources such as disk space, memory, CPU, or database connections are expected to approach their limits. Demand and business planning analyzes expected demand, costs, production, or other business metrics.

Forecast-based alerts compare predicted future values with threshold conditions

Standard alert rules evaluate the current state of a signal. Forecast-based alerts compare a predicted future value with a threshold condition, allowing action before a threshold is breached.

Linear forecasts for steady trends, seasonal forecasts for recurring patterns

Linear forecasts are used for signals that follow a relatively steady trend. Seasonal forecasts are better suited to signals with recurring patterns, such as daily or weekly traffic.

When to use predict_linear: steady trends, not recurring patterns

predict_linear is useful when a signal follows a relatively steady trend, such as disk utilization that is consistently increasing. It is not suitable for signals with strong recurring patterns. For signals that follow daily or weekly cycles, a straight-line projection might produce values that do not reflect the signal's normal behavior. predict_linear assumes the current trend continues, so a sudden signal change can make the extrapolated trend inaccurate.

Alert on future predicted value example with predict_linear

To alert on a future predicted value, compare the predicted value with a threshold directly: predict_linear(<metric>[14d], 7 * 24 * 3600) > 85. This evaluates whether the predicted value seven days from now exceeds 85.

PostgreSQL disk utilization forecast alert example

Example alert rule for forecasting disk utilization on a PostgreSQL persistent volume: predict_linear(max by (persistentvolumeclaim) (100 * (1 - kubelet_volume_stats_available_bytes{persistentvolumeclaim="<id>"} / kubelet_volume_stats_capacity_bytes{persistentvolumeclaim="<id>"})) [14d:1h], 7 * 24 * 3600). This calculates disk utilization percentage, uses the previous 14 days to predict 7 days into the future, evaluates historical data at 1-hour intervals, and uses a threshold that fires when predicted utilization exceeds 85%. The rule evaluates every 12 hours.

Grafana Machine Learning forecast produces multiple metrics including bounds

Unlike predict_linear which returns a single value, a Grafana Machine Learning forecast produces a series of predicted values over the forecast period with upper and lower bounds. Each forecast produces these Prometheus metrics: <forecast_metric_name>:predicted{ml_forecast="yhat"} for the predicted value, <forecast_metric_name>:predicted{ml_forecast="yhat_upper"} for the upper bound, <forecast_metric_name>:predicted{ml_forecast="yhat_lower"} for the lower bound, and <forecast_metric_name>:actual for the current signal value.

Two types of forecast-based alerts: threshold breach and deviation from range

Forecast-based alerts can detect two scenarios: when the forecast is expected to breach a threshold at any point in the future, or when the current signal deviates from the expected forecast range.

Use forecast upper bound for conservative capacity planning alerts

In capacity planning forecast alerts, use the upper forecast bound (yhat_upper) rather than the central prediction to account for prediction uncertainty and provide a more conservative capacity-planning signal.

Forecast as adaptive baseline compared to fixed threshold

A forecast acts as an adaptive baseline. While a standard alert rule evaluates a signal against a fixed threshold, a forecast-based alert rule evaluates the signal against the forecast, which can change over time based on seasonal patterns.

Anomalous metric for detecting values outside forecast range

Each forecast produces an anomalous metric (<forecast_metric_name>:anomalous) that returns 0 when the actual value is within the prediction interval. The metric returns 1 when the actual value is above the upper bound and -1 when below the lower bound.

Alert on anomalous values outside forecast range example

To fire when the actual value is outside the prediction interval, use: <forecast_metric_name>:anomalous. To alert only on unexpectedly high values exceeding the upper bound, use: <forecast_metric_name>:anomalous == 1.

Compare actual value directly with predicted upper bound

An alternative to using the anomalous metric is to compare the actual value directly with the predicted upper bound: <forecast_metric_name>:actual > ignoring (ml_forecast) <forecast_metric_name>:predicted{ml_forecast="yhat_upper"}.

When to use adaptive alerts with forecast deviations

Adaptive alerts based on forecast deviations are useful when a signal varies over time and a fixed threshold would miss meaningful deviations or generate alerts during normal changes in behavior. For example, traffic might be higher during business hours and lower overnight, so comparing the current value with the forecast range adapts to these recurring patterns.

How to create an Elasticsearch alert rule

To create an alert rule using Elasticsearch: 1) Navigate to Alerting > Alert rules. 2) Click New alert rule. 3) Enter a name for the alert rule. 4) Select your Elasticsearch data source. 5) Build your query using the query editor by adding metric aggregations (for example, Average, Count, Sum), adding a Date histogram under Group by, and optionally adding filters using Lucene query syntax. 6) Configure the alert condition (for example, when the average is above a threshold). 7) Set the evaluation interval and pending period. 8) Configure notifications and labels. 9) Click Save rule.

Elasticsearch alert: high error count example

Monitor the number of error-level log entries with: Query set to level:error, Metric set to Count, Group by set to Date histogram with interval 1m, and Condition set to when count is above 100.

Elasticsearch alert: average response time example

Monitor API response times with: Query set to type:api_request, Metric set to Average on field response_time, Group by set to Date histogram with interval 5m, and Condition set to when average is above 500 milliseconds.

Elasticsearch alert: unique user count drop example

Detect drops in active users with: Query set to * (all documents), Metric set to Unique count on field user_id, Group by set to Date histogram with interval 1h, and Condition set to when unique count is below 100.

MSSQL alert rule creation steps

To create an alert rule using Microsoft SQL Server: 1) Navigate to Alerting > Alert rules. 2) Click New alert rule. 3) Enter a name for the alert rule. 4) Select the Microsoft SQL Server data source. 5) Build the query using the query editor: set Format to Time series, include a time column using $__timeGroup() or $__timeGroupAlias() macro, add numeric columns for values to monitor, use $__timeFilter() to filter data by evaluation time range. 6) Configure the alert condition. 7) Set the evaluation interval and pending period. 8) Configure notifications and labels. 9) Click Save rule.

InfluxDB alert rule creation steps

To create an alert rule: (1) Navigate to Alerting > Alert rules, (2) Click New alert rule, (3) Enter a name for the alert rule, (4) Select InfluxDB data source, (5) Build query using query editor, (6) Configure alert condition (e.g., when average above threshold), (7) Set evaluation interval and pending period, (8) Configure notifications and labels, (9) Click Save rule.

Alert queries cannot use template variables

Alert queries cannot use template variables such as $hostname or $region because Grafana evaluates alert rules on the backend without dashboard context. Replace template variables with hard-coded values in alert queries. Maintain two separate queries if needed: one with variables for the dashboard and one with hard-coded values for alerting.

Alert evaluation data requirements

Alert evaluation requires time-series data that Grafana can evaluate. For InfluxQL, use an aggregation function (such as mean, sum, count) with GROUP BY time($__interval). For Flux, use aggregateWindow() to produce time-bucketed results. For SQL, use $__dateBin(time) or $__timeGroup(time) to aggregate by time.

InfluxQL GROUP BY time for alert evaluation

InfluxQL alert queries must use an aggregation function (such as mean, sum, count) with GROUP BY time($__interval) to produce time-series data that Grafana can evaluate. Without this, the alert evaluation will return no data.

Transient alert errors triggering false alarms

The error messages 'sse.dependencyError', 'sse.dataQueryError', 'context deadline exceeded', or 'i/o timeout' with alert rules intermittently firing due to execution errors rather than genuine threshold breaches occurs because by default when an alert rule encounters an execution error or timeout, Grafana sets alert state to Alerting which fires the alert. Transient connectivity issues between Grafana and Prometheus (i/o timeouts, deadline exceeded, brief outages) trigger this behavior even though the underlying metric hasn't crossed its threshold.

Transient alert errors solution - Alert state if execution error

To fix transient alert errors: open each affected alert rule for editing, in the alert conditions section change Alert state if execution error or timeout from Alerting to Keep Last State, and save the rule. This ensures the alert retains its previous state during transient errors and only fires when a successful evaluation confirms the threshold is breached.

Transient alert errors investigation steps

If errors are frequent, also investigate: network stability between Grafana and Prometheus, Prometheus resource utilization (CPU, memory, disk I/O), the Query timeout setting in data source configuration (increase it if complex queries regularly exceed the limit), and query complexity (simplify queries or use recording rules to pre-compute expensive expressions).

Alert rule fails to evaluate causes and solutions

When an alert rule using a Prometheus query shows evaluation errors or remains in 'No Data' state: if template variables in query, alert queries don't support template variables so replace with hard-coded values; if query timeout, simplify the query or increase evaluation timeout and use recording rules for complex expressions; if data source unreachable, verify Prometheus data source connection is working by testing it in data source settings; if no data in range, ensure the metric has recent data and check that Prometheus is actively scraping the target.

Data source-managed rules not visible troubleshooting

When Prometheus alerting rules don't appear in Grafana Alerting UI: verify that Manage alerts via Alerting UI is enabled in data source configuration, check that Prometheus has alerting rules configured in its rule files, ensure Grafana can access the Prometheus rules API endpoint (/api/v1/rules), and note that for Prometheus (unlike Mimir) the Alerting UI only supports viewing existing rules, not creating new ones.

Give your agent this brain