alerting

260 notes in this subject, read out of this brain and free to use. This is page 4 of 5.

Verify Knowledge Graph notification alerts configuration

After applying Terraform configuration, verify that notification alerts configurations are created in the Knowledge Graph instance. Configurations should appear in the Knowledge Graph UI under Observability > Rules > Notify. Match labels should correctly filter the intended alerts. Custom labels should be properly applied to generated alerts.

Knowledge Graph notification alerts label best practices

Use specific and meaningful labels in match_labels to ensure precise alert filtering. Leverage existing label conventions from your monitoring setup. Consider using asserts_env and asserts_site labels for multi-environment setups.

Knowledge Graph notification alerts silencing best practices

Use the silenced parameter for temporary suppression rather than deleting notification alerts configurations. Document the reason for silencing in your Terraform configuration comments. Regularly review silenced configurations to ensure they are still needed.

Knowledge Graph notification alerts duration best practices

Set appropriate duration values based on your alerting requirements. Consider the nature of the monitored condition when choosing duration. Use consistent duration formats across similar alert types.

Forward slashes in folder names cause alert routing issues with subfolders

When upgrading to enable subfolders, folders with forward slashes in their names can cause alert routing issues. If an alert rule is in a folder with a forward slash in its name and the notification policy matches that folder's name, notifications will be sent to the default receiver instead of the configured receiver. Before upgrade, create copies of affected routes and rewrite matchers by escaping the forward slash, for example `grafana_folder=MyFolder/sub-folder`.

Legacy alerting entirely removed in v11

Legacy alerting has reached end-of-life in Grafana v11. Grafana will fail to start if legacy alerting settings are not updated to run the new Grafana Alerting system. It is no longer possible to migrate from legacy alerting to new alerting in v11. Grafana v10.4.x is the last version that offers migration.

Azure Monitor alerting and recording rules

All Azure Monitor query types (Metrics, Logs, Azure Resource Graph, and Traces) can be used with Grafana Alerting and recording rules. For detailed information about creating alert rules, supported query types, authentication requirements, and examples, refer to the Azure Monitor alerting documentation.

Elasticsearch alerting overview

Grafana Alerting with Elasticsearch allows you to create alerts based on Elasticsearch data to monitor metrics, detect anomalies, and receive notifications when specific conditions are met.

Elasticsearch alerting query types support

Query type compatibility for Elasticsearch alerting: Metrics with Date histogram has full support (recommended); Metrics without Date histogram has limited support and may not evaluate correctly over time; Logs, Raw data, and Raw document (deprecated since v10.1) are not supported. Use metrics queries instead for alerting.

Creating valid Elasticsearch alert queries

To create a valid Elasticsearch alert query, use a Date histogram as the last bucket aggregation under Group by, and select appropriate metric aggregations such as Count, Average, Sum, Min, or Max. Queries that return time series data allow Grafana to evaluate values over time and trigger alerts when thresholds are crossed.

Elasticsearch alert rule creation steps

To create an Elasticsearch alert rule: navigate to Alerting > Alert rules; click New alert rule; enter a name; select the Elasticsearch data source; build the query by adding metric aggregations, a Date histogram under Group by, and optionally Lucene query filters; configure the alert condition; set evaluation interval and pending period; configure notifications and labels; click Save rule.

Elasticsearch alert example: high error count

To alert on high error count: use query `level:error`; select Count metric; group by Date histogram with 1 minute interval; set condition to trigger when count is above 100.

Elasticsearch alert example: average response time

To alert on average response time: use query `type:api_request`; select Average metric on field `response_time`; group by Date histogram with 5 minute interval; set condition to trigger when average is above 500 milliseconds.

Elasticsearch alert example: unique user count drop

To alert on unique user count drop: use query `*` (all documents); select Unique count metric on field `user_id`; group by Date histogram with 1 hour interval; set condition to trigger when unique count is below 100.

Template variables not supported in Elasticsearch alerts

Alert queries cannot contain template variables because Grafana evaluates alert rules on the backend without dashboard context. Variables like $hostname or $environment will not be resolved. If your dashboard query uses template variables, create a separate query for alerting with hard-coded values.

Elasticsearch alerting query complexity limitations

Complex queries with many nested aggregations may timeout or fail to evaluate. Simplify Elasticsearch alert queries by reducing the number of bucket aggregations, using appropriate time intervals, and adding filters to limit the data scanned.

Logs queries not supported in Elasticsearch alerting

Queries using the Logs metric type cannot be used for Elasticsearch alerting. Convert your query to use metric aggregations with a Date histogram instead.

Elasticsearch alerting best practices

When creating Elasticsearch alerts, use specific Lucene query filters to focus on relevant data and improve performance; match the Date histogram interval to your evaluation frequency; verify queries return expected results in Explore before creating alerts; set thresholds based on historical data patterns; use descriptive names that indicate what the alert monitors.

Elasticsearch alerting prerequisites

Before creating alerts with Elasticsearch, you need an Elasticsearch data source configured in Grafana, appropriate permissions to create alert rules, and understanding of the metrics you want to monitor.

InfluxDB alerting supported query languages

InfluxDB alerting in Grafana supports all three query languages: InfluxQL, SQL, and Flux. Queries must return time-series data for Grafana to evaluate values over time and trigger alerts when thresholds are crossed.

InfluxQL alerting query requirements

InfluxQL queries for alerting must use aggregation functions with GROUP BY time clause. Example: SELECT mean("usage_system") FROM "cpu" WHERE $timeFilter GROUP BY time($__interval) fill(null)

Flux alerting query requirements

Flux queries for alerting must use the aggregateWindow() function for time-based aggregation to properly evaluate values over time.

SQL alerting query requirements

SQL queries for alerting must use $__timeFilter and $__dateBin macros for time filtering and time-based binning to properly aggregate data.

Flux memory pressure alert example

This example shows how to create an alert for memory usage with Flux: from(bucket: "telegraf") |> range(start: v.timeRangeStart, stop: v.timeRangeStop) |> filter(fn: (r) => r["_measurement"] == "mem") |> filter(fn: (r) => r["_field"] == "used_percent") |> aggregateWindow(every: v.windowPeriod, fn: mean) |> yield(name: "mean") Set condition to alert when mean value exceeds 85.

SQL disk usage alert example

This example shows how to create an alert for disk usage in SQL: SELECT $__dateBin(time), mean(used_percent) FROM disk WHERE $__timeFilter(time) GROUP BY $__dateBin(time) Set condition to alert when mean value exceeds 80.

InfluxDB alerting does not support template variables

Alert queries cannot contain template variables like $hostname or $region. Grafana evaluates alert rules on the backend without dashboard context, so variables are not resolved. If a dashboard query uses template variables, create a separate query for alerting with hard-coded values.

InfluxDB alert query complexity limitations

Complex queries with many nested functions or large result sets may timeout or fail to evaluate. Simplify alerting queries by reducing the time range, using appropriate aggregation intervals, and adding filters to limit data scanned.

InfluxDB alerting best practices

When creating InfluxDB alerts: use specific filters with WHERE clauses or Flux filters to focus on relevant data and improve performance; match GROUP BY time interval to evaluation frequency; test queries first in Explore before creating alerts; set realistic thresholds based on historical data patterns; use descriptive names for alert rules that indicate what they monitor.

Create InfluxDB alert rule steps

To create an alert rule using InfluxDB: Navigate to Alerting > Alert rules. Click New alert rule. Enter a name for the alert rule. Select InfluxDB data source. Build query using the query editor. Configure alert condition (e.g., when average is above threshold). Set evaluation interval and pending period. Configure notifications and labels. Click Save rule.

InfluxQL CPU usage alert example

This example shows how to create an alert for high CPU usage in InfluxQL. The query is: SELECT mean("usage_system") FROM "cpu" WHERE $timeFilter GROUP BY time($__interval) fill(null). Alert when mean is above 90.

Microsoft SQL Server alerting overview

Grafana Alerting can be used with Microsoft SQL Server to create alerts based on SQL Server data. This allows monitoring metrics, detecting anomalies, and receiving notifications when specific conditions are met.

MSSQL alert query example: high CPU utilization

This example monitors server resource metrics: ```sql SELECT $__timeGroupAlias(recorded_at, '5m'), AVG(cpu_percent) AS avg_cpu FROM sys_metrics WHERE $__timeFilter(recorded_at) GROUP BY $__timeGroup(recorded_at, '5m') ORDER BY 1 ``` Condition: When avg_cpu is above 85.

MSSQL alerting limitation: template variables not supported

Alert queries cannot contain template variables. Grafana evaluates alert rules on the backend without dashboard context, so variables like $hostname or $environment are not resolved. If a dashboard query uses template variables, create a separate query for alerting with hard-coded values.

MSSQL alerting limitation: query timeout

Complex queries with large datasets may time out during alert evaluation. Optimize queries for alerting by adding appropriate WHERE clauses to limit data, using indexes on time and filter columns, and reducing the time range evaluated.

MSSQL alerting limitation: Current User authentication not supported

If a Microsoft SQL Server data source uses Azure Entra ID Current User authentication, alerting, reporting, and recorded queries are not supported. These features require backend-level credentials that do not rely on a specific user's session.

MSSQL alerting best practices

Best practices for creating Microsoft SQL Server alerts: use time series format and always set query format to time series for alert queries, include time filters using the $__timeFilter() macro to limit data to the evaluation window, use MSSQL macros preferring $__timeGroupAlias() and $__timeGroup() over manual time-bucketing expressions, optimize queries by adding indexes on columns used in WHERE clauses and GROUP BY, test queries first by verifying query returns expected results in Explore before creating an alert, set realistic thresholds based on historical data patterns, and use meaningful names giving alert rules descriptive names that indicate what they monitor.

MSSQL alerting limitation: table format not supported

Queries using the Table format cannot be used for alerting. Set the query format to Time series and ensure your query returns a time column.

MSSQL alerting requires configured data source and permissions

Before creating alerts with Microsoft SQL Server, you must have a configured Microsoft SQL Server data source, appropriate permissions to create alert rules, and understanding of the metrics you want to monitor.

MSSQL alerting supported query types

Microsoft SQL Server alerting works with time series queries that return numeric data over time. Table-formatted queries are not supported in alert rule conditions.

MSSQL alert query format requirements

To create a valid alert query for Microsoft SQL Server: include a time column that returns an SQL datetime/datetime2 or Unix epoch timestamp, return numeric values for the metrics you want to alert on, and sort results by the time column.

MSSQL alert query time series format

Query format support for alerting: Time series format is supported and required for alerting. Table format is not supported and must be converted to time series format for alerts.

Steps to create MSSQL alert rule

To create an alert rule using Microsoft SQL Server: navigate to Alerting > Alert rules, click New alert rule, enter a name for the alert rule, select your Microsoft SQL Server data source, build your query using the query editor by setting format to Time series and including a time column using $__timeGroup() or $__timeGroupAlias() macro, add numeric columns for values to monitor, use $__timeFilter() to filter data by evaluation time range, configure the alert condition, set the evaluation interval and pending period, configure notifications and labels, and click Save rule.

MSSQL alert query example: high error count

This example monitors the number of errors over time: ```sql SELECT $__timeGroupAlias(created_at, '1m'), COUNT(*) AS error_count FROM error_logs WHERE $__timeFilter(created_at) AND level = 'error' GROUP BY $__timeGroup(created_at, '1m') ORDER BY 1 ``` Condition: When error_count is above 100.

MSSQL alert query example: average response time

This example monitors API response times: ```sql SELECT $__timeGroupAlias(request_time, '5m'), AVG(response_time_ms) AS avg_response_time FROM api_requests WHERE $__timeFilter(request_time) GROUP BY $__timeGroup(request_time, '5m') ORDER BY 1 ``` Condition: When avg_response_time is above 500 milliseconds.

MSSQL alert query example: low order volume

This example detects drops in order activity: ```sql SELECT $__timeGroupAlias(order_date, '1h'), COUNT(*) AS order_count FROM orders WHERE $__timeFilter(order_date) GROUP BY $__timeGroup(order_date, '1h') ORDER BY 1 ``` Condition: When order_count is below 10.

Alert rule fails with template variable errors

When alert rules fail because they contain template variables such as $hostname or $region, alert queries cannot use template variables because Grafana evaluates alert rules on the backend without dashboard context. Replace all $variable references with literal values in the alert rule. If the same query is needed in both dashboard panels and alert rules, maintain two separate queries: one with variables for the dashboard and one with hard-coded values for alerting.

Alert evaluation returns no data for InfluxDB queries

When alert queries return 'no data', the query doesn't return time-series data that Grafana can evaluate. Test the query in Explore first to verify it returns data. For InfluxQL ensure the query uses an aggregation function (mean, sum, count) with GROUP BY time($__interval). For Flux use aggregateWindow() to produce time-bucketed results. For SQL use $__dateBin(time) or $__timeGroup(time) to aggregate by time. Check that the alert evaluation time range contains data (alerts use fixed time range, not dashboard time picker). Verify the data source connection is working by clicking Save & test in data source settings.

Alert query format must be Time series, not Table

Alert queries have different requirements than dashboard queries. The query format must be set to Time series. Table format is not supported for alerting.

Alert queries cannot use template variables

Alert queries cannot resolve dashboard template variables. Ensure the query doesn't use template variables and returns data within the alert evaluation time range.

How to create a new alert rule

To create a new alert rule, click Alerts & IRM, then Alert rules, then click the + New alert rule button.

Alert rule name becomes alertname label

The name entered for an alert rule is displayed in the alert rule list and is used as the alertname label for every alert instance created from that rule.

Silence setup and behavior

Silences match alerts using labels to determine whether to silence them. They have a fixed start and end time, unlike mute timings which can repeat periodically.

When to use silences

Use silences for one-time or as-needed suppression of notifications, such as active incident response (suppress notifications while investigating), immediate suppression of a specific alert or group of alerts, one-time maintenance or deployment events, and temporarily suppress alerts for specific services or components.

Mute timing and active timing vs silences - notification suppression methods

Mute timing, active timing, and silences are two distinct methods to suppress notifications in Grafana alerting. Neither prevents alert rules from being evaluated nor stops alert instances from appearing in the user interface; they only prevent notifications from being created.

Mute timing and active timing setup and behavior

Mute timing and active timing are created first and then added to notification policies. They use time interval definitions that can repeat periodically, making them suitable for predictable, recurring time periods.

When to use mute timings

Use mute timing for predictable, recurring time periods when you don't want to receive notifications, such as regular maintenance windows (for example, every Sunday from 2:00 AM to 4:00 AM), non-business hours (nights and weekends), scheduled deployments or known change windows, and regular testing periods.

Grafana managed alerts clustering limitations

The current alerting system does not support high availability. Alert notifications are not deduplicated and load balancing is not supported between instances. For example, silences from one instance will not appear in another instance.

Grafana managed alerts evaluation

Grafana managed alerts are evaluated by the Grafana backend. Rule evaluations are scheduled according to the alert rule configuration, and queries are evaluated by an engine that is part of core Grafana.

Data sources supported for alerting rules

Alerting rules can only query backend data sources with alerting enabled. Built-in or Grafana-maintained data sources that support alerting include: Graphite, Prometheus, Loki, InfluxDB, Elasticsearch, Google Cloud Monitoring, Cloudwatch, Azure Monitor, MySQL, PostgreSQL, MSSQL, OpenTSDB, Oracle, and Azure Data Explorer. Additionally, any community backend data sources with alerting enabled (where both the backend and alerting properties are set in plugin.json) can be used.

Alerting engine metrics reference

The alerting engine publishes internal metrics with the following names and types: alerting.alerts (gauge) - How many alerts by state; alerting.request_duration_seconds (histogram) - Histogram of requests to the Alerting API; alerting.active_configurations (gauge) - The number of active, non-default alertmanager configurations for grafana managed alerts; alerting.rule_evaluations_total (counter) - The total number of rule evaluations; alerting.rule_evaluation_failures_total (counter) - The total number of rule evaluation failures; alerting.rule_evaluation_duration_seconds (histogram) - The time to evaluate a rule; alerting.rule_process_evaluation_duration_seconds (histogram) - The time to process the evaluation results for a rule; alerting.rule_send_alerts_duration_seconds (histogram) - The time to send the alerts to Alertmanager; alerting.rule_group_rules (gauge) - The number of rules; alerting.state_calculation_duration_seconds (histogram) - The duration of calculation of a single state.

Give your agent this brain