alerting

260 notes in this subject, read out of this brain and free to use. This is page 1 of 5.

Grafana Alerting resources supported in migration

The migration assistant can migrate Alert rules, Notifications, Contact points, Mute timings, Notification policy tree, and Notification templates. However, Silences and Alert History are not supported for migration and must be configured manually.

Alert rule limits when migrating large numbers of rules

Attempting to migrate a large number of alert rules might result in the error 'Maximum number of alert rule groups reached: Delete some alert rule groups or upgrade your plan and try again.' Refer to Alert rule limits in Grafana Cloud when migrating alert rules to avoid this error.

alert_rules_state configuration option for migration

The alert_rules_state configuration option in custom.ini or grafana.ini controls how Alert Rules are migrated. The option is set to 'paused' by default to avoid duplicate notifications. Available values are: 'paused' (creates all Alert rules in paused state on the Cloud instance to avoid double notifications) and 'unchanged' (Alert rules maintain their original state from the source instance). Set to 'unchanged' when ready to use alert rules from the cloud instance.

Recording rules in Grafana Alerting

In Grafana Alerting, you can configure recording rules, which pre-compute queries and save the results as new time series metrics for use in other alert rules or dashboard queries.

Grafana-managed alerts recommended over data source-managed alerts

Grafana-managed alert rules are the recommended option for alerting in Grafana. They offer a richer feature set compared to data source-managed alert rules.

Components of an alert rule

An alert rule consists of one or more queries and expressions that select the data you want to measure. It contains a condition to trigger the alert, an evaluation period that determines how often the rule is evaluated, and additional options to manage alert events and their notifications.

Pre-provisioned Loki and Prometheus data source-managed alerts deprecated

Pre-provisioned Loki and Prometheus data source-managed alerts have been deprecated in Grafana and cannot be created in new stacks. New Grafana stacks will use Grafana-managed alerting by default. Datasource-managed alerting is not provisioned in new stacks, though existing stacks are not affected. This applies to the default Loki and Prometheus Grafana Cloud data sources managed by Grafana Labs and Cloud Alertmanager. If you add your own Mimir, Loki, or Alertmanager data sources, you can continue to use data source-managed alerts.

Importing data source-managed alerts to Grafana-managed alerts

Cloud users can import data source-managed rules into Grafana-managed rules using the import tool.

Two types of alert rules in Grafana

Grafana supports two types of alert rules: Grafana-managed alert rules (the recommended option that can query backend data sources including Prometheus-based ones and offer a richer feature set) and data source-managed alert rules (supported for Prometheus-based data sources such as Mimir, Loki, and Prometheus, with rules stored in the data source itself). Data source-managed rules can be converted and imported into Grafana-managed rules.

Namespaces organize rule groups in recording rules

Namespaces can contain one or more rule groups and only have an organizational purpose. From the Namespace dropdown, select an existing rule namespace or add a new one.

Recording rules pre-compute expensive queries

Recording rules allow you to periodically pre-compute frequently used or computationally expensive queries, saving the results as a new time series metric. Alert rules and dashboards can then query the new metric resulting from the recording rule. This is faster than querying real-time data and can help to reduce system load.

Navigate to create data source recording rule

To create a new data source-managed recording rule, click Alerts & IRM, then Alerting, then Alert rules. At the top of the Alert rules page, click More, then New Data source recording rule.

Recording rules in data source-managed groups evaluate sequentially

In data source-managed groups, the alert rules and recording rules within the same evaluation group are evaluated sequentially. This ensures that a recording rule is evaluated before any other alert rule queries the pre-computed metric.

Mimir ruler API configuration requires /prometheus prefix

For Mimir data sources, use the /prometheus prefix. The Prometheus data source supports both Grafana Mimir and Prometheus, and Grafana expects that both the Query API and Ruler API are under the same URL. You cannot provide a separate URL for the Ruler API.

Add custom labels to recording rule metrics

You can optionally add custom labels to the resulting metric by selecting existing key-value pairs from the dropdown or entering new key or value.

Loki requires non-local rule storage type for editing rules

For Loki data sources, the local rule storage type, which is the default, supports only viewing of rules. To edit rules, configure one of the other rule storage types.

Recording rule name must be valid Prometheus metric name

The recording rule name must be a Prometheus metric name and contain no whitespace.

Permission required for data source-managed recording rules

Verify that you have write permission to the Prometheus or Loki data source. Otherwise, you will not be able to create or update data source-managed recording rules.

Data source-managed recording rules always run as instant queries

The queries used in data source-managed recording rules always run as instant queries.

Rule groups define sequential evaluation interval

Rules within a group are run sequentially at a regular interval, with the same evaluation time. Newly created rules are appended to the end of the group, and you can reorder them from the Alert rules page.

Data source-managed recording rules support Prometheus-based sources

Data source-managed recording rules can query Prometheus-based data sources like Mimir or Loki.

How to access $labels in annotation templates

Alert instance label values can be accessed in annotation templates using the syntax {{ $labels.instance }} or alternatively using the index function with {{ index $labels "instance" }}.

Common use cases for templating annotations

Common use cases for templating annotations include: displaying the query value that triggered the alert, highlighting label information that identifies the alert (environment, instance, region), providing specific instructions based on query values, customizing runbook links depending on query labels, and including contact information based on query labels.

Optional annotations included in Grafana

Grafana includes several optional annotations that can be edited in the alert rule: description, summary, and runbook_url. Custom annotations can also be created.

Template annotations and labels use Go templating system

Both annotation/label templates and notification templates are written in the Go templating system. However, the variables and functions available differ between them.

Purpose of annotations in alert rules

Annotations add additional information to alert instances and are often used to help identify the alert and guide responders on how to address the issue. They are key-value pairs that can contain plain text or template code evaluated when the alert fires.

Annotation and label template variables vs notification template variables

Template annotations and labels use variables like $labels and $values that represent alert query data of individual alert instances. Template notifications use different variables like .Alerts that include all firing and resolved alerts in the notification.

How to preview label templates in alert rule

To preview label values when creating or editing an alert rule, select Use notification policy and then click on Preview routing.

Two ways to template in Grafana Alerting

Grafana Alerting supports two distinct templating approaches: (1) Template annotations and labels in the alert rule definition to include extra information from query data, and (2) Template notifications to control the content and appearance of notifications sent to contact points.

How to access $values in annotation templates

Query values can be accessed in annotation templates using the syntax {{ $values.A.Value }} or alternatively using the index function with {{ index $values "A" }}.

Alert instance uniqueness and label values

An alert instance is uniquely identified by its set of labels. Avoid displaying query values in labels, as this can create numerous alert instances—one for each distinct label set. Use annotations for query values instead. If a templated label's value changes, it maps to a different alert instance and the previous instance is considered stale.

Example: templating severity label based on query value

{{ if (gt $values.A.Value 90.0) -}} critic {{ else if (gt $values.A.Value 80.0) -}} high {{ else if (gt $values.A.Value 60.0) -}} medium {{ else -}} low {{- end }} This example sets severity label to critical, high, medium, or low based on the query value, allowing different notifications for different severity levels.

Use cases for templating labels

Template labels when: (1) A new label based on a query value can group a subset of alerts differently, changing how notifications are sent, (2) A new label based on a query value can be used in a notification policy to alter the notification contact point.

Sources of labels in alert rules

Labels can be returned from an alert rule query (such as the pod label in a Kubernetes Prometheus query) or defined as additional labels in the alert rule to provide extra information for processing alerts.

Purpose of labels in alert rules

The set of labels for an alert instance is used to uniquely identify that alert among all other alert instances. Labels determine how alerts are routed and managed for notifications, making their design key to the effectiveness of the alerting system.

Methods to preview annotation templates

Two methods can be used to test and preview annotation templates: (1) Trigger the alert and view the alert instance state in the Grafana UI where all annotations are displayed, (2) Use a notification template that displays all annotations, then preview the notification template using the alert instance.

Steps to template an annotation in alert rule

To template an annotation: (1) Navigate to Alerts & IRM > Alert rules > create or edit an alert rule, (2) Scroll down to the Configure notification message section, (3) Copy in your template in the corresponding annotation field (summary, description, runbook_url, or custom).

Steps to template a label in alert rule

To template a label: (1) Navigate to Alerts & IRM > Alert rules > create or edit an alert rule, (2) Scroll down to the Configure labels and notifications section, (3) Click + Add labels, (4) Enter a key that identifies the label, (5) Copy in your template in the value field.

Notification policies determine alert routing to contact points

Notification policies determine how alerts are routed to contact points. Policies have a tree structure where each policy can have one or more child policies and a set of label matchers. Each alert is evaluated by the default policy and subsequently by each child policy, with alerts routed to appropriate policies by matching alert labels with the policy's label matchers.

Contact point inheritance in notification policies

If the Contact point field is left empty in a child policy, the contact point of the parent policy is inherited.

Override general timings in child policy

The Override general timings option on a child policy allows you to set different timing options than the parent policy. If disabled, the timing options of the parent policy are inherited.

Continue matching subsequent sibling nodes option

The Continue matching subsequent sibling nodes option allows an alert to match multiple policies even after matching the current policy. When enabled on a child policy, subsequent sibling policies will continue to be evaluated for the same alert.

Multiple notification policies for managing complex routing

Multiple notification policies allow you to split routing logic into separate, independently managed routing trees. By default, Grafana uses a single notification policy tree for all alert routing. As organizations grow in size and complexity, you can create additional routing trees to organize routing logic by team, service, or domain. Each notification policy contains a routing tree with its own name, root policy, and child policies which function similar to the default policy. The Default Policy cannot be deleted, but you can reset the policy to clear it.

Mute timings are not inherited from parent notification policy

Mute timings are not inherited from a parent notification policy and have to be configured on each level.

Search notification policies by label matchers or contact points

Grafana allows you to search within the tree of policies by label matchers or contact points. To search by contact point, select a contact point from the Search by contact point dropdown and policies using that contact point are highlighted. To search by label matchers, enter a valid matcher in the Search by matchers input field and multiple matchers can be combined with a comma. All matched policies are exact matches. Grafana supports regular expressions for creating label matchers but does not support regular expression or partial matching in the search for policies.

Notification policies are evaluated top to bottom

Policies are evaluated from top to bottom. If a matching policy is found, the system continues to evaluate its child policies in the order they are displayed.

Add sibling policy to notification policy

To add a sibling policy: In the left-side menu, click Alerts & IRM then Alerting. Click Notification configuration, then select Notification policies tab. Find the child policy to create a sibling for. Click Add new policy -> New sibling above or New sibling below. It is important to determine which policy receives the alert first and to set the correct order of sibling and child policies, as policies are evaluated from top to bottom. If a matching policy is found, the system continues to evaluate its child policies in the order they are displayed. Follow the instructions from step 5 onward in adding a child policy.

Add child policy to notification policy

To add a child policy under a default policy or existing child policy: In the left-side menu, click Alerts & IRM then Alerting. Click Notification configuration, then select Notification policies tab. From the Choose Alertmanager dropdown, select an Alertmanager. Click +New child policy from the default policy or existing child policy. In the Matching labels section, add one or more matching label rules to narrow down a specific case. In the Contact point dropdown, select a contact point for notifications (if left empty, the contact point of the parent policy is inherited). Optionally enable Continue matching subsequent sibling nodes to continue matching sibling policies even after alert matches current policy. Optionally enable Override grouping to set different grouping than the parent policy. Optionally enable Override general timings to set different timing options than parent policy. Click Save policy to save changes.

Edit default notification policy steps

To edit the default notification policy: In the left-side menu, click Alerts & IRM and then Alerting. Click Notification configuration, then select the Notification policies tab. From the Choose Alertmanager dropdown, select an external Alertmanager (by default, Grafana Alertmanager is selected). In the Default policy section, click ... -> Edit. Update the default contact point, choose labels to group alerts in the Group by field, and set timing options for group wait, group interval, and repeat interval. Click Save to save changes.

Default notification policy timing option defaults

The default notification policy has three timing options: Group wait (default 30 seconds) is the time to wait before sending the first notification for a new group of alerts; Group interval (default 5 minutes) is the time to wait before sending a notification about changes in the alert group; Repeat interval (default 4 hours) is the time to wait before sending a notification if the group has not changed since the last notification.

Default notification policy and child policies are assigned to specific Alertmanager

The default notification policy and its child policies are assigned to a specific Alertmanager, and they cannot use contact points or mute timings from other Alertmanagers.

Notification policy responsibilities

When an alert instance is assigned to a notification policy, the policy is responsible for grouping similar alerts to minimize notifications, controlling when notifications are sent using timing options, and determining the contact points that receive the alert notification.

Example notification policy configuration

An example notification policy configuration includes: creating a default contact point for slack notifications and setting it on the default policy; editing the default policy grouping to group alerts by cluster, namespace, and severity; creating a specific route for alerts from the development cluster with an appropriate contact point; creating a specific route for alerts with critical severity with a more invasive contact point integration like pager duty; and creating specific routes for particular teams that handle their own on-call rotations.

Override grouping in child policy

The Override grouping option on a child policy allows you to set different grouping than the parent policy. If disabled, the grouping of the parent policy is inherited.

Amazon SNS settings fields

The Amazon SNS API URL (optional): The SNS API URL, e.g., https://sns.us-east-2.amazonaws.com. If not specified, the SNS API URL from the SNS SDK is used. Signature Version (sigv4): Configures AWS's Signature Verification 4 signing process. Region (optional): The AWS region; if blank, the region from the default credentials chain is used. Access Key (optional): The AWS API access key. Secret Key (optional): The AWS API secret key. Both Access Key and Secret Key must be provided together or left blank together. If left blank, Grafana searches for credentials using the default credentials chain, including environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY), the shared credential file, and EC2 instance roles. Profile (optional): Named AWS profile used to authenticate. SNS topic ARN (optional): If not specified, either Phone number or Target ARN must be specified. For FIFO SNS topics, set a message group interval longer than 5 minutes to prevent deduplication. Phone number (optional): Phone number in E.164 format if message is delivered via SMS. If not specified, either SNS topic ARN or Target ARN must be specified. Target ARN (optional): The mobile platform endpoint ARN if message is delivered via mobile notifications. If not specified, either SNS topic ARN or Phone number must be specified. Subject (optional): Customize the subject; supports notification templates and defaults to default.title; cannot be an empty string. Message (optional): Customize the message; supports notification templates and defaults to default.message. Attributes (optional): Add any SNS message attributes.

AWS SNS IAM policy for Grafana alerting

The minimal IAM policy for Grafana to publish alerts to SNS requires the sns:Publish and sns:GetTopicAttributes actions on the specific SNS topic ARN. Example policy: {"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Action": ["sns:Publish", "sns:GetTopicAttributes"], "Resource": "arn:aws:sns:<region>:<account_id>:<topic_name>"}]}

Steps to create Amazon SNS contact point

To create a contact point with SNS integration: 1. Navigate to Alerts & IRM → Alerting → Notification configuration and select the Contact points tab. 2. Click + Add contact point. 3. Enter a name for the contact point. 4. From the Integration list, select AWS SNS. 5. Set up the required SNS settings. 6. Click Save contact point.

Prerequisites for Amazon SNS integration

Before configuring Amazon SNS notifications, you need an AWS SNS Topic to send notifications to and an AWS IAM Identity with necessary permissions to publish messages to that SNS topic.

Amazon SNS contact point integration overview

The Amazon SNS integration in Grafana Alerting allows you to send alert notifications to an SNS topic. The SNS topic can then be configured to forward notifications to distinct subscriber channels within your SNS account.

SNS FIFO topic message group interval setting

When using a FIFO SNS topic, you should set a message group interval longer than 5 minutes to prevent messages with the same group key being deduplicated by the SNS default deduplication window.

Give your agent this brain