alerting

260 notes in this subject, read out of this brain and free to use. This is page 2 of 5.

AWS SNS IAM policy for Grafana alerting

The minimal IAM policy for Grafana to publish alerts to SNS requires the sns:Publish and sns:GetTopicAttributes actions on the specific SNS topic ARN. Example policy: {"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Action": ["sns:Publish", "sns:GetTopicAttributes"], "Resource": "arn:aws:sns:<region>:<account_id>:<topic_name>"}]}

Amazon SNS settings fields

The Amazon SNS API URL (optional): The SNS API URL, e.g., https://sns.us-east-2.amazonaws.com. If not specified, the SNS API URL from the SNS SDK is used. Signature Version (sigv4): Configures AWS's Signature Verification 4 signing process. Region (optional): The AWS region; if blank, the region from the default credentials chain is used. Access Key (optional): The AWS API access key. Secret Key (optional): The AWS API secret key. Both Access Key and Secret Key must be provided together or left blank together. If left blank, Grafana searches for credentials using the default credentials chain, including environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY), the shared credential file, and EC2 instance roles. Profile (optional): Named AWS profile used to authenticate. SNS topic ARN (optional): If not specified, either Phone number or Target ARN must be specified. For FIFO SNS topics, set a message group interval longer than 5 minutes to prevent deduplication. Phone number (optional): Phone number in E.164 format if message is delivered via SMS. If not specified, either SNS topic ARN or Target ARN must be specified. Target ARN (optional): The mobile platform endpoint ARN if message is delivered via mobile notifications. If not specified, either SNS topic ARN or Phone number must be specified. Subject (optional): Customize the subject; supports notification templates and defaults to default.title; cannot be an empty string. Message (optional): Customize the message; supports notification templates and defaults to default.message. Attributes (optional): Add any SNS message attributes.

Amazon SNS contact point integration overview

The Amazon SNS integration in Grafana Alerting allows you to send alert notifications to an SNS topic. The SNS topic can then be configured to forward notifications to distinct subscriber channels within your SNS account.

Jira integration supports Cloud and Server/Data Center

The Jira integration for alerting supports both Jira Cloud and Jira Server/Data Center installations.

Jira contact point optional settings table

Optional settings for Jira contact points are: Summary (issue title, supports templating, max 255 characters); Description (issue description, text/Markdown/JSON for v3 API, max 32kb); Labels (custom labels for organizing issues, supports templating); Priority (issue priority level like Low/Medium/High/Critical, customize in Jira); Resolve Transition (transition name to move issue to resolved state when alert resolves, empty means no transition); Reopen Transition (transition name to reopen issue when alert reoccurs, empty means no reopening); Reopen Duration (time in minutes to control whether to reopen closed issue or create new one, if not specified uses most recent matching issue); "Won't fix" Transition (resolution status to ignore when searching existing issues, matching issues won't be reopened or updated); Deduplication Key Field (custom text field for deduplication key, if not specified uses label format ALERT(hash sum)); Fields (custom Jira issue fields in format customfield_10001).

Jira contact point required settings table

Required settings for Jira contact points are: URL (REST API URL, versions 2 and 3 supported, e.g. https://your-domain.atlassian.net/rest/api/3, protected in Grafana Cloud); Basic Auth User (username, use email for Jira Cloud); Basic Auth Password (password or personal token, for Jira Cloud obtain token from https://id.atlassian.com/manage-profile/security/api-tokens); API Token (alternative to basic auth, bearer token for authorization); Project Key (unique project identifier for issue creation); Issue Type (type of issue to create, e.g. Task, Bug, Incident, must be available in project).

Jira integration prerequisites

Before configuring Jira integration for alerting, ensure you have: a Jira instance (Cloud, Server, or Data Center), API access credentials for Jira, and appropriate permissions to create issues in your target Jira project.

Create Jira contact point for alerting

To create a contact point with Jira integration: (1) Navigate to Alerts & IRM -> Alerting -> Notification configuration and select Contact points tab. (2) Click + Add contact point. (3) Enter a name for the contact point. (4) From Integration list, select Jira. (5) Set up required settings. (6) Click Save contact point.

Telegram bot naming requirement

When creating a Telegram bot, the bot name must end in 'bot' or '_bot'. For example, valid names include 'my_bot'.

Requirements for Telegram bot setup

To integrate Grafana with Telegram, you need a Telegram bot API token and a chat ID (the ID of the Telegram chat where you want to receive alert notifications). The browser version of Telegram should be used to complete the integration.

Adding Telegram contact point to alert rules

To add a Telegram contact point to an alert rule: Navigate to Alerting > Alert rules. Edit or create a new alert rule. Scroll down to the Configure labels and notifications section. Under Notifications click Select contact point. From the drop-down menu, select the previously created Telegram contact point. Click Save rule and exit.

Telegram contact point configuration steps

To create a Telegram integration in Grafana Alerting: Navigate to Alerts & IRM > Alerting > Notification configuration, then select the Contact points tab. Click + Add contact point. Enter a contact point name. From the Integration list, select Telegram. In the BOT API Token field, enter the bot API token. In the Chat ID field, enter the chat ID. Click Test to verify the integration works (for Grafana Alertmanager only). Click Save contact point.

How to extract Telegram chat ID

The chat ID can be obtained from the URL in the browser's address bar when viewing a Telegram group. The URL format is https://web.telegram.org/a/#CHATID. The chat ID is the sequence of numbers that follows the # symbol. For example, in the URL https://web.telegram.org/a/#-4266674385, the chat ID is -4266674385.

Telegram message character limit

Telegram messages are limited to 4096 UTF-8 characters. If you use a parse_mode other than None, truncation may result in an invalid message, causing the notification to fail. For longer messages, an alternative contact method is recommended.

Alerting examples available for multiple Grafana editions

Alerting examples are available for Grafana Cloud, Grafana Enterprise, and Grafana OSS editions.

Alerting examples documentation scope

The Examples section of Grafana alerting documentation provides practical guides for working with different types of alerting data, applying alert design patterns, reusing alert logic, and taking advantage of specific Grafana Alerting features.

Infinity plugin CSV data type specification

When the Infinity plugin fetches CSV data, all columns are parsed and returned as strings by default, which causes the query expression to fail in Alerting. To make it work, specify the column names and their types in the Infinity query editor to ensure that only one column is treated as a number, matching the format expected by Grafana Alerting.

Tabular data alert evaluation requirements

When a query returns data in table format, Grafana transforms each row into a separate alert instance. To evaluate each row, the data must meet three conditions: (1) Only one numeric column, which is the value used for evaluating the alert condition. (2) Non-numeric columns that define the label set, where the column name becomes a label name and the cell value becomes the label value. (3) Unique label sets per row, so each row is uniquely identifiable by its labels and represents a distinct alert instance. All three conditions must be met or Grafana cannot evaluate the table data and the rule will fail.

Difference between tabular and time series alert data

Tabular data and time series data are evaluated differently in alerting. Time series data contains multiple values over time, each with its own timestamp, and alert rules must reduce each series to a single number using a function like last(), avg(), or max(). Tabular data does not require reduction because each row contains only a single numeric value used to evaluate the alert condition.

Testing tabular alerts with TestData data source

To test tabular alert rules, add the TestData data source through the Connections menu. In an alert rule, select TestData as the data source, then select CSV Content from the Scenario dropdown and paste CSV data. For example, paste: host, disk, percentFree\nweb1, /etc, 3\nweb2, /var, 4\nweb3, /var, 8. Set a condition like $A < 5 and preview the alert to see how Grafana evaluates the table data.

Tabular data transformation into alert instances

When Grafana evaluates tabular data for alerting, each row is transformed into a separate alert instance. The numeric column value is used for the alert condition, and non-numeric columns form the label set for that instance. For example, a row with Host='web1', Disk='/etc', and PercentFree=3 becomes an alert instance labeled {Host="web1", Disk="/etc"} with a value of 3.

SQL query pattern for tabular alert data

When creating alert rules on tabular data from SQL databases, use the `$__timeFilter` macro in the WHERE clause to filter by time without returning the date column to Grafana. Example query: SELECT Host, Disk, AVG(PercentFree) AS PercentFree FROM DiskSpace WHERE $__timeFilter(Time) GROUP BY Host, Disk. This returns only the non-numeric label columns and one numeric column for the alert condition.

Alert rule state and health determination

The alert rule state and alert rule health are determined by the state of the alert instances.

Alert rule state: Pending

An alert rule is in Pending state when at least one alert instance returned by the evaluation engine is Pending.

Alert rule health: No Data

An alert rule has No Data health status when the alert rule query returns no data.

Alert rule health: KeepLast status modifier

An alert rule can have a status with KeepLast modifier, such as '{status}, KeepLast'. This occurs when the rule would have received another status but was configured to keep the last state of the alert rule.

Alert rule health: Ok

An alert rule has Ok health status when there is no error when evaluating the alert rule.

Alert rule health: Error

An alert rule has Error health status when an error occurred when evaluating the alert rule.

Alert rule state: Firing

An alert rule is in Firing state when at least one alert instance returned by the evaluation engine is Alerting.

Alert rule state: Normal

An alert rule is in Normal state when none of the alert instances returned by the evaluation engine is in a Pending or Alerting state.

Alert instance definition in multi-dimensional rules

Each alert rule can generate one or more alert instances. There is one alert instance for each series or dimension in the data. Each alert instance of the same alert rule represents a different target and can be in a different state independently.

Label types in alerting

An alert's label set can contain three types of labels: (1) User-configured labels that you manually configure in the alert rule to identify generated alert instances and manage alerts, which can use templates to customize values from query data; (2) Query labels returned by the data source query that can generate multiple alert instances from the same rule; (3) Reserved labels automatically added by Grafana, including alertname (the name of the alert rule) and grafana_folder (the title of the folder containing the alert). Labels prefixed with grafana_ are reserved by Grafana for special use and can be disabled via the unified_alerting.reserved_labels option.

How labels are used in alerting

The complete set of labels for an alert is what uniquely identifies an alert instance. The alerting UI shows labels for every alert instance generated during evaluation of that rule. Notification policies and silences use labels to match alert instances and route them to contact points or stop their notifications. Contact points can include information from labels in notification messages.

Labels are unique identifiers for alert instances

Labels are unique identifiers of an alert instance. You can use them for searching, silencing, and routing notifications. The complete set of labels for an alert rule is called its label set, and it is this label set that identifies the alert. For example, one alert instance might have the label set {alertname="High CPU usage",server="server1"} while another might have {alertname="High CPU usage",server="server2"}, making them two separate alert instances because their server labels differ.

Built-in annotation fields

Grafana provides several optional annotations that you can edit: (1) summary: A short summary of what the alert has detected and why; (2) description: A detailed description of what happened and what the alert does; (3) runbook_url: The runbook page to guide operators managing a potential incident; (4) __dashboardUid__ and __panelId__: Used to link the alert to a dashboard and panel to facilitate alert investigation. Like labels, annotations can use templates to include dynamic data from queries.

Annotations add information to alert instances

Annotations add additional information to alert instances, helping responders identify and address potential issues. They should be clear and self-explanatory so that first responders can investigate without needing deeper knowledge of the alert setup. Annotations are displayed in Grafana and are included by default in notifications.

Label key format requirements

Grafana's built-in Alertmanager supports both Unicode label keys and values. If using an external Prometheus Alertmanager, label keys must be compatible with Prometheus data model: they must only contain ASCII letters, numbers, and underscores, and must match the regular expression [a-zA-Z_][a-zA-Z0-9_]*. Invalid characters are removed or replaced by the Grafana alerting engine before being sent to external Alertmanager according to these rules: whitespace is removed, ASCII characters are replaced with underscore, and all other characters are replaced with their lower-case hex representation (prefixed with underscore if it's the first character). If multiple label keys are sanitized to the same value, duplicates have a short hash of the original label appended as a suffix.

Label precedence and conflicts

Two alert rules cannot produce alert instances with the same labels. If a configured label has the same name as a data source query label, the configured label replaces the data source label. If a configured label has the same name as a reserved label, it is omitted. The label set for an alert must not have two or more labels with the same name.

Excluding labels with label matchers

You can exclude labels using three methods: team = "" (empty string), team !~ .+ (not matching one or more characters), or team =~ ^$ (regex matching empty string).

Child policy matching precedence

When an alert matches both a parent policy and a child policy, the routing follows the child policy as it provides a more specific match.

Notification policy inheritance from parent policies

Child policies inherit the following notification properties from their parent policy by default: Contact point, Grouping options, and Timing options. Each policy can overwrite these inherited properties if needed.

Multi-label routing requirement

If alerts use multiple labels, these labels must also be present in a notification policy to match and route notifications to a specific contact point.

Default notification policy behavior

The default notification policy matches all alert instances. It always handles alert instances if there are no child policies or if none of the child policies match the alert instance's labels, preventing any alerts from being missed.

Continue matching siblings option

If you want sibling policies of one matching policy to handle the alert instance as well, enable Continue matching siblings on the particular matching policy.

Alert routing evaluation order and matching

To determine which notification policies handle an alert instance, the system looks for matching policies starting from the top of the tree, beginning with the default notification policy. If a matching policy is found, the system evaluates its child policies in the order they are displayed. If a child policy matches the alert, the system evaluates its child policies recursively until no more matching child policies are found. Only the deepest matching child policy handles the alert instance. By default, once a matching policy is found, the system does not continue to look for sibling policies.

Notification policies structure and tree hierarchy

Notification policies are structured as a tree, not a list. The root is the Default notification policy. Each policy can have child policies and sibling policies that share the same parent and hierarchical level. Using label matchers, alert instances are routed to notification policies which can then group multiple alert instances into a single notification and deliver it to a contact point.

Label matcher operators

Label matchers support four operators: = (select labels that are exactly equal to the value), != (select labels that are not equal to the value), =~ (select labels that regex-match the value), !~ (select labels that do not regex-match the value).

Label matcher parts and matching logic

A label matcher consists of three parts: Label (the name of the label to match, must exactly match the label name), Value (matches against the corresponding value for the specified Label, how it matches depends on the Operator), and Operator (the operator to match against the label value). When using multiple label matchers, they are combined using the AND logical operator, meaning all matchers must match to link a rule to a policy.

Multiple notification policies feature

Grafana supports setting up and configuring multiple notification policies with the multiple policy feature, which allows you to split routing logic into separate, independently managed routing trees. By default, Grafana uses a single notification policy tree for all alert routing.

Slack text formatting in notification templates

Slack notifications support text formatting using: asterisks for bold (`*text*`), underscores for italic (`_text_`), and plain text without formatting. Each contact point type has its own formatting requirements.

GroupLabels in notification templates

`.GroupLabels` contains the labels used to group multiple alerts into a single notification. It can be accessed using `.GroupLabels.SortedPairs.Values` to get the values in sorted order. GroupLabels can be excluded from CommonLabels output using `.CommonLabels.Remove .GroupLabels.Names`.

Alert status field in notification templates

Each alert in a notification has a `.Status` field that indicates whether it is firing or resolved. This can be used in conditional logic to handle different alert states differently.

Alert URLs available in Grafana-managed alerts

Grafana-managed alerts expose several URLs in notification templates: `.DashboardURL` (link to the alert's dashboard), `.PanelURL` (link to the specific panel), `.GeneratorURL` (URL for the alert rule), `.SilenceURL` (link to create a silence), and `.Annotations.runbook_url` (custom runbook annotation).

Alert labels and annotations access in templates

Individual alert labels and annotations can be accessed using `.Labels.SortedPairs` and `.Annotations.SortedPairs` respectively. These return ordered collections of key-value pairs that can be iterated to display all labels and annotations for an alert.

CommonAnnotations in notification templates

Templates can access annotations that are common to all alerts in a notification using `.CommonAnnotations`. Like CommonLabels, CommonAnnotations can be iterated using `.CommonAnnotations.SortedPairs` to access all shared annotations as key-value pairs.

Alerts.Firing and Alerts.Resolved in notification templates

Notification templates can access separate collections of firing and resolved alerts using `.Alerts.Firing` and `.Alerts.Resolved`. This allows templates to handle firing and resolved alerts differently, such as displaying them in separate sections of the notification.

Iterate alerts in notification templates using range

Use the `range` keyword to iterate over alerts in a notification template. Access individual alert properties like `.Labels.alertname` or use `index .Labels "alertname"` notation to retrieve specific label values. Two syntax forms are available: dot notation (`.Labels.alertname`) and index notation (`index .Labels "alertname"`).

CommonLabels in notification templates

Templates can check if alerts have common labels using `.CommonLabels`. CommonLabels represent labels that are shared across all alerts in a notification. They can be accessed and iterated using `.CommonLabels.SortedPairs` which returns a collection of key-value pairs ordered by name.

Notification templates access notification data with dot notation

Notification templates can access notification data using the dot (.) operator. This allows templates to dynamically reference alert information, labels, annotations, and other notification content.

Conditional logic in notification templates

Use `if` statements in notification templates to execute blocks conditionally. For example, `{{ if .CommonLabels }}` checks if CommonLabels exist, and `{{ if gt (.Alerts.Firing | len) 0 }}` checks if there are firing alerts.

Give your agent this brain