alerting

260 notes in this subject, read out of this brain and free to use. This is page 3 of 5.

Conditional logic in notification templates

Use `if` statements in notification templates to execute blocks conditionally. For example, `{{ if .CommonLabels }}` checks if CommonLabels exist, and `{{ if gt (.Alerts.Firing | len) 0 }}` checks if there are firing alerts.

Template nesting and execution in Grafana alerts

Notification templates can define reusable sub-templates and execute them from main templates. Use `{{ template "template_name" . }}` to execute a sub-template, passing the current context (dot) as argument. Sub-templates are defined with `{{ define "template_name" -}} ... {{ end }}`.

Alerts.Firing and Alerts.Resolved length in templates

Get the count of firing alerts using `{{ .Alerts.Firing | len }}` and resolved alerts using `{{ .Alerts.Resolved | len }}`. The pipe operator applies the len filter to get the length of each collection.

Alert StartsAt and EndsAt timestamps in templates

Each alert has `.StartsAt` and `.EndsAt` timestamp fields. These can be used to calculate time ranges or perform arithmetic. Use `.StartsAt.Add -3600000000000` to subtract time (in nanoseconds), or `.UnixMilli` to convert to Unix milliseconds for URL parameters.

Accessing current time in notification templates

Use `(time.Now).UnixMilli` in templates to get the current time in Unix milliseconds. This is useful for setting the `to` parameter in dashboard URLs when the alert is still firing.

Dashboard URL time range in alert notifications

Dashboard and panel URLs in alert notifications automatically include time range parameters. The `from` parameter is set to one hour before the alert started (`.StartsAt`). The `to` parameter is set to the current time if the alert is firing, or the alert's end time (`.EndsAt`) if resolved.

Join function in notification templates

Use the `join` filter to combine array values into a string. For example, `{{ .GroupLabels.SortedPairs.Values | join " " }}` joins all values from sorted pairs with a space separator.

toUpper function in notification templates

Use the `toUpper` filter to convert text to uppercase. For example, `{{ .Status | toUpper }}` converts the alert status to uppercase.

Comparison operators in notification templates

Notification templates support comparison operators like `eq` (equal), `gt` (greater than), `lt` (less than) for conditional logic. Use them in if statements: `{{ if eq .Status "firing" }}` or `{{ if gt (.Alerts.Firing | len) 0 }}`.

Alert annotations in notification templates

Access individual alert annotations using `.Annotations.annotation_name`. Common annotations include `.Annotations.summary` for a summary description and `.Annotations.description` for detailed information about the alert.

Variable assignment in notification templates

Assign values to variables in templates using `{{- $variableName := value }}` syntax. The dash before the opening braces removes whitespace. For example: `{{- $from := (.StartsAt.Add -3600000000000).UnixMilli }}`.

Add information to alerts using annotations and labels instead of templates

Avoid adding extra information in notification templates because that information is only visible in the notification message. Instead, use annotations or labels to add information directly to the alert rule, ensuring the information is visible in the alert state and alert history within Grafana. Print the annotation or label in the template.

Notification title/subject template example

Example template that shows the number of firing and resolved alerts with emoji: `{{ define "custom_title" -}}{{ if gt (.Alerts.Firing | len) 0 }}🚨 {{ .Alerts.Firing | len }} firing alerts. {{ end }}{{ if gt (.Alerts.Resolved | len) 0 }}✅ {{ .Alerts.Resolved | len }} resolved alerts.{{ end }}{{ end }}`

Notification template example with summary and description

Example showing how to display summary and description annotations for each alert: ```go {{ define "custom.alerts" -}} {{ len .Alerts }} alert(s) {{ range .Alerts -}} {{ template "alert.summary_and_description" . -}} {{ end -}} {{ end -}} {{ define "alert.summary_and_description" }} Summary: {{.Annotations.summary}} Status: {{ .Status }} Description: {{.Annotations.description}} {{ end -}} ```

Notification template example displaying common labels and annotations

Example showing how to display labels and annotations shared by all alerts: ```go {{ define "custom.common_labels_and_annotations" -}} {{ len .Alerts.Resolved }} resolved alert(s) {{ len .Alerts.Firing }} firing alert(s) Common labels: {{ len .CommonLabels.SortedPairs }} {{ range .CommonLabels.SortedPairs -}} - {{ .Name }} = {{ .Value }} {{ end }} Common annotations: {{ len .CommonAnnotations.SortedPairs }} {{ range .CommonAnnotations.SortedPairs }} - {{ .Name }} = {{ .Value }} {{ end }} {{ end -}} ```

Notification template example displaying individual alert labels and annotations

Example showing how to display all labels and annotations for each alert: ```go {{ define "custom.alert_labels_and_annotations" -}} {{ len .Alerts.Resolved }} resolved alert(s) {{ range .Alerts.Resolved -}} {{ template "alert.labels_and_annotations" . -}} {{ end }} {{ len .Alerts.Firing }} firing alert(s) {{ range .Alerts.Firing -}} {{ template "alert.labels_and_annotations" . -}} {{ end -}} {{ end -}} {{ define "alert.labels_and_annotations" }} Alert labels: {{ len .Labels.SortedPairs }} {{ range .Labels.SortedPairs -}} - {{ .Name }} = {{ .Value }} {{ end -}} Alert annotations: {{ len .Annotations.SortedPairs }} {{ range .Annotations.SortedPairs -}} - {{ .Name }} = {{ .Value }} {{ end -}} {{ end -}} ```

Notification template example displaying URLs for Grafana-managed alerts

Example showing how to display dashboard, panel, generator, silence, and runbook URLs for Grafana-managed alerts: ```go {{ define "custom.alert_additional_details" -}} {{ len .Alerts.Resolved }} resolved alert(s) {{ range .Alerts.Resolved -}} {{ template "alert.additional_details" . -}} {{ end }} {{ len .Alerts.Firing }} firing alert(s) {{ range .Alerts.Firing -}} {{ template "alert.additional_details" . -}} {{ end -}} {{ end -}} {{ define "alert.additional_details" }} - Dashboard: {{ .DashboardURL }} - Panel: {{ .PanelURL }} - AlertGenerator: {{ .GeneratorURL }} - Silence: {{ .SilenceURL }} - RunbookURL: {{ .Annotations.runbook_url}} {{ end -}} ```

Notification template example with dashboard link and time range

Example showing how to include dashboard and panel links with correct time range: ```go {{ define "custom.link_to_dashboard" -}} {{ range .Alerts -}} Dashboard: {{.DashboardURL}} Panel: {{ .PanelURL }} {{ end -}} {{ end -}} ``` Alternatively, build a custom URL with calculated time ranges: ```go {{ define "custom.my_dashboard_url_annotation" -}} {{ range .Alerts -}} {{- $from := (.StartsAt.Add -3600000000000).UnixMilli }} {{- $to := "" }} {{- if eq .Status "resolved" }} {{- $to = (.EndsAt).UnixMilli }} {{- else -}} {{- $to = (time.Now).UnixMilli }} {{- end -}} Dashboard: {{.Annotations.MyDashboardURL}}?from={{$from}}&to={{$to}} {{ end }} {{ end }} ```

Slack formatted message template example

Example of a Slack-formatted notification template: ```go {{ define "slack_formatted_message" -}} *This text will be bold in Slack* _This text will be italic in Slack_ Regular text without formatting {{ end }} ``` Execute with: `{{ template "slack_formatted_message" . }}`

Conditional template example checking for CommonLabels

Example of checking if alerts have common labels: ```go {{ define "custom_message" -}} {{ if .CommonLabels }} Alerts have common labels {{ else }} There are no common labels {{ end }} {{ end }} ```

Template example iterating alerts and printing specific labels

Example of iterating over alerts and accessing specific labels: ```go {{ define "custom_message" -}} {{ range .Alerts }} The name of the alert is {{ index .Labels "alertname" }} {{ end }} {{ end }} ``` Or using dot notation: ```go {{ define "custom_message" -}} {{ range .Alerts }} The name of the alert is {{ .Labels.alertname }} {{ end }} {{ end }} ```

Default notification title template in Grafana

Example of the default title/subject template used in Grafana: ```go {{ define "copy_of_default_title" -}} [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ if gt (.Alerts.Resolved | len) 0 }}, RESOLVED:{{ .Alerts.Resolved | len }}{{ end }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }} {{ end }} ``` This template prints the status and alert count, group labels, and common labels excluding group labels.

Hover over notification error message for details

In the Contact points tab, hover over the red error message next to a contact point to see the notification error details.

Access Active notifications view

To view alert groups and their notification states, click Alerts & IRM -> Alerting -> Alert activity, then select the Active notifications tab.

Active notifications view displays alert groups and notification state

The Active notifications page lists groups of alerts that are actively triggering notifications. By default, alert groups are grouped by the notification policies grouping. Each group displays its label set, contact point, and the number of alert instances.

Notification states in Active notifications view

Alert instances in the Active notifications view can be in one of three notification states: Unprocessed (alert is received but its notification has not been processed yet), Suppressed (the alert has been silenced), or Active (the alert notification has been handled and the alert is still firing and continues to be managed).

View notification errors for contact points

To view notification errors, navigate to Alerts & IRM -> Alerting -> Notification configuration, then select the Contact points tab. Each contact point displays a message about the status of their latest notification deliveries. If a contact point is failing, a red message indicates errors delivering notifications.

Filter alerts by label in Active notifications

In the Search field of the Active notifications view, enter an existing label to view alerts matching that label. Label filtering supports syntax like environment=production,region=~US|EU,severity!=warning.

Filter alerts by state in Active notifications

In the States dropdown of the Active notifications view, select from Active, Suppressed, or Unprocessed states to filter and view only alerts matching your selected state.

Filter alerts by Alertmanager in Active notifications

In the Alertmanager dropdown of the Active notifications view, select an external Alertmanager to view only alert groups for that specific Alertmanager. By default, the Grafana Alertmanager is selected.

Custom group by labels in Active notifications

From the Custom group by dropdown in the Active notifications view, select a combination of labels to view a grouping other than the default. This helps validate the grouping settings of your notification policies. If an alert does not contain labels specified in the grouping, the alert is added to a catch all group with a header of No grouping.

Notification errors availability

Notification errors are only available with pre-configured Grafana Alertmanagers.

Techniques to handle missing data in queries

When NoData is not an issue, rewrite the query to always return data. In Prometheus, use your_metric_query OR on() vector(0) to return 0 when your_metric_query returns nothing. If data is frequently missing due to scrape delays, adjust the Time Range query option in Grafana by setting To to now-1m to account for late data points. In Prometheus, use last_over_time(metric_name[window]) to pick the most recent sample.

MissingSeries eviction does not behave like No Data

Missing Series is designed to prevent alert flapping when pods or instances disappear in dynamic environments. By default, No Data triggers an alert to indicate a potential problem, while MissingSeries eviction reduces alert noise from infrastructure or deployment changes.

Detect missing series dynamically with present_over_time

Use present_over_time(metric_name{}[past_window]) unless present_over_time(metric_name{}[recent_window]) to find targets present in the past but missing recently. To group by a label: group(present_over_time(metric_name{}[past_window])) by (label) unless group(present_over_time(metric_name{}[recent_window])) by (label). This approach scales better for dynamic environments.

Detect missing data in Prometheus with absent_over_time

Use absent_over_time(metric_name[duration]) == 1 to trigger when all series are absent for the specified duration. For detecting missing data per-label, specify the label in the query: absent_over_time(metric_name{label="value"}[duration]) == 1. This approach does not scale well for dynamic cloud environments with many instances.

Missing series eviction process

Grafana marks missing series as stale after two evaluation intervals and triggers alert instance eviction: alert instances with missing data keep their last state for two evaluation intervals; if data is still missing, Grafana adds the annotation grafana_state_reason: MissingSeries, transitions the alert instance to Normal state, sends a resolved notification if the alert was previously firing, and removes the alert instance from the Grafana UI.

DatasourceNoData alert characteristics

DatasourceNoData alerts are distinct alert instances separate from the original alert instance. They use alertname DatasourceNoData and do not inherit all labels from the original alert instances. They may require dedicated setup for handling notifications.

No Data alert configuration options

When an alert query returns no data, configure the alert rule behavior by clicking 'Alert state if no data or all values are null' with four options: No Data (default) triggers a DatasourceNoData alert; Alerting transitions each existing alert instance to Alerting state; Normal ignores missing data and transitions instances to Normal state; Keep Last State leaves the alert in its previous state until data returns.

No Data vs Missing Series definitions

No Data occurs when a query returns no data for any series. Missing Series occurs when one or more specific series, which previously returned data, are missing, but other series still return data. Both are distinct from Connectivity Error, which occurs when the alert rule query itself fails. Multi-dimensional alerts return multiple time series and can generate multiple alert instances.

Best practices for alerting in dynamic environments

Do not alert on every instance by default. In dynamic environments, it is better to aggregate and alert on symptoms unless a missing individual instance directly impacts users. If receiving too much noise from disappearing data, consider adjusting alerts, using Keep Last State, or routing those alerts differently. Silence or mute alerts during planned maintenance or rollouts. Review MissingSeries notifications to confirm whether something broke or if the alert was unnecessary.

Handle MissingSeries resolved notifications

A stale alert instance triggers a resolved notification if it transitions from a firing state (Alerting, No Data, or Error) to Normal with grafana_state_reason annotation set to MissingSeries. This indicates the alert was not resolved by recovery but evicted due to missing series data. Display the grafana_state_reason annotation or process these alerts differently to handle them appropriately.

GRAFANA_ALERTS metric format

The GRAFANA_ALERTS metric used for Prometheus alert state storage has the following label structure: alertname, alertstate, grafana_alertstate, grafana_rule_uid, and additional alert labels. Example: GRAFANA_ALERTS{alertname="", alertstate="", grafana_alertstate="", grafana_rule_uid="", <additional alert labels>}

Alert state history recording overview

Alerting can record all alert rule state changes for Grafana-managed alert rules in a Loki instance, a Prometheus instance, or both. With Prometheus, you can query the GRAFANA_ALERTS metric for alert state changes in Grafana Explore. With Loki, you can query and view alert state changes in Grafana Explore and the Grafana Alerting History views.

Loki configuration for alert state history

Configure Loki with the following settings to optimize for alert state history queries that may span up to 30 days: split_queries_by_interval set to '24h' and max_query_parallelism set to 32. These settings should be placed in the limits_config section. Since this may impact performance of an existing Loki instance, use a separate Loki instance for alert state history.

Grafana configuration for Loki alert state history

Enable Grafana to write alert state history to Loki by configuring the [unified_alerting.state_history] section with: enabled = true, backend = loki, and loki_remote_url set to the Loki server URL (example: http://localhost:3100). Additionally, enable the alertingCentralAlertHistory feature toggle under [feature_toggles].

Prometheus configuration for alert state history

Enable the remote write receiver in Prometheus by setting the --web.enable-remote-write-receiver command-line flag. This enables the endpoint to receive alert state data from Grafana Alerting.

Grafana configuration for Prometheus alert state history

Enable Grafana to write alert state history to Prometheus by configuring the [unified_alerting.state_history] section with: enabled = true, backend = prometheus, and prometheus_target_datasource_uid set to the target data source UID. Optional parameters include prometheus_metric_name (default is GRAFANA_ALERTS) and prometheus_write_timeout (default is 10s).

Grafana configuration for Loki and Prometheus alert state history

To configure both Loki and Prometheus for alert state history, set backend = multiple in the [unified_alerting.state_history] section. Configure primary = loki with loki_remote_url, and secondaries = prometheus with prometheus_target_datasource_uid. Both systems must be configured as described in their respective sections.

Query alert state with Prometheus

Use Grafana Explore to query the GRAFANA_ALERTS metric for alert state information. Example query: GRAFANA_ALERTS{alertstate='firing'} to find all currently firing alerts.

Identifying provisioned resources in Grafana UI

Provisioned alerting resources are labeled with a 'Provisioned' tag in the Grafana UI to indicate they were not created manually.

Exporting provisioned and manual alerting resources

Both manually created and provisioned alerting resources can be exported. You can also edit and export an alert rule without applying the changes.

HTTP API JSON output compatibility

The JSON output from the majority of Alerting HTTP endpoints is not compatible for provisioning via configuration files. Use the Export Alerting API endpoints to return or download alerting resources in provisioning format instead.

Alerting HTTP API scope

The Alerting provisioning HTTP API can be used to create, modify, and delete resources for Grafana-managed alerts. To manage resources related to data source-managed alerts, including recording rules, use the Mimir or Cortex tool instead.

File provisioning availability

Provisioning alerting resources with configuration files is not available in Grafana Cloud.

Methods to provision alerting resources

Grafana provides three methods to import or provision alerting resources: configuration files on disk (not available in Grafana Cloud), Terraform, or the Alerting provisioning HTTP API.

Editing restrictions for imported alerting resources

Imported alerting resources cannot be edited in the Grafana UI in the same way as manually created resources. Imported contact points, notification policies, templates, and mute timings can only be edited in the source where they were created (for example, if managed via files, you cannot edit them in Terraform or within Grafana).

Alerting resources that can be imported

You can import the following alerting resources in Grafana: alert rules, contact points, notification policies, mute timings, and templates.

Notification alerts Terraform configuration

In Terraform, notification alerts are configured to manage how alerts are processed and routed in your Knowledge Graph.

Suppressed assertions Terraform configuration

Suppressed assertions in Terraform define suppression rules to temporarily disable specific alerts during maintenance windows or testing.

Give your agent this brain