Access notification history from History page
To view notification history from the History page: Click Alerts & IRM > Alerting > History, then select the Notifications tab. Each row represents a notification sent to a contact point.
Grafana dashboards · all subjects
notification policies
25 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.
Notification history records both firing and resolved notifications
Grafana records a notification event each time it attempts to send a notification to a contact point for a group of alerts. Both firing and resolved notifications are recorded, providing a complete delivery history even after an alert has recovered.
Difference between notification history and active notifications
Notification history shows notifications that were already sent, while active notifications show alerts that are currently grouped and waiting to be sent.
Grafana OSS and Enterprise notification history requires Loki configuration
Grafana OSS and Grafana Enterprise users must store notification history in Loki. They must set enabled = true in the [unified_alerting.notification_history] section of the Grafana configuration file and configure the connection to Loki. Grafana Cloud manages notification history automatically.
Notification history Loki configuration parameters
The [unified_alerting.notification_history] section in the Grafana configuration file includes the following parameters: enabled (enables notification history functionality; Alertmanager notification logs stored in Loki), loki_remote_url (URL of the Loki instance, default http://localhost:3100), loki_tenant_id (optional tenant ID to attach to requests sent to Loki), loki_basic_auth_username (optional basic authentication username for requests sent to Loki), loki_basic_auth_password (optional basic authentication password for requests sent to Loki). Extra labels can be attached to every notification history record with the [unified_alerting.notification_history.external_labels] section.
History page notification list columns
The History page notification list displays the following columns: Time (when the notification was sent), Status (whether the notification was for Firing or Resolved alert), Group Labels (labels that identify the group of alerts included in the notification), Contact point (contact point and integration that received the notification). Failed notifications are marked with a Failed badge, with error details shown on hover.
History page filters for notification list
The History page allows filtering notifications by: Labels (enter a label or click a label on a notification), Status (Firing or Resolved), Delivery outcome (successful or failed notifications), Contact point (by contact point name), Time range (adjust with time picker).
Expand notification row to see included alerts
Expanding a notification row shows the alerts included in the notification, grouped into Firing Alerts and Resolved Alerts. Each alert shows its labels, annotations, summary, description, and links to the alert rule that generated it.
View single notification detail page
Selecting View on a notification row opens a detail page showing the full context of the notification, including the contact point and integration used, the delivery outcome, any error returned, and the complete list of firing and resolved alerts that were included.
View notification history for specific alert rule
To view notification history for a specific alert rule: Click Alerts & IRM > Alerting > Alert rules, click an alert rule, then select the Notifications tab. The tab shows the same notification list as the History page, pre-filtered to alerts generated by that rule.
Query notification history through Loki in Explore
Because notification history is stored in Loki, you can query it directly in Loki through Grafana Explore. Use Explore to build custom dashboards or run free-form investigations of notification delivery.
View notification errors location
To view notification errors, navigate to Alerts & IRM -> Alerting -> Notification configuration, then select the Contact points tab. Each contact point displays a message about the status of their latest notification deliveries. If a contact point is failing, a red message indicates that there are errors delivering notifications. Hover over the error message to see the notification error details.
Notification errors availability
Notification errors are only available with pre-configured Grafana Alertmanagers.
Notification policies tree structure and purpose
Notification policies determine how alerts are routed to contact points. Policies have a tree structure where each policy can have one or more child policies and a set of label matchers. Each alert instance is evaluated by the default policy and subsequently by each child policy, with alerts routed to appropriate policies by matching alert labels with policy label matchers.
Notification policy responsibilities
When an alert instance is assigned to a notification policy, the notification policy is responsible for: grouping similar alerts to minimize notifications, controlling when notifications are sent using timing options, and determining the contact points that receive the alert notification.
Default notification policy and Alertmanager scope
The default notification policy and its child policies are assigned to a specific Alertmanager, and they cannot use contact points or mute timings from other Alertmanagers.
Edit default notification policy steps
To edit the default notification policy: click Alerts & IRM then Alerting in the left menu, click Notification configuration and select the Notification policies tab, select an Alertmanager from the Choose Alertmanager dropdown (defaults to Grafana Alertmanager), click the ellipsis menu and select Edit in the Default policy section, update the default contact point, choose labels to group alerts in the Group by field, set timing options (Group wait, Group interval, Repeat interval), then click Save.
Default notification policy timing options and defaults
Notification policy timing options include: Group wait (time to wait before sending first notification for new alert group, default 30 seconds), Group interval (time to wait before sending notification about changes in alert group, default 5 minutes), and Repeat interval (time to wait before sending notification if group has not changed since last notification, default 4 hours).
Add a child policy steps
To add a child policy: click Alerts & IRM then Alerting in the left menu, click Notification configuration and select Notification policies tab, select an Alertmanager from dropdown, click +New child policy from the default policy or existing child policy, add one or more matching label rules in the Matching labels section, select a contact point from dropdown (inherits parent if empty), optionally enable Continue matching subsequent sibling nodes to allow multiple policies to handle same alert, optionally enable Override grouping to set different grouping than parent, optionally enable Override general timings to set different timing options than parent, then click Save policy.
Add a sibling policy steps
To add a sibling policy: click Alerts & IRM then Alerting in the left menu, click Notification configuration and select Notification policies tab, find the child policy to create sibling for, click Add new policy and select New sibling above or New sibling below. Policies are evaluated from top to bottom, and if a matching policy is found, the system evaluates its child policies in the order displayed.
Search notification policies by contact point and matchers
Grafana allows searching within the tree of policies by label matchers or contact points. To search by contact point, select a contact point from the Search by contact point dropdown to highlight policies using that contact point. To search by label matchers, enter a valid matcher in the Search by matchers input field. Multiple matchers can be combined with a comma. All matched policies are exact matches; regular expressions are supported for creating label matchers but not for policy search.
Mute timings inheritance in notification policies
Mute timings are not inherited from a parent notification policy and must be configured on each level individually.
Multiple notification policy trees
By default, Grafana uses a single notification policy tree for all alert routing. Multiple notification policy trees allow splitting routing logic into separate, independently managed routing trees organized by team, service, or domain. Each notification policy tree has its own name, root policy, and child policies. The Default Policy cannot be deleted but can be reset to clear it.
Example notification policy configuration setup
An example notification policy configuration includes: creating a default contact point for slack notifications and setting it on the default policy, editing the default policy grouping to group alerts by cluster, namespace, and severity, creating a specific route for alerts from the development cluster with an appropriate contact point, creating a specific route for critical severity alerts with more invasive contact point integration like pagerduty, and creating specific routes for particular teams that handle their own on-call rotations.
Child policy matching labels example
A child policy of the default policy can handle team=security alerts, or a child policy can handle only severity=critical alerts of a parent policy.