alerting

260 notes in this subject, read out of this brain and free to use. This is page 5 of 5.

Example LogQL query for counting non-200 status logs

The following LogQL query counts log lines with status codes other than 200 over a 10-minute interval and sums results by message type: sum by (message)(count_over_time({filename="/var/log/web_requests.log"} != "status=200" | pattern "<_> <message> duration<_>" [10m])). This query uses an instant query to return a single result set and extracts the message label using the pattern parser.

Log-based alerting with Loki advantages

Loki stores logs and only indexes labels for each log stream. Using Loki with Grafana Alerting allows creating metric alert rules based on log line content and sending team notifications. A key advantage is the ability to add label data extracted directly from log messages into alert notifications.

Creating alert rules based on Loki log queries

Alert rules can be created in Grafana Alerting based on Loki queries using LogQL. Navigate to Alerting > Alert rules, click New alert rule, enter a descriptive name, select the Loki datasource, and write a LogQL query in Code mode. The alert condition threshold determines when the alert fires; for example, a threshold of 0 means the alert fires when the query returns any results above zero. Preview the alert rule condition to verify it returns expected alert instances before saving.

LogQL pattern parser for extracting label data from log lines

The LogQL pattern parser can extract new labels directly from log message content using the syntax | pattern "<_> <field_name> <_>". For example, the pattern "<_> <message> duration<_>" extracts a message label from log lines. This extracted label data can be included directly in alert notifications. When log lines follow a consistent format like '2023-04-22T02:49:32.562825+00:00 level=info method=GET url=test.com status=200 duration=171ms', the pattern parser can extract specific fields for use in alerting logic.

Alert rule evaluation group and evaluation interval settings

An evaluation group determines how frequently an alert rule is evaluated. An evaluation interval (for example, 1 minute) sets how long the alert condition must be met before the alert fires. Setting the pending period to 0 seconds causes the alert to fire immediately when the condition is met. These settings help reduce unnecessary notifications by allowing data to stabilize before triggering alerts.

Webhook integration for alert notifications

Alert rules can send notifications to a webhook contact point. To set up a webhook contact point in Grafana Alerting UI: hover over the Alerting (bell) icon, click Contact points, click + New contact point, enter a name, select Webhook as the Integration type, paste the webhook URL, click Test to send a test notification, and then Save contact point. The webhook receives POST requests containing alert notification details.

Alert rule creation steps in Grafana Alerting

To create an alert rule: 1) Navigate to Alerting > Alert rules and click New alert rule. 2) Enter a short, descriptive alert rule name. 3) Select a datasource (such as Loki). 4) Define the query and alert condition (using the Last reducer function and a threshold value). 5) Click Preview alert rule condition to verify the query runs correctly. 6) Set the Folder and Evaluation group, or create new ones. 7) Choose an Evaluation interval (for example, 1 minute). 8) Set the pending period (for example, 0 seconds). 9) Select a Contact point from the dropdown. 10) Click Save rule and exit.

Preview alert instances in notification template editor

In the notification template editor, you can preview how actual alert instances will appear in the notification. Click 'Edit payload', then 'Use existing alert instance' to select an alert rule and one of its instances, and 'Add alert data to payload' to add it to the preview.

Apply notification template to contact point

To use a notification template with a contact point, navigate to Alerts & IRM > Alerting > Contact points, edit the contact point, click 'Edit Message' in the Optional email settings section, search for the notification template name under 'Select notification template', and save.

Two types of templates in Grafana Alerting

Grafana Alerting uses two types of templates: labels and annotations templates, and notification templates. Labels and annotations templates are applied to alert rule labels and annotations using the Go template language. Notification templates allow you to pull in details from annotations to create clear and consistent messages, and can be reused across different contact points.

Main advantages of using templating in annotations

One of the main advantages of using templating in annotations is the ability to incorporate dynamic data from queries, allowing alerts to reflect real-time information relevant to the triggered condition. By using templating in annotations, you can customize the content of each alert instance, such as including instance names and metric values, so the notification becomes more informative.

Notification template benefits for reducing alert noise

Notification templates are helpful when you want to reduce notification noise, especially in situations where multiple instances of an alert are firing at the same time. You can leverage templates to create a unified, easy-to-read notification that includes all pertinent details in one well-structured message instead of receiving a long list of individual alert notifications.

Alert rule annotation templating with $labels and $values

When templating alert rule annotations, you can use $labels to reference label values and $values to reference query result values. For example, the template `{{ index $labels "instance" }}` retrieves the instance label value, and `{{ index $values "A" }}` retrieves the result of query A. These can be combined with Go template functions like special characters `\n` for new lines and `\t` for tabs.

Notification template structure for firing and resolved alerts

A notification template can organize alert instances into separate sections for firing and resolved alerts. The built-in template structure uses `.Alerts.Firing` to reference firing alert instances and `.Alerts.Resolved` to reference resolved alert instances. Sub-templates like `alert.summary_and_description` can be defined using `{{ define "template.name" }}` and called with `{{ template "alert.summary_and_description" . }}`.

Alert instance data available in notification templates

Notification templates have access to alert instance data including `.Annotations.summary` for the summary annotation, `.Annotations.description` for the description annotation, and `.Status` for the alert status.

Notification template name uniqueness requirement

Your notification template name defined with `{{ define "<NAME>" }}` must be unique. You cannot have two templates with the same name in the same notification template group or in different notification template groups.

Example notification template for firing and resolved alerts

This notification template displays firing and resolved alerts separately with their count and details: ``` {{- /* Example displaying firing and resolved alerts separately in the notification. */ -}} {{- /* Edit the template name and template content as needed. */ -}} {{ define "custom.firing_and_resolved_alerts" -}} {{ len .Alerts.Resolved }} resolved alert(s) {{ range .Alerts.Resolved -}} {{ template "alert.summary_and_description" . -}} {{ end }} {{ len .Alerts.Firing }} firing alert(s) {{ range .Alerts.Firing -}} {{ template "alert.summary_and_description" . -}} {{ end -}} {{ end -}} {{ define "alert.summary_and_description" }} Summary: {{.Annotations.summary}} Status: {{ .Status }} Description: {{.Annotations.description}} {{ end -}} ```

Example alert rule annotation with instance and CPU usage

This annotation template includes instance name and CPU usage from a query: ``` {{- "\n" -}} Instance: {{ index $labels "instance" }} {{- "\t" -}} Usage: {{ index $values "A"}}%{{- "\n" -}} ``` It uses `$labels` to get the instance name and `$values "A"` to get the CPU usage from query A, with `\t` for tab spacing and `\n` for new lines.

Grafana Alerting does not support semi-relative time ranges

Semi-relative time ranges are not supported by Grafana Alerting.

Grafana Alerting time range limitations

Grafana Alerting does not support now+n for future timestamps and does not support now-1n/n for start of n until end of n because this is an absolute timestamp.

Give your agent this brain