correlations

36 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Correlation permission requirements

Only users with write permissions to data sources can define new correlations.

Create correlation in Administration page steps

To create a correlation via Administration page: Go to Administration section, open Correlations page under Plugins and data, click Add button, provide label and optional description, go to next page, provide target data source and target query using variables, go to next page, provide source data source and results field, optionally add transformations by clicking Add button and selecting transformation type, then save correlation. You cannot change selected data sources when editing an existing correlation.

Correlation provisioning structure

Correlations are provisioned by extending provisioned data sources. Correlations are defined as a subsection of the source data source configuration under the correlations key. Each correlation object contains: targetUID (string, required), label (string, required), description (string, optional), type (string, required, valid values: "query" or "external"), config (object, required containing target, field, and transformations), config.target (target query model, required), config.field (string, required, name of field where link is shown), config.transformations (list, optional, list of transformation objects). Transformation objects contain: type (string, required, valid values: "regex" or "logfmt"), field (string, optional, field to transform; if not defined applies to config.field), expression (regex string, regex transformation only), mapValue (string, regex transformation only, new name of variable from first regex match).

Determine target query model structure for provisioning

To determine the target query model structure for provisioning: Open Explore, select the data source to use as correlation target, open inspector tab and select Query, run a sample query, inspect results, look for the queries list object. Each object uses the query model structure defined by the data source. Properties refId and datasource are added at runtime and can be omitted from provisioning.

Correlation types

Correlations support two types: query (for linking to a data source query) and external (for linking to an external URL).

Correlation transformation types

Correlations support two transformation types: regex and logfmt. Regex transformations require the fields: type, field (optional), expression (required), mapValue (required). Logfmt transformations require the fields: type, field (optional).

Explore correlations editor

Correlations can be created in Explore's correlations editor as an alternative to the Administration page.

Log configurations Terraform correlation

Log configurations in Terraform enable you to configure log data correlation with entities using data source mappings and filtering options.

Trace configurations Terraform correlation

Trace configurations in Terraform enable you to configure trace data correlation with entities using data source mappings and filtering options.

Profile configurations Terraform correlation

Profile configurations in Terraform enable you to set up profile data correlation with entities using data source mappings and filtering options.

grafana_asserts_trace_config resource arguments

The grafana_asserts_trace_config Terraform resource manages Knowledge Graph trace configurations. Arguments: name (string, required) - immutable name; priority (number, required) - lower number means higher priority; default_config (bool, required) - whether this is the default configuration; data_source_uid (string, required) - DataSource UID for Tempo or trace system; match (list(object), optional) - list of match rules for entity properties; entity_property_to_trace_label_mapping (map(string), optional) - mapping of entity properties to trace labels.

Match block operators for trace configuration

The match block in grafana_asserts_trace_config supports the following operators: = (equality), <> (not equal), < (less than), > (greater than), <= (less than or equal), >= (greater than or equal), IS NULL, IS NOT NULL, STARTS WITH, CONTAINS. All match blocks require property (string) and op (string). The values parameter (list(string)) can be empty for IS NULL and IS NOT NULL operations.

Trace configuration priority order

In Knowledge Graph trace configurations, priority is a required number field where lower numbers indicate higher priority. Configurations are evaluated in ascending priority order, meaning configurations with lower priority numbers are evaluated first and take precedence.

Entity property to trace label mapping

Entity property to trace label mapping in grafana_asserts_trace_config is a map(string) that correlates entity properties with trace labels. Common mappings follow OpenTelemetry semantic conventions such as resource.k8s.cluster.name, resource.k8s.namespace, resource.container.name, resource.service.name, resource.service.namespace, span.k8s.pod.name, resource.service.type, resource.team.owner, resource.deployment.environment, resource.service.version, and resource.cloud.region.

Basic trace configuration example with Terraform

resource "grafana_asserts_trace_config" "production" { provider = grafana.asserts name = "production" priority = 1000 default_config = false data_source_uid = "grafanacloud-traces" match { property = "asserts_entity_type" op = "=" values = ["Service"] } match { property = "deployment_environment" op = "=" values = ["production", "staging"] } entity_property_to_trace_label_mapping = { "cluster" = "resource.k8s.cluster.name" "namespace" = "resource.k8s.namespace" "container" = "resource.container.name" "otel_service" = "resource.service.name" "otel_namespace" = "resource.service.namespace" } } This example shows a basic trace configuration for services in production environment.

Advanced trace configuration with multiple operators

resource "grafana_asserts_trace_config" "advanced" { provider = grafana.asserts name = "advanced" priority = 1500 default_config = false data_source_uid = "tempo-advanced" match { property = "service_type" op = "CONTAINS" values = ["web", "api"] } match { property = "deployment_environment" op = "<>" values = ["test"] } match { property = "team" op = "IS NOT NULL" values = [] } match { property = "priority_level" op = ">=" values = ["5"] } entity_property_to_trace_label_mapping = { "service_type" = "resource.service.type" "team" = "resource.team.owner" "environment" = "resource.deployment.environment" "version" = "resource.service.version" "region" = "resource.cloud.region" } } This example demonstrates trace configuration with multiple match operators including CONTAINS, not equal, IS NOT NULL, and comparison operators.

Trace configuration best practices for priority management

When managing trace configurations with Terraform: assign lower priority numbers to more specific configurations, document the reasoning behind priority assignments, and use consistent priority ranges for different configuration types. Remember that lower priority numbers mean higher priority and configurations are evaluated in ascending priority order.

Trace configuration best practices for match rules

When designing match rules for trace configurations: start with broad match rules and refine based on needs, use specific property names that exist in your entity model, test match rules with sample data before deploying, combine multiple match rules for precise entity targeting, and leverage comparison operators for numeric or version-based filtering.

OpenTelemetry semantic conventions for trace label mapping

Trace label mappings should use OpenTelemetry semantic conventions for trace label names. Common convention patterns include resource.service.name, resource.k8s.namespace, resource.k8s.cluster.name, resource.container.name, resource.service.namespace, resource.service.type, resource.team.owner, resource.deployment.environment, resource.service.version, resource.cloud.region, and span.k8s.pod.name.

Minimal trace configuration for all entities

resource "grafana_asserts_trace_config" "minimal" { provider = grafana.asserts name = "minimal" priority = 3000 default_config = false data_source_uid = "tempo-minimal" match { property = "asserts_entity_type" op = "IS NOT NULL" values = [] } entity_property_to_trace_label_mapping = { "cluster" = "resource.k8s.cluster.name" "otel_service" = "resource.service.name" "otel_namespace" = "resource.service.namespace" } } This example creates a minimal configuration that applies to all entities by matching any non-null asserts_entity_type.

Trace to logs supported data sources besides Loki

Trace to logs supports Elasticsearch, Splunk, OpenSearch, Falcon LogScale, Google Cloud Logging, and VictoriaMetrics Logs in addition to Loki.

Trace to logs correlation overview

Trace to logs correlation lets you navigate from a span in Tempo directly to matching logs in Loki, and from a log line in Loki back to its trace in Tempo. When configured, a 'Logs for this span' link appears in the trace view and a link to the configured tracing data source appears on log lines containing a trace ID.

Trace to logs requires two-way data source configuration

Trace to logs is a two data source configuration. The Tempo data source controls how Grafana queries logs when you click a span, including the time window, query, and tags to match. The Loki data source controls how trace IDs are extracted from log lines and linked back to traces. Both sides must be configured for the correlation to work in either direction.

Span start time shift and span end time shift default and purpose

The Span start time shift and Span end time shift fields expand the log query time window around the span's start and end timestamps. Span timestamps are precise to the millisecond, but log lines are often written slightly before or after the span boundary. The default value of 0 can return no logs if timestamps don't align exactly. A common starting point is -2s for start shift and 2s for end shift. These values can be increased for spans representing slow or batch operations, for example to 5s.

Default tags for trace to logs mapping

If you don't configure any tags, Grafana uses the following defaults: cluster, hostname, namespace, pod, service.name (remapped to service_name), and service.namespace (remapped to service_namespace). The automatic dot-to-underscore remapping only applies to these default tags. For custom tags, you must specify the remapped name explicitly.

Trace to logs tag mapping table for common span attributes

Common tags and their typical Loki label equivalents for trace to logs mapping are: service.name → service_name, service.namespace → service_namespace, cluster → cluster, namespace → namespace, pod → pod, hostname → hostname. The Loki label name you enter must exactly match the label name on your Loki streams. A mismatch silently breaks the correlation with no error shown.

Trace to logs cardinality guidelines for label selection

Choose labels with low cardinality for trace to logs mapping. In Loki, every unique combination of label values creates a separate stream. Labels that take on many distinct values, such as pod, host, thread, duration, traceId, or spanId, can create hundreds of thousands or even millions of streams, causing slow queries, high memory usage, and log loss at ingest. Labels with bounded, small sets of values like service_name, namespace, and cluster are good choices. If you need to query high-cardinality values like trace IDs or Pod names, store them as structured metadata instead, which are queryable without creating additional streams.

Trace to logs filter by trace ID vs filter by span ID

The 'Filter by trace ID' toggle filters Loki results to all logs for the entire trace, not just the selected span. This works well for most use cases. The 'Filter by span ID' toggle filters to logs for a single span and only works if your logs include a span ID field. These toggles are disabled when 'Use custom query' is enabled because the custom query controls filtering directly. Clicking 'Logs for this span' filters by trace ID, not span ID, so results include all logs for the entire trace that contain a matching trace ID, not only logs emitted during that specific span.

Trace to logs custom query variables

When configuring a custom LogQL expression for trace to logs, use ${__tags} to inject the mapped tag filters automatically. Example custom query: {${__tags}} | logfmt | trace_id=`${__trace.traceId}`. If a tag like pod is stored as Loki structured metadata instead of an indexed label, move it to a pipeline filter instead: {${__tags}} | pod=`${__span.tags["k8s.pod.name"]}` |= `${__trace.traceId}`.

Trace to logs derived field setup in Loki data source

To enable trace to logs navigation from Loki, configure a derived field in the Loki data source's Additional settings section. Enter a Name for the field (e.g., TraceID), select a Type and pattern to extract the trace ID from log lines. If selecting 'Regex in log line', enter a regular expression with one capture group. Enable the 'Internal link' toggle and select your Tempo data source. In the Query field, enter ${__value.raw} to pass the extracted trace ID to Tempo when the link is clicked.

Trace to logs regex patterns for common log formats

Common regex patterns for extracting trace IDs from log lines are: for traceID=<VALUE> use traceID=(\w+); for trace_id=<VALUE> use trace_id=(\w+); for JSON field "traceId": "<VALUE>" use "traceId":(\w+). If selecting Label as the type, enter a regular expression matching the label key, for example trace[_]?id to match both traceid and trace_id.

Trace to logs link appears only when configured tags exist on span

When using Loki, the 'Logs for this span' link only appears on a span if at least one of the configured tags exists on that span. If none of the tags match, the link doesn't appear and no error is shown.

Trace to logs NGINX example configuration

Example trace to logs configuration for an NGINX service in Kubernetes: Tempo data source settings - Data source: Loki, Span start time shift: -2s, Span end time shift: 2s, Tags: service.name mapped to service_name, namespace, pod, Filter by trace ID: Enabled, Filter by span ID: Disabled. Loki data source derived field settings - Name: TraceID, Type: Regex in log line, Regex: trace_id=(\w+), Internal link: Enabled (select Tempo), Query: ${__value.raw}.

Trace to logs expected behavior in Explore

After configuring both data sources for trace to logs: In Explore with the Tempo data source selected, clicking any span and then clicking 'Logs for this span' opens Loki filtered to the mapped labels (e.g., service_name=nginx, namespace=production) within the configured time window around the span. In Explore with the Loki data source selected, any log line containing a trace ID shows a link to the Tempo data source; clicking it opens the full trace in Tempo.

Trace to logs requires shared identifier in traces and logs

To configure trace to logs correlation, applications must emit traces and logs with a shared identifier such as a trace ID, service name, or both.

Trace to logs narrowing results to specific span activity

Clicking 'Logs for this span' filters by trace ID by default, so results include all logs for the entire trace containing a matching trace ID, not only logs emitted during that specific span. To narrow results to a specific span's activity, write a custom LogQL query that filters on span ID or span-specific attributes if your application logs them, for example {service_name="my-service"} | json | spanId="".

Give your agent this brain