Remove a span filter
To remove a filter, click the × button on the filter chip below the Filters text box.
Grafana dashboards · all subjects
data-source/tempo
25 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.
Span filter example: find errors from a service
To find all errors from a specific service, select service.name = checkout, then click the Errors pill.
Span filter example: isolate slow database calls
To isolate slow database calls, select span.name =~ db\..* (using regular expression), then select duration > 200ms.
Span filter example: narrow to critical path
To narrow results to the critical path only, click the Critical path pill, then turn off Show all spans.
Quick filter pills in Tempo
Three quick-filter pills are available below the Filters text box: Critical path (highlights spans in the longest sequence of dependent tasks that determines the trace's minimum duration), Errors (selects spans where status = error), and High latency (selects the 10% longest spans in the trace using the p90 threshold). Click a pill to toggle the filter on or off.
Adding a span filter
To add a filter: Click the Filters text box with placeholder text 'Filter by attribute or text', select a key from the dropdown (for example service.name or duration), choose an operator (for example = or >=), then select or type a value. The text box suggests values extracted from the current trace.
Show all spans toggle behavior
The Show all spans toggle is enabled by default, meaning all spans remain visible with matching spans highlighted. When turned off, non-matching spans are hidden and only spans meeting the filter criteria are displayed. The toggle is disabled when no filters are active.
Navigate between matching spans
Use the up and down arrow buttons next to the match count to jump between matching spans in the trace timeline. The match count displays the current position and total matches in the format 'X/Y matches'.
Span filters location and purpose
Span filters are located in the Filters bar above the trace timeline viewer in Tempo. They allow you to refine the spans displayed based on specific criteria such as service, duration, or attributes. Adding more filters produces more specific results.
Available span filter keys
Span filter keys include: Text search (free-text search across span content), duration (operators: =, >=, <=, >, <), service.name (the service that produced the span), span.name (the operation name of the span), status (span status: ok, error, unset), kind (span kind: server, client, internal), and any span tag, process tag, or log field key present in the trace.
Span filter operators
For most attribute keys, the available operators are: = (equals), != (not equals), =~ (regular expression match), and !~ (regular expression not match). The duration filter uses: =, >=, <=, >, <.
Span filter attributes available in TraceQL
The same attributes available as filter keys in the Filters bar (such as span.name, service.name, duration) can be used directly in TraceQL queries.
Service Graph integration with span filters
When arriving from the Service Graph, you can use service.name and status = error filters to drill into the specific service nodes showing errors.
Tempo query editor three query types
The Tempo data source query editor provides three query types for exploring tracing data: Search (visual query builder), TraceQL (text editor with complex query support), and Service Graph (visualizes service connections and RED metrics). These can be used individually or in combination by adding multiple query blocks.
TraceQL query pattern structure
TraceQL queries follow the pattern `{ conditions } | pipeline`, where conditions inside curly braces select spans and pipeline operators like `count()` or `avg()` aggregate results.
Search query builder for visual trace queries
The Search query builder provides drop-down lists and text fields to build TraceQL queries visually without needing to know TraceQL syntax. Selections automatically generate a TraceQL query that can be copied to the editor for further refinement.
TraceQL editor for complex queries
The TraceQL editor supports complex filters, structural queries across parent and child spans, and aggregations. It provides autocomplete for attribute names, scopes, and operators, and allows searching for traces by trace ID.
Service Graph view requirements
The Service Graph view requires metrics generation to be configured. It visualizes how services connect and highlights request rate, error rate, and duration (RED metrics) across connections.
Tempo search non-deterministic behavior
Tempo search is non-deterministic. When running a search, Tempo performs a massively parallel scan over the selected time range and returns the first N matching results. Even identical searches can return different results due to variations in machine load and network latency. To make results deterministic, append a `with` clause to TraceQL queries, for example `{ status = error } with (most_recent=true)`.
Tempo query editor options
The following options are available in the Options section for Search and TraceQL query types: Limit (maximum traces to return, default 20), Span Limit (maximum spans per spanset, default 3), Table Format (display focused on Traces or Spans, default Traces), Step (for metrics queries, use duration notation like 30ms or 1m), and Streaming (view partial results before query completes).
Query block management in Tempo editor
Multiple query blocks can be created using '+ Add query'. Each block contains Search, TraceQL, or Service Graph interface. Query blocks can be renamed using the edit icon, duplicated, or removed using the trash icon. Query blocks appear in the Queries and Tables navigation sections.
Tempo query block toolbar icons
Each query block has icons in the top right corner: Show data source help (displays Tempo Cheat Sheet), Create recorded query (available in Grafana Cloud and Enterprise for saving queries), Duplicate query (copies current block), and Remove query (deletes the block).
Upload and visualize JSON trace files
JSON trace files containing a single trace can be uploaded and visualized in Grafana. In Explore, select 'Import trace' next to the query type selector and upload the JSON file. If the file contains multiple traces, Grafana visualizes the first trace. Traces or Service Graphs can be downloaded through the Inspector panel's Data tab using 'Download traces' or 'Download Service Graph' buttons.
Cross-tenant TraceQL query limitations
In multi-stack Tempo data sources configured with X-Scope-OrgID header, TraceQL queries comparing multiple spansets may not correctly return all traces if the conditions are satisfied in separate tenants. This occurs because TraceQL evaluates a contiguously stored trace, and conditions satisfied in separate tenants will not return the trace.
Tempo Explore integration
The Tempo data source query editor is used within Grafana's Explore feature to query and display traces from Tempo using TraceQL. Explore provides query history and an inspector for viewing stats, inspecting queries, viewing JSON, and general information for data source queries.