Prometheus interval behavior settings
Interval behavior configuration includes: Scrape interval field (sets standard scrape and evaluation interval; default is 15s; should match typical scrape and evaluation interval in Prometheus configuration file; higher values than Prometheus configuration result in Grafana evaluating data at this interval with fewer data points) and Query timeout field (sets Prometheus query timeout; default is 60s; without timeout, complex or inefficient queries can run indefinitely consuming resources).
Prometheus data source configuration overview
Grafana includes built-in support for Prometheus without requiring a plugin. Configuration requires the Organization administrator role. The data source can be configured via UI, YAML provisioning, or Terraform. Configuration varies depending on the Prometheus-compatible database type being connected: Prometheus, Mimir, Cortex, or Thanos.
Prometheus server URL configuration
The Prometheus server URL field specifies where the Prometheus server is located. For locally running Prometheus, use http://localhost:9090. For networked servers, provide the full URL with port, for example http://prometheus.example.orgname:9090. When running Grafana and Prometheus in separate containers, localhost refers to each container's own network namespace. Use the Prometheus container's IP address, hostname (if using Docker Compose), or http://host.docker.internal:9090 to reference the host machine.
Prometheus authentication options
Three primary authentication options are available: Basic authentication (username and password), Forward OAuth identity (forwards OAuth access token and OIDC ID token if available), and No authentication. Additional deprecated options include Azure AD authentication (available in Grafana Enterprise and self-managed with azure_auth_enabled = true configured) and AWS SigV4 authentication (available in self-managed instances with sigv4_auth_enabled = true configured). Azure AD and SigV4 are deprecated in Grafana 13 and existing data sources using these methods are automatically migrated to dedicated plugins on startup.
Prometheus TLS configuration
TLS settings include: Add self-signed certificate option with CA certificate field for verifying self-signed TLS certificates; TLS client authentication option with Server name field (used to verify hostname on returned certificate), Client certificate field (generated from Certificate Authority or self-signed), and Client key field (encrypts data between client and server); Skip TLS verify toggle to bypass TLS certificate validation (not recommended unless absolutely necessary or for testing). TLS must be configured in Prometheus configuration file prior to setting these options in Grafana.
Prometheus HTTP headers and advanced HTTP settings
HTTP headers can be added with Header field (custom header name) and Value field. Advanced HTTP settings include: Allowed cookies field (specify cookies by name to forward to data source; Grafana proxy deletes all forwarded cookies by default) and Timeout field (HTTP request timeout in seconds).
Prometheus alerting configuration
Two alerting-related toggles are available: Manage alerts via Alerting UI (toggled on by default, enables data source-managed rules in Grafana Alerting; for Mimir enables managing data source-managed rules and alerts; for Prometheus only supports viewing existing rules displayed as data source-managed; can be changed via default_manage_alerts_ui_toggle in grafana.ini) and Allow as recording rules target (toggled on by default, allows data source to be selected as target destination for writing Grafana-managed recording rules; can be changed via default_allow_recording_rules_target_alerts_ui_toggle in grafana.ini).
Prometheus query editor options
Query editor configuration includes: Default editor dropdown with options Builder (visual interface for building queries) or Code (for experienced Prometheus users with PromQL expertise); users can switch between editors in the query editor UI. Disable metrics lookup toggle disables the metrics chooser and metric and label support in query field autocomplete, which can improve performance for large Prometheus instances.
Prometheus type capability comparison table
The Prometheus type setting determines API endpoints and features enabled. Capability differences by type: Regex label matching in variable queries - Prometheus: No, Mimir: Yes, Thanos: No, Cortex: Yes. Metadata API (metric type/help) - Prometheus: Yes (2.4+), Mimir: Yes, Thanos: No, Cortex: Yes. Exemplars - Prometheus: Yes (2.26+), Mimir: Yes, Thanos: No, Cortex: No. Data source-managed alert rules (read/write) - Prometheus: Read only, Mimir: Read/Write, Thanos: Read only, Cortex: Read/Write. Recording rules target (write-back) - Prometheus: Yes, Mimir: Yes, Thanos: No, Cortex: Yes. LBAC (Team-based access control) - Prometheus: No, Mimir: Yes (Cloud/GEM), Thanos: No, Cortex: No. Native histograms - Prometheus: Yes (2.40+), Mimir: Yes, Thanos: No, Cortex: No. Cortex is end-of-life; migration to Mimir is recommended. For label-based access control (LBAC), backend must be Grafana Cloud Metrics (Mimir) or Grafana Enterprise Metrics (GEM); LBAC does not work with external Prometheus-compatible endpoints like Google Managed Prometheus, self-hosted Prometheus, or Thanos even with teamHttpHeadersMimir enabled, as LBAC relies on Mimir-specific HTTP headers.
Prometheus performance settings
Performance configuration includes: Cache level dropdown with options Low, Medium, High, or None (higher cache settings recommended for high-cardinality data sources). Incremental querying (beta) toggle enables caching query results and fetching only new records instead of always requesting fresh data, reducing database and network load; Query overlap window field specifies duration (default 10m) added to each incremental request to account for delayed data ingestion. Disable recording rules (beta) toggle disables recording rules fetching and parsing, improving dashboard performance by reducing processing overhead.
Prometheus other configuration options
Additional configuration fields include: Custom query parameters field (add custom parameters to Prometheus query URL for control over query execution, examples: timeout, partial_response, dedup, max_source_resolution; join multiple parameters with &). HTTP method dropdown (select POST or GET; POST recommended and selected by default as it supports larger queries; select GET if network restricts POST). Series limit field (maximum number of returned series for all resources including metrics, labels, and values; leave empty for default limit of 40000; set to 0 to disable limit, which may cause performance issues). Use series endpoint toggle (use series endpoint /api/v1/series with match[] parameter instead of label values endpoint /api/v1/label/<label_name>/values; label values endpoint generally more performant but series endpoint supports POST method).
Prometheus exemplars configuration
Exemplars configuration (support available only for Prometheus data source; exemplar is a trace representing specific measurement within given time interval): Internal link toggle enables internal link with data source selector for choosing backend tracing data store. URL field (visible if Internal link disabled) defines external link's URL trace backend and supports ${__value.raw} macro for interpolating field value. Data source dropdown (visible when Internal link enabled) selects tracing data source for exemplar links. URL label field adds custom display label to override Label name field value. Label name field specifies field name in labels object used to obtain traceID property. Remove exemplar link button removes existing links. Multiple exemplar configurations can be added.
Prometheus private data source connect (PDC)
Private data source connect (PDC) is available only for Grafana Cloud users and allows establishing a private, secured connection between Grafana Cloud instance and data sources within private network without opening network to inbound traffic. Private data source connect network dropdown selects PDC network where data source is available. PDC supports both querying and writing to Prometheus-compatible data sources, enabling Grafana-managed recording rules to write results to Prometheus or Mimir instance behind PDC connection. If using PDC with SigV4 (AWS Signature Version 4 Authentication), PDC agent must allow internet egress to sts.<region>.amazonaws.com:443. Manage private data source connect networks link provides PDC configuration details.
Prometheus Azure authentication deprecation
Azure AD authentication on the core Prometheus data source is deprecated in Grafana 13. Data sources using Azure AD authentication are automatically migrated to the dedicated Azure Monitor Managed Service for Prometheus plugin on startup. To enable Azure AD authentication on older Grafana versions, update the [auth] section of the .ini configuration file with: azure_auth_enabled = true. Do not enable Forward OAuth identity when using Azure authentication, as both methods use the same HTTP authorization headers and the OAuth token will override Azure credentials.
Failed to connect to Prometheus error and solutions
The error message 'There was an error returned querying the Prometheus API' occurs when Grafana cannot establish a network connection to the Prometheus server. To resolve: verify the Prometheus server URL is correct in the data source configuration; check that Prometheus is running and accessible from the Grafana server; ensure the URL includes the protocol (http:// or https://); verify the port is correct (default is 9090); ensure no firewall rules block the connection; if Grafana and Prometheus run in separate containers, use container IP address or hostname instead of localhost; for Grafana Cloud, configure Private data source connect if the Prometheus instance is not publicly accessible.
Request timed out error causes and remediation
The error 'context deadline exceeded' or 'request timeout' occurs when the connection to Prometheus times out before receiving a response. Solutions include: check network latency between Grafana and Prometheus; verify Prometheus is not overloaded or experiencing performance issues; increase the Query timeout setting in the data source configuration under Interval behavior; check Grafana server timeout configuration for server-level timeout settings; reduce the time range or complexity of the query; check if network devices (load balancers, proxies) are timing out the connection.
Failed to parse data source URL error
The error 'Failed to parse data source URL' occurs when the URL entered in the data source configuration is not valid. To resolve: verify the URL format is correct (for example, http://localhost:9090 or https://prometheus.example.com:9090); ensure the URL includes the protocol (http:// or https://); remove any trailing slashes or invalid characters from the URL.
Data doesn't appear in Metrics Drilldown or Explore
If you have successfully tested a Prometheus data source connection but no metric data appears in Explore or Metrics Drilldown, verify you have selected the correct data source from the data source drop-down menu. When using remote_write to send metrics to Grafana Cloud, the data source name follows the convention grafanacloud-<stackname>-prom. Check that metrics are being actively scraped by querying 'up' in the Explore view. If using Grafana Cloud, verify the remote_write endpoint URL and credentials are correct.
PDC connectivity errors symptoms and causes
Private data source connect (PDC) error messages include 'host unreachable', 'EOF', 'network unreachable', 'connection reset by peer', and 'dial tcp: lookup ... no such host'. Prometheus queries fail intermittently or consistently when using PDC to reach a Prometheus instance behind a private network. The data source test may pass occasionally but queries fail under load. PDC tunnels traffic through an SSH connection between Grafana Cloud and the PDC agent. Connectivity failures are most commonly caused by DNS resolution issues, network configuration on the customer side, or the PDC agent's default connection limits being too low for the query volume.
PDC connectivity troubleshooting steps
To troubleshoot PDC connectivity: verify DNS resolution from the PDC agent host by running nslookup or dig for the Prometheus URL from the agent machine; check network connectivity from the agent (for example, curl http://prometheus-host:9090/-/healthy from the agent machine); increase parallel SSH connections by setting the --ssh-connections flag or PDC_SSH_CONNECTIONS environment variable to a higher value (for example, 4 or 8); check firewall rules to ensure the PDC agent's outbound SSH connection to Grafana Cloud is not being interrupted; verify the PDC agent is running and healthy by checking agent logs for connection errors or restarts; check for idle timeout issues and configure TCP keepalives if the connection drops after periods of inactivity. PDC connectivity issues are almost always caused by networking on the customer side (DNS, firewall rules, routing), not by Grafana Cloud.
Certificate verification failed error and solutions
The error 'x509: certificate signed by unknown authority' or 'certificate verify failed' occurs when Grafana cannot verify the TLS certificate presented by Prometheus. To resolve: if using a self-signed certificate, enable Add self-signed certificate in the TLS settings and add the CA certificate; verify the certificate chain is complete and valid; ensure the certificate has not expired; as a temporary workaround for testing, enable Skip TLS verify (not recommended for production).
TLS handshake error causes and remediation
The error 'TLS: handshake failure' or 'connection reset' occurs when the TLS handshake between Grafana and Prometheus fails. To resolve: verify that Prometheus is configured to use TLS; check that the TLS version and cipher suites are compatible; if using client certificates, ensure they are correctly configured in the TLS client authentication section; verify the server name matches the certificate's Common Name or Subject Alternative Name.
Unauthorized (401) error causes and solutions
The error '401 Unauthorized' or 'Authorization failed' occurs when authentication credentials are invalid or missing. To resolve: verify that the username and password are correct if using basic authentication; check that the authentication method selected matches your Prometheus configuration; if using a reverse proxy with authentication, verify the credentials are correct; for AWS SigV4 authentication, verify the IAM credentials and permissions, or consider using the Amazon Managed Service for Prometheus data source for simplified AWS authentication.
OAuth token expiration errors in alerting with GCP and Azure
The error messages 'ACCESS_TOKEN_EXPIRED' or '401 Unauthorized in alerting but not in Explore' indicate that alert rule evaluations fail intermittently while queries in Explore and dashboards work correctly. This is most common with Google Managed Prometheus (GMP) and Azure-managed Prometheus endpoints using OAuth/OIDC authentication. The cause is that the Grafana alerting backend and the interactive query path (Explore, dashboards) handle credential refreshes differently. The alerting evaluator can use a cached OAuth token beyond its expiry window due to a token staleness check issue in the Prometheus data source.
OAuth token expiration fix for Google Managed Prometheus
To fix OAuth token expiration errors with Google Managed Prometheus (GMP): use the GCP datasource-syncer pattern by running a sidecar process that refreshes OAuth tokens and updates the Grafana data source credentials on a schedule shorter than the token lifetime (typically every 45 minutes for a 60-minute token). Alternatively, use the Google Cloud Monitor data source with its built-in GCP credential management rather than pointing the core Prometheus data source at a GMP endpoint. If running on GKE with Workload Identity, ensure the Kubernetes service account token refresh is functioning and the projected token volume is mounted correctly.
OAuth token expiration fix for Azure Managed Prometheus
To fix OAuth token expiration errors with Azure Managed Prometheus: verify the Azure AD app registration's client secret has not expired; if using Managed Identity, ensure the Grafana instance's identity has the Monitoring Data Reader role on the Azure Monitor workspace; check that Forward OAuth identity is not enabled alongside Azure AD authentication, as both use the same HTTP authorization headers and conflict with each other.
General OAuth token expiration troubleshooting steps
To troubleshoot OAuth token expiration issues in Prometheus alerting: check the Grafana server logs for token refresh errors around the time alerts fail; verify the data source test (Save & test) passes, which confirms current credentials are valid but does not guarantee the alerting backend has a fresh token; if the issue persists, set Alert state if execution error or timeout to Keep Last State to prevent false alarms while investigating. This token caching behavior is a known issue that has received code fixes in recent Grafana releases, so upgrading may resolve it.
LBAC not restricting data on non-Mimir backends
If you have enabled teamHttpHeadersMimir and configured Team LBAC rules but users can still see all metrics regardless of their team assignments, Label-Based Access Control (LBAC) for the Prometheus data source only works when the backend is Grafana Cloud Metrics (Mimir) or Grafana Enterprise Metrics (GEM). It does not work with Google Managed Prometheus, self-hosted Prometheus, Thanos, or other Prometheus-compatible endpoints. The LBAC enforcement relies on Mimir-specific HTTP headers (X-Scope-OrgID and team-scoped label matchers) that other backends ignore.
LBAC workarounds for non-Mimir backends
If LBAC is not restricting data on non-Mimir Prometheus backends, verify your backend is Grafana Cloud Metrics or GEM. If you are using a different Prometheus-compatible backend, LBAC is not supported. For Google Managed Prometheus or other external endpoints, use data source permissions to control which teams can access the data source entirely, rather than per-label access control. If you need per-label restrictions on a non-Mimir backend, consider proxying through a Mimir instance or using a separate data source per team with different credentials scoped to the appropriate data.
Azure AD or SigV4 authentication options not appearing
If the Azure AD or SigV4 authentication options do not appear in the authentication drop-down when configuring the Prometheus data source, these authentication methods require server-side feature flags that are not enabled by default, particularly on Grafana Cloud.
Enable Azure AD or SigV4 authentication options
To enable Azure AD or SigV4 authentication options for Prometheus: for Grafana Cloud, contact Grafana Support to request the feature flag be enabled for your stack (this is not a self-service setting). For self-managed Grafana, add the required setting to your Grafana configuration file: for Azure AD, set azure_auth_enabled = true under [auth]; for SigV4, set sigv4_auth_enabled = true under [auth]. For Amazon Managed Prometheus, consider using the dedicated Amazon Managed Service for Prometheus data source instead, which handles AWS authentication natively without needing the SigV4 flag.
Forbidden (403) error causes and solutions
The error '403 Forbidden' or 'Access denied' occurs when the authenticated user does not have permission to access the requested resource. To resolve: verify the user has read access to the Prometheus API; check Prometheus security settings and access control configuration; if using a reverse proxy, verify the proxy is not blocking the request; for AWS Managed Prometheus, verify the IAM policy grants the required permissions, or consider using the Amazon Managed Service for Prometheus data source for simplified AWS authentication.
Query syntax error causes and solutions
The error 'parse error: unexpected character' or 'bad_data: 1:X: parse error' occurs when the PromQL query contains invalid syntax. An alternative cause is that a proxy between Grafana and Prometheus requires authentication. When proxy authentication fails, the proxy redirects the request to an HTML authentication page, and Grafana cannot parse the HTML response, which results in a parse error appearing to be a query issue but actually being a proxy authentication issue. To resolve: check query syntax for typos or invalid characters; verify metric names and label names are valid identifiers; ensure string values in label matchers are enclosed in quotes; use the Prometheus expression browser to test the query directly; if you have a proxy between Grafana and Prometheus, verify proxy authentication is correctly configured and check proxy logs for authentication failures or redirects.
Query returns no data for a metric
If a query returns no data and the visualization is empty, the specified metric does not exist in Prometheus or there is no data for the selected time range. To resolve: verify the metric name is spelled correctly; check that the metric is being scraped by Prometheus; use the Prometheus API to browse available metrics at /api/v1/label/__name__/values; use the target metadata API to verify which metrics a target exposes; verify the time range includes data for the metric.
Query timeout limit exceeded error
The error 'query timed out in expression evaluation' or 'query processing would load too many samples' occurs when the query took longer than the configured timeout limit or would return too many samples. To resolve: reduce the time range of the query; add more specific label filters to limit the data scanned; increase the Query timeout setting in the data source configuration; use aggregation functions like sum(), avg(), or rate() to reduce the number of time series; increase the query.timeout or query.max-samples settings in Prometheus if you have admin access.
Too many time series error
The error 'exceeded maximum resolution of 11,000 points per timeseries' or 'maximum number of series limit exceeded' occurs when the query is returning more time series or data points than the configured limits allow. To resolve: reduce the time range of the query; add label filters to limit the number of time series returned; increase the Min interval or Resolution in the query options to reduce the number of data points; use aggregation functions to combine time series; adjust the Series limit setting in the data source configuration under Other settings.
Memory limit exceeded for high-cardinality queries
The error messages 'max-estimated-memory-consumption-per-query limit exceeded', 'query requires too much memory', or 'max samples limit reached' occur when queries against high-cardinality metrics (thousands of unique label combinations) over long time ranges (days or weeks) fail. High-cardinality metrics include those with labels like pod, request_id, or user_id. Prometheus and Mimir enforce per-query memory and sample limits to protect the system from resource exhaustion. Short time ranges may work but expanding the range causes the failure because high-cardinality metrics multiplied by long time ranges produce result sets that exceed these limits.
High-cardinality query solutions: reduce scope
To address memory limit exceeded errors with high-cardinality Prometheus queries, reduce query scope by: shortening the time range; adding label matchers to select fewer series (for example, filter to a specific namespace or job); increasing the Min step or using a larger step interval to reduce the number of data points per series.
High-cardinality query solutions: recording rules example
To address memory limit exceeded errors with high-cardinality Prometheus queries, create recording rules that pre-compute the aggregation needed. Example YAML configuration: groups: [{ name: 'aggregations', rules: [{ record: 'service:http_requests_total:rate5m', expr: 'sum(rate(http_requests_total[5m])) by (service)' }] }]. Query the pre-aggregated metric instead of the high-cardinality source metric.
High-cardinality query solutions: Adaptive Metrics
If you are on Grafana Cloud and experiencing memory limit errors with high-cardinality Prometheus queries, Adaptive Metrics automatically identifies and aggregates high-cardinality metrics that are not being queried at full resolution, reducing storage and query costs.
High-cardinality query solutions: dashboard restructuring
To address memory limit exceeded errors with high-cardinality Prometheus queries by restructuring dashboards: use template variables to scope queries to a subset of labels rather than querying all series at once; split long-range overviews (weekly/monthly) into separate panels that use pre-aggregated recording rules; use the Max data points setting in the query options to cap the resolution for overview panels.
High-cardinality query solutions: increase limits
If you have admin access to Prometheus or Mimir and need to address memory limit exceeded errors, you can increase limits in the server configuration. For Prometheus, use the --query.max-samples flag. For Mimir, use -querier.max-samples or -querier.max-estimated-memory-consumption-per-query. Note: increasing limits allows larger queries to succeed but also increases the risk of resource exhaustion. Prefer reducing query scope or using recording rules over raising limits.
Invalid function or aggregation error
The error 'unknown function' or 'parse error: unexpected aggregation' occurs when the query uses an invalid or unsupported PromQL function. To resolve: verify the function name is spelled correctly and is a valid PromQL function; check that you are using the correct syntax for the function; ensure your Prometheus version supports the function you are using; refer to the PromQL functions documentation for available functions.
rate() or increase() returning unexpected values causes
The symptom increase() returns fractional values on integer counters, rate() shows an ever-increasing value instead of a steady per-second rate, or counter resets cause large spikes in visualizations. Possible causes: increase() fractional values (expected behavior — Prometheus uses linear interpolation); rate() grows over time (multiple instances write to the same series without unique labels); counter reset spikes after pod restarts; values differ between edit mode and dashboard (panel width affects $__interval which affects rate() window calculations).
rate() or increase() solutions table
Solutions for rate() or increase() returning unexpected values: for increase() fractional values, use ceil() or floor() if you need integers. For rate() growing over time, ensure each target has unique instance/pod labels and aggregate with sum by. For counter reset spikes after pod restarts, use $__rate_interval or a longer range vector to smooth spikes and investigate frequent restarts as root cause. For values differing between edit mode and dashboard, set a Min step on the query to enforce consistent interval calculation.
Aggregation by labels with dots bug
Prior to Grafana 13, there was a bug where labels with dots in their names (for example, container.name) were not handled correctly during aggregation operations like sum by or avg by, causing incorrect or incomplete results.
Fix aggregation by labels with dots
To fix queries that aggregate by label names containing dots: upgrade to Grafana 13 or later, which correctly handles labels with dots in aggregation queries. Verify your query uses the correct label name with dots (for example, sum by (container.name) (metric_name)).
Invalid Prometheus type error
If you experience unexpected behavior when querying metrics or labels, the Prometheus type setting may not match your actual Prometheus-compatible database. The Prometheus type setting is found under Performance in the data source configuration and should be set to the correct database type: Prometheus, Cortex, Mimir, or Thanos. Different database types support different APIs, so setting this incorrectly causes unexpected behavior.
Scrape interval mismatch symptoms and causes
If data appears sparse or rate() queries return no data or incomplete results, the Scrape interval setting in Grafana may not match the actual scrape interval in Prometheus. This especially affects rate() queries, which require at least two data points within the specified time window. For example, if the actual scrape interval is 5 minutes but Grafana uses the default (15 seconds for OSS, 1 minute for Grafana Cloud), a query like rate(http_requests_total[1m]) returns no data because there are no data points within that 1-minute window.
Fix scrape interval mismatch
To fix scrape interval mismatch issues: check your Prometheus configuration file for the scrape_interval setting; update the Scrape interval in the Grafana data source configuration under Interval behavior to match; use $__rate_interval instead of hard-coded time windows in rate() queries, as this variable automatically adjusts based on your scrape interval.
$__rate_interval returns no data or incorrect values
$__rate_interval is calculated as max($__interval + scrape_interval, 4 * scrape_interval). If any input to this formula is incorrect, the resulting window is wrong — either too small (no data) or inconsistent across contexts. Common causes include: data source scrape interval left at default 15s while actual Prometheus scrape interval is longer; query works in edit mode but shows gaps on dashboard; LBAC-enabled data source doesn't inherit scrape interval; using $__rate_interval in recording rules or alerting (unsupported context).
$__rate_interval troubleshooting steps
To troubleshoot $__rate_interval issues: set the Scrape interval under Interval behavior in the data source configuration to match your Prometheus scrape_interval; if the query works in edit mode but shows gaps on the dashboard, panel size affects $__interval, so set a Min step on the query to enforce a consistent floor; for LBAC-enabled data sources that don't inherit scrape interval, set the Min step explicitly on each query panel; do not use $__rate_interval in recording rules or alerting contexts (use fixed intervals instead).
Debug $__rate_interval current value
To debug the current value of $__rate_interval in a Prometheus query: open the query inspector in a panel (click the panel title, then Inspect > Query); look at the expanded query sent to Prometheus — the actual interval value replacing $__rate_interval is visible in the request; compare this value against your actual scrape interval. If it is smaller than your scrape interval, you need to configure the data source scrape interval or set a Min step.
Slow query performance symptoms and causes
If queries take a long time to execute, dashboards are slow to load, or the loading spinner persists, queries may be scanning too much data, the Prometheus server may be overloaded, or the network connection may be slow.
Slow query performance solutions
To address slow Prometheus query performance: reduce the time range of your query; add more specific label filters to limit the data scanned; increase the Min step in the query options to reduce the number of data points; check Prometheus server performance and resource utilization; enable Disable metrics lookup in the data source configuration for large Prometheus instances; enable Incremental querying (beta) to cache query results; use recording rules to pre-aggregate frequently queried data; for high-cardinality metrics, refer to memory limit exceeded troubleshooting.
Data appears delayed or missing recent points
If the visualization does not show the most recent data even after refreshing, causes include scrape timing, clock drift, or dashboard refresh settings. To resolve: check the dashboard time range and refresh settings; verify the Scrape interval is configured correctly in the data source settings; ensure Prometheus has finished scraping the target (there is a delay between the scrape interval and data availability); check for clock synchronization issues between Grafana and Prometheus (use NTP); for rate() and similar functions, the most recent partial scrape interval will not have enough data points — this is expected.
Exemplars not showing
If exemplar data does not appear on graphs even though you expect it, exemplars require specific configuration in both the data source and the query editor. To resolve: verify that exemplars are configured in the data source settings under Exemplars (at least one exemplar link must be defined); check that your Prometheus version supports exemplars (2.26 or later); ensure your instrumented application is actually sending exemplar data with metrics; verify the tracing data source is correctly configured for the exemplar link; enable the Exemplars toggle in the query editor for the specific query; exemplars only appear with Range query type, not Instant.
Annotations not appearing causes
If you have configured a Prometheus annotation query but no annotations appear on your dashboard, possible causes include: query returns no data in the current time range; query returns continuous data with too many annotations (every returned data point creates an annotation, so hundreds of points may render but be too dense to see); wrong data source selected; annotation is disabled; time range mismatch.
Annotations not appearing solutions
To fix Prometheus annotations not appearing: verify the query returns results in Explore for the dashboard's time range; if the query returns continuous data with too many annotations, increase the Min step or refine the query to only return data at event moments; verify the correct Prometheus data source is selected in the annotation configuration; check that the annotation toggle is enabled (eye icon) in the dashboard's annotation settings; expand the dashboard time range to include the events you expect to see. Note: Prometheus annotations create a marker for every data point returned by the query with no automatic filtering of zero values. If you only want annotations at specific moments, your PromQL expression must return data only at those times (for example, using > 0, changes() > 0, or the ALERTS metric).
Transient alert errors triggering false alarms
The error messages sse.dependencyError, sse.dataQueryError, 'context deadline exceeded', or 'i/o timeout' indicate transient alert errors. Alert rules intermittently fire due to execution errors rather than genuine threshold breaches, causing on-call teams to receive false positive notifications. Alert state history shows error states caused by transient backend issues (network blips, HTTP 502/500 responses, timeouts) rather than actual metric conditions being met. By default, when an alert rule encounters an execution error or timeout, Grafana sets the alert state to Alerting, which fires the alert. Transient connectivity issues between Grafana and Prometheus trigger this behavior even though the underlying metric has not crossed its threshold.