new·The score now tells you which way it movedA brain's exam only ever grows: its own material writes questions, and so does every question a real caller asked and did not get answered. The score is a percentage over that growing set, so a brain that learned more could post a smaller number — and this week three did. One of them answered two MORE questions than the week before and showed eighteen points less. Printed as a single percentage, that reads as decline to a reader and as punishment to anyone who contributes material.all news →
mozg.beta
Sign in

Grafana dashboards · all subjects

data-source/prometheus

75 notes in this subject, read out of this brain and free to use. This is page 2 of 2.

Fix transient alert errors

To fix transient alert errors triggering false alarms: open each affected alert rule for editing; in the alert conditions section, change Alert state if execution error or timeout from Alerting to Keep Last State; save the rule. This ensures the alert retains its previous state during transient errors and only fires when a successful evaluation confirms the threshold is breached. If errors are frequent, also investigate network stability between Grafana and Prometheus; Prometheus resource utilization (CPU, memory, disk I/O); the Query timeout setting in the data source configuration (increase if complex queries regularly exceed the limit); query complexity (simplify queries or use recording rules).

Alert rule fails to evaluate causes

If an alert rule using a Prometheus query shows evaluation errors or remains in a 'No Data' state, possible causes include: template variables in query (alert queries do not support template variables); query timeout; data source unreachable; no data in range.

Alert rule fails to evaluate solutions

To fix alert rules that fail to evaluate: replace template variables with hard-coded values (alert queries do not support template variables); simplify the query or increase the evaluation timeout, or use recording rules for complex expressions; verify the Prometheus data source connection is working by testing it in the data source settings; ensure the metric has recent data and that Prometheus is actively scraping the target.

Data source-managed rules not visible

If Prometheus alerting rules do not appear in the Grafana Alerting UI, verify that Manage alerts via Alerting UI is enabled in the data source configuration; check that Prometheus has alerting rules configured in its rule files; ensure Grafana can access the Prometheus rules API endpoint (/api/v1/rules). For Prometheus (unlike Mimir), the Alerting UI only supports viewing existing rules, not creating new ones.

Exemplars support only Prometheus data source

Support for exemplars is available for the Prometheus data source only. After you enable the functionality, exemplar data is available by default.

Prometheus Node exporter metrics endpoint

Node exporter exposes system metrics at http://localhost:9100/metrics when running locally. This endpoint can be checked to verify that Node exporter is successfully exporting metrics.

Prometheus configuration for Node exporter scraping

To configure Prometheus to scrape Node exporter running locally, add a scrape configuration with job_name 'node' and static_configs targets set to ['localhost:9100']. The metrics_path defaults to '/metrics' and scheme defaults to 'http'. Example: scrape_configs: - job_name: 'node' static_configs: - targets: ['localhost:9100']

Prometheus default port and web UI

Prometheus runs on port 9090 by default. The web UI can be accessed at http://localhost:9090 to confirm that Prometheus is running and metrics are being collected.

Prometheus remote_write configuration for Grafana Cloud

To send metrics from a local Prometheus instance to Grafana Cloud, add remote_write configuration to prometheus.yml file with the remote write endpoint URL and basic authentication credentials (username and Grafana.com API Key as password).

Prometheus configuration file location

Prometheus looks for the prometheus.yml configuration file in the current working directory by default. This behavior can be changed using the --config.file command line flag. Some installers set it to /etc/prometheus/prometheus.yml.

Windows alternative to Node exporter

On Windows systems, use windows_exporter instead of the Linux-based Node exporter to expose system metrics to Prometheus.

Starting Prometheus service

Start Prometheus with the command ./prometheus --config.file=./prometheus.yml where the config file path points to your prometheus.yml configuration file.

Checking Prometheus metrics endpoint

Send a request to http://localhost:9090/metrics to retrieve metrics and metadata about the metrics being exposed by Prometheus. This confirms Prometheus is capturing metrics.

Prometheus service unit file configuration

The Prometheus service unit file on Linux/MacOS should contain an ExecStart line that specifies the binary path (e.g., /usr/local/bin/prometheus), the configuration file path (e.g., /etc/prometheus/prometheus.yml), and the storage path (e.g., /var/lib/prometheus/). Example: ExecStart=/usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus/

Checking Prometheus readiness

After restarting Prometheus, verify it is responsive and ready to serve requests by sending a request to http://localhost:9090/-/ready. A successful response confirms Prometheus is ready.

Give your agent this brain