data-source/opentsdb

11 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

OpenTSDB annotation types

OpenTSDB supports two types of annotations: Metric annotations are attached to a specific time series (TSUID) and retrieved by querying the associated metric, used to track events affecting a specific host or service. Global annotations are not tied to any time series and apply system-wide, used to track deployments, maintenance windows, or infrastructure-wide events.

OpenTSDB annotation query retrieval

When you configure an annotation query in Grafana, it queries OpenTSDB for the specified metric and retrieves any annotations associated with that metric's time series. The query includes the globalAnnotations=true parameter, which allows Grafana to also retrieve global annotations when enabled. Grafana displays the description field from each annotation as the annotation text.

Configure OpenTSDB annotation query step-by-step

To add OpenTSDB annotations to a dashboard: Navigate to the dashboard and click Edit. Click the Add new element icon (blue plus sign) and select Annotation query. Enter a name for the annotation query. Optionally clear the Enabled checkbox if you don't want to use it right away. Select a color for annotation event markers. Select an option in the Show annotation controls in drop-down list to control where the annotation is displayed on the dashboard. Select an option in the Show in drop-down list to control which panels display the annotation. Click Open query editor to open the Annotation Query dialog box. Select the OpenTSDB data source from the Data source drop-down list. Configure the annotation query and field mappings. Optionally click Test annotation query to verify it works. Click Close when query setup is complete. Click Save, optionally enter a description of changes, click Save again, and click Exit edit.

OpenTSDB annotation query configuration fields

Annotation query configuration fields are: Name (a descriptive name for this annotation query that appears in the annotation legend), Data source (select the OpenTSDB data source), Enabled (toggle to enable or disable this annotation query), OpenTSDB metrics query (the metric name to query for annotations, for example events.deployment), Show Global Annotations (toggle to include global annotations that aren't tied to a specific time series).

OpenTSDB annotation data structure

OpenTSDB annotations have the following fields: startTime (required, Unix epoch timestamp in seconds when the event started), endTime (optional, Unix epoch timestamp in seconds when the event ended, useful for duration-based events), tsuid (optional, the time series UID to associate this annotation with; if empty, the annotation is global), description (optional, brief description of the event that displays in Grafana), notes (optional, detailed notes about the event), custom (optional, a map of custom key-value pairs for additional metadata).

Create global annotation in OpenTSDB

Use the OpenTSDB API to create a global annotation with a POST request to http://<OPENTSDB_HOST>:4242/api/annotation including startTime, description, notes, and custom fields. Example: curl -X POST http://<OPENTSDB_HOST>:4242/api/annotation -H "Content-Type: application/json" -d '{"startTime": 1609459200, "description": "Production deployment v2.5.0", "notes": "Deployed new feature flags and performance improvements", "custom": {"version": "2.5.0", "environment": "production", "deployer": "jenkins"}}'

Create metric-specific annotation in OpenTSDB

To attach an annotation to a specific time series, include the tsuid field in the POST request to http://<OPENTSDB_HOST>:4242/api/annotation. Example: curl -X POST http://<OPENTSDB_HOST>:4242/api/annotation -H "Content-Type: application/json" -d '{"startTime": 1609459200, "endTime": 1609462800, "tsuid": "000001000001000001", "description": "Server maintenance", "notes": "Scheduled maintenance window for hardware upgrade"}'. To find the TSUID for a metric, use the OpenTSDB /api/uid/tsmeta endpoint.

OpenTSDB annotation display behavior

Annotations appear as vertical lines on time series panels at the timestamps where events occurred. Hover over an annotation marker to view the annotation name from your query configuration, the event description from the OpenTSDB annotation's description field, and the timestamp. Different annotation queries can be assigned different colors in the annotation settings to distinguish between event types.

OpenTSDB annotations troubleshooting

Common annotation issues: (1) Annotations don't appear - Possible causes: Time range doesn't include annotations (expand the dashboard time range to include the annotation timestamps), Wrong metric name (verify the metric name in your annotation query matches the metric associated with annotations in OpenTSDB), Annotations are global but toggle is off (enable Show Global Annotations if your annotations don't have a TSUID), No annotations exist (verify annotations exist in OpenTSDB using the API: curl http://<OPENTSDB_HOST>:4242/api/annotation?startTime=<START>&endTime=<END>). (2) Annotation text is empty - The annotation displays but has no description text. Solution: Ensure the description field is populated when creating annotations in OpenTSDB, as Grafana displays the description field as the annotation text.

OpenTSDB annotation example: track application deployments

To monitor when deployments occur for a specific application, configure an annotation query with Name set to 'App Deployments', OpenTSDB metrics query set to 'deploy.myapp', and Show Global Annotations disabled. This query retrieves annotations attached to the deploy.myapp metric, showing deployment events for that specific application.

OpenTSDB annotation example: monitor infrastructure-wide events

To capture system-wide events such as network changes or datacenter maintenance, configure an annotation query with Name set to 'Infrastructure Events', OpenTSDB metrics query set to 'events.infrastructure', and Show Global Annotations enabled. This query retrieves both metric-specific and global annotations, providing a complete picture of infrastructure events.

Give your agent this brain