data-source/cloudwatch

38 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

CloudWatch annotations based on alarm history

CloudWatch annotations overlay alarm history on graphs. Annotations are created from CloudWatch alarm state changes, configuration updates, and actions. Each alarm history item within the dashboard time range becomes an annotation.

CloudWatch annotation alarm matching methods

Grafana finds matching alarms using one of two methods: Match alarms by metric using cloudwatch:DescribeAlarmsForMetric, or match alarms by prefix using cloudwatch:DescribeAlarms with alarm name and action prefixes.

CloudWatch annotation query fields reference

The annotation query editor provides these fields: Region (required, AWS region to query), Namespace (required, metric namespace like AWS/EC2), Metric name (required, name of metric the alarm watches), Statistic (required, statistic the alarm uses like Average), Dimensions (optional, identifies resource like InstanceId), Period (optional, minimum interval between data points in seconds, defaults to 300 when prefix matching disabled), Enable Prefix Matching (optional, match alarms by name/action prefix instead of metric), Action (required when prefix matching enabled, match alarms whose actions start with prefix), Alarm Name (required when prefix matching enabled, match alarms whose names start with prefix).

Match alarms by metric in CloudWatch annotations

This is the default behavior when Enable Prefix Matching is off. Grafana returns annotations for any alarm attached to the specified metric. Region, Namespace, Metric name, and Statistic are all required; missing any returns an invalid annotations query error. Dimensions are optional but narrow results to alarms on a specific resource.

Match alarms by prefix in CloudWatch annotations

Enable Prefix Matching to find alarms by name and action prefix instead of by metric. Grafana requests up to 100 alarms from cloudwatch:DescribeAlarms using the Alarm Name and Action prefixes, then filters by namespace, metric name, dimensions, statistic, and period. Both Alarm Name and Action prefixes are required for the query to run.

CloudWatch annotation required IAM permissions

Annotation queries require these CloudWatch API actions: cloudwatch:DescribeAlarms, cloudwatch:DescribeAlarmsForMetric, and cloudwatch:DescribeAlarmHistory. These must be in the IAM policy attached to the role or user running the data source.

CloudWatch annotation query troubleshooting

If annotations don't appear: verify the alarm has state changes within dashboard time range (annotations come from alarm history); confirm Region, Namespace, Metric name, Statistic, and Dimensions exactly match an existing alarm; check that IAM role/user has cloudwatch:DescribeAlarms, cloudwatch:DescribeAlarmsForMetric, and cloudwatch:DescribeAlarmHistory permissions. The 'invalid annotations query' error appears when prefix matching is disabled and Region, Namespace, Metric name, or Statistic is empty; provide all four fields or enable Prefix Matching.

CloudWatch annotation include alarm details

Each annotation includes the alarm name as its title, the alarm history item type as a tag (for example StateUpdate, ConfigurationUpdate, or Action), and the history summary as its text.

CloudWatch data source health check validates both metrics and logs permissions

The data source health check validates both metrics and logs permissions. If an IAM policy only grants access to one of these (for example, metrics-only or logs-only), the health check displays a red status. However, the service you have permissions for is still usable—you can query metrics or logs based on whichever permissions are configured.

Access Denied error troubleshooting table

When encountering Access Denied errors with CloudWatch data source, the likely causes and solutions are: (1) IAM policy missing required permissions—attach appropriate policy with metrics permissions: cloudwatch:ListMetrics, cloudwatch:GetMetricData, and related permissions; or logs permissions: logs:DescribeLogGroups, logs:StartQuery, logs:GetQueryResults, and related permissions; (2) Incorrect access key or secret key—verify credentials in AWS Console under IAM > Users > your user > Security credentials, generate new if needed; (3) Credentials have expired—for temporary credentials generate new ones, for access keys verify they haven't been deactivated or deleted; (4) Wrong AWS region—verify default region in data source configuration matches where resources are located; (5) IAM role lacks permissions in queried region—AWS evaluates permissions per region, if role only has CloudWatch access in some regions, querying another returns AccessDenied, set data source Default Region to region where role has permissions; (6) Assume Role ARN is incorrect—verify role ARN format is arn:aws:iam::<account-id>:role/<role-name> and role exists in AWS Console.

InvalidClientTokenId or UnrecognizedClientException troubleshooting

When encountering InvalidClientTokenId or UnrecognizedClientException errors: (1) Access key or secret key is invalid or mistyped—verify credentials in AWS Console under IAM > Users > your user > Security credentials, re-enter in data source configuration; (2) Credentials were deactivated, deleted, or rotated—generate new credentials in AWS and update data source configuration; (3) Temporary credentials expired—generate new temporary credentials or switch to long-lived authentication method such as Assume Role; (4) Transient Grafana Cloud incident—if multiple data sources fail at same time with InvalidClientTokenId and credentials are unchanged, check Grafana Cloud status page for ongoing incident before rotating credentials.

Assume role trust policy example for CloudWatch

Example trust policy for CloudWatch Assume Role: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::<your-account-id>:user/<grafana-user>" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "<your-external-id>" } } } ] } ```

Unable to assume role troubleshooting

When authentication fails using Assume Role ARN with error referencing STS or AssumeRole: (1) Verify the trust relationship on IAM role allows Grafana credentials to assume it; (2) Check trust policy includes correct principal (user or role running Grafana); (3) If using Grafana Assume Role authentication method, confirm trust policy uses exact Grafana AWS account ID and external ID shown in instructions box on data source Settings tab; (4) If using external ID, ensure it matches exactly in both role's trust policy and Grafana data source configuration; (5) Verify base credentials have sts:AssumeRole permission; (6) Check role ARN is correct and role exists.

Save & test passes but queries return no data troubleshooting

When Save & test succeeds but panels and query editor return no data: (1) Connection test only validates assume-role step—when configuring Assume Role ARN, Save & test confirms primary credentials can perform sts:AssumeRole but doesn't verify assumed role has CloudWatch or Logs permissions, attach required query permissions to assumed role, not only to primary credentials; (2) Assumed role missing CloudWatch permissions—add metrics and logs actions such as cloudwatch:ListMetrics, cloudwatch:GetMetricData, logs:DescribeLogGroups, logs:StartQuery to assumed role's policy; (3) Permissions attached to wrong identity—confirm policy attached to role named in Assume Role ARN, not to IAM user or role providing primary credentials.

CloudWatch data source stops working after about an hour

When data source works initially then queries fail after roughly an hour with errors referencing expired or invalid security token: the cause is temporary credentials expired and weren't refreshed. Methods using temporary credentials such as Assume Role or STS issue short-lived tokens. If data source uses static temporary credentials, switch to authentication method that refreshes automatically such as Assume Role with long-lived base credentials or EKS/EC2 instance role.

Access blocked by VPC endpoint policy or service control policy troubleshooting

When credentials are valid and IAM policy grants action but requests fail with AccessDenied: (1) VPC endpoint policy restricts allowed actions—update policy on CloudWatch, Logs, EC2, or STS VPC endpoints to allow required actions such as cloudwatch:ListMetrics and cloudwatch:GetMetricData; (2) Service control policy (SCP) explicitly denies required action—review SCPs applied to account or organizational unit, explicit Deny in SCP overrides any Allow in IAM policy so action must be permitted at both levels; (3) Requests route through wrong endpoint—confirm data source region and any custom endpoint match VPC endpoints that permit required actions.

AWS SDK Default authentication troubleshooting

When data source test fails using AWS SDK Default: (1) Verify AWS credentials configured in environment where Grafana runs; (2) Check for credentials in default locations: Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY), Shared credentials file (~/.aws/credentials), EC2 instance metadata (if running on EC2), ECS task role (if running in ECS), EKS service account (if running in EKS); (3) Ensure Grafana process has permission to read credentials file; (4) For EKS with IRSA, set pod's security context to allow user 472 (grafana) to access projected token.

CloudWatch credentials file not found troubleshooting

When authentication fails with Credentials file option: (1) Create credentials file at ~/.aws/credentials for user running grafana-server service; (2) Verify file has correct permissions (0644); (3) If file exists but not working, move to /usr/share/grafana/ and set permissions to 0644; (4) Ensure profile name in data source configuration matches profile in credentials file.

CloudWatch request timeout troubleshooting

When data source test times out or queries fail with timeout errors: (1) Verify network connectivity from Grafana server to AWS endpoints; (2) Check firewall rules allow outbound HTTPS (port 443) to AWS services; (3) If using VPC, ensure proper NAT gateway or VPC endpoint configuration; (4) For Grafana Cloud connecting to private resources, configure Private data source connect; (5) Check if default region is correct—incorrect regions may cause longer timeouts; (6) Increase timeout settings if queries involve large data volumes.

CloudWatch custom endpoint configuration troubleshooting

When connection fails using custom endpoint: (1) Verify endpoint URL format is correct; (2) Ensure endpoint is accessible from Grafana server; (3) Check endpoint supports required AWS APIs; (4) For VPC endpoints, verify endpoint policy allows required actions.

CloudWatch Metrics no data or empty results troubleshooting

When query executes without error but returns no data: (1) Time range doesn't contain data—expand dashboard time range, CloudWatch metrics have different retention periods based on resolution; (2) Wrong namespace or metric name—verify namespace (e.g., AWS/EC2) and metric name (e.g., CPUUtilization) are correct; (3) Incorrect dimensions—ensure dimension names and values match AWS resources exactly; (4) Match Exact enabled incorrectly—when Match Exact enabled, all dimensions must be specified, try disabling to see if metrics appear; (5) Period too large—reduce period setting or set to auto to ensure data points returned for time range; (6) Custom metrics not configured—add custom metric namespaces in data source configuration under Namespaces of Custom Metrics.

CloudWatch metric not found troubleshooting

When expected metrics don't appear in query editor: (1) Verify metric exists in selected region; (2) For custom metrics, add namespace to Namespaces of Custom Metrics in data source configuration; (3) Check IAM policy includes cloudwatch:ListMetrics permission; (4) CloudWatch limits ListMetrics to 500 results per page—to retrieve more metrics, increase list_metrics_page_limit setting in Grafana configuration file; (5) Type metric name directly into Metric name field—for some AWS namespaces data source uses predefined metric list, so newer or less common metrics might not appear in drop-down even though can still query them, if metric exists in CloudWatch console, enter exact name; (6) Use Query Inspector to verify API request and response.

CloudWatch dimension values not loading troubleshooting

When dimension value drop-down doesn't populate: (1) Verify IAM policy includes cloudwatch:ListMetrics permission; (2) Check that namespace and metric are selected before dimension values can load; (3) For EC2 dimensions, ensure ec2:DescribeTags and ec2:DescribeInstances permissions granted; (4) Dimension values require existing metrics—if no metrics match, no values appear.

CloudWatch too many data points or API throttling troubleshooting

When queries fail with throttling errors or performance degrades with multiple panels: (1) Increase period setting to reduce number of data points; (2) Reduce time range of queries; (3) Use fewer dimensions or wildcard queries per panel; (4) Request quota increase for GetMetricData requests per second in AWS Service Quotas console; (5) Enable query caching in Grafana to reduce API calls.

CloudWatch metric math expression errors troubleshooting

When metric math expression returns errors: (1) Verify each referenced metric has unique ID set; (2) Check that metric IDs start with lowercase letter and contain only letters, numbers, and underscores; (3) Ensure all referenced metrics are in same query; (4) Verify expression syntax follows AWS Metric Math documentation; (5) Metric math expressions can't be used with Grafana alerting if they reference other query rows.

CloudWatch Logs query failed troubleshooting

When log queries return errors or no log data displayed: (1) Verify log group names are correct and exist in selected region; (2) Check IAM policy includes logs:StartQuery, logs:GetQueryResults, logs:DescribeLogGroups permissions; (3) Ensure time range contains log data; (4) Verify query syntax is valid, for CloudWatch Logs Insights QL test query in AWS Console; (5) Select correct query language (Logs Insights QL, OpenSearch PPL, or OpenSearch SQL) based on query syntax.

CloudWatch log query timeout troubleshooting

When log query runs for long time then fails with timeout error: (1) Increase Query Result Timeout setting in data source configuration (default is 30 minutes); (2) Narrow time range to reduce amount of data scanned; (3) Add filters to query to limit results; (4) Break complex queries into smaller, more focused queries; (5) For alerting, timeout defined in Grafana configuration file takes precedence.

CloudWatch log groups not appearing in selector troubleshooting

When log group selector is empty or can't find expected log groups: (1) Verify IAM policy includes logs:DescribeLogGroups permission; (2) Check log groups exist in selected region; (3) For cross-account observability, ensure proper IAM permissions for oam:ListSinks and oam:ListAttachedLinks; (4) Use prefix search to filter log groups if have many groups; (5) Verify selected account (for cross-account) contains expected log groups.

CloudWatch OpenSearch SQL query errors troubleshooting

When OpenSearch SQL queries fail with syntax errors: (1) Specify log group identifier or ARN in FROM clause: SELECT * FROM `log_group_name` WHERE `@message` LIKE '%error%'; (2) For multiple log groups, use logGroups function: SELECT * FROM `logGroups(logGroupIdentifier: ['LogGroup1', 'LogGroup2'])`; (3) Amazon CloudWatch supports only subset of OpenSearch SQL commands, refer to CloudWatch Logs documentation for supported syntax.

CloudWatch template variable returns no values troubleshooting

When template variable drop-down is empty or dashboard fails to load: (1) Verify data source connection is working; (2) Check IAM policy includes permissions for variable query type: Regions requires no additional permissions; Namespaces requires no additional permissions; Metrics requires cloudwatch:ListMetrics; Dimension Values requires cloudwatch:ListMetrics; EC2 Instance Attributes requires ec2:DescribeInstances; EBS Volume IDs requires ec2:DescribeVolumes; Resource ARNs requires tag:GetResources; Log Groups requires logs:DescribeLogGroups; (3) For dependent variables, ensure parent variables have valid selections; (4) Verify region is set correctly (use default for data source's default region).

CloudWatch multi-value template variables cause query failures troubleshooting

When queries fail selecting multiple dimension values: (1) Search expressions limited to 1,024 characters—reduce number of selected values; (2) Use asterisk (*) wildcard instead of selecting All to query all metrics for dimension; (3) Multi-valued template variables only supported for dimension values—not for Region, Namespace, or Metric Name.

CloudWatch cross-account observability queries fail troubleshooting

When can't query metrics or logs from linked accounts: (1) Verify cross-account observability configured in AWS CloudWatch console; (2) Add required IAM permissions: {"Version": "2012-10-17", "Statement": [{"Action": ["oam:ListSinks", "oam:ListAttachedLinks"], "Effect": "Allow", "Resource": "*"}]}; (3) Check monitoring account and source accounts properly linked in AWS; (4) Cross-account observability works within single region—verify all accounts in same region; (5) EC2 Instance Attributes can't be queried across accounts because they use EC2 API, not CloudWatch API.

CloudWatch API throttling errors troubleshooting

When dashboard shows Rate exceeded errors or panels intermittently fail to load: (1) Reduce frequency of dashboard refreshes; (2) Increase period setting to reduce GetMetricData requests; (3) Enable query caching in Grafana (available in Grafana Enterprise and Grafana Cloud); (4) Request quota increase in AWS Service Quotas console; (5) Consider consolidating similar queries using metric math.

CloudWatch unexpectedly high costs troubleshooting

When AWS CloudWatch costs higher than expected: (1) GetMetricData API doesn't qualify for CloudWatch API free tier; (2) Reduce dashboard auto-refresh frequency; (3) Increase period setting to reduce data points returned; (4) Use query caching to reduce repeated API calls; (5) Review variable query settings—set variable refresh to On dashboard load instead of On time range change; (6) Avoid using wildcards in dimensions when possible, as they generate search expressions with multiple API calls.

CloudWatch custom metrics don't appear troubleshooting

When custom metrics from applications or agents don't show in namespace drop-down: (1) Add custom metric namespace to Namespaces of Custom Metrics field in data source configuration; (2) Separate multiple namespaces with commas (e.g., CWAgent,CustomNamespace); (3) Verify custom metrics published to CloudWatch in selected region.

CloudWatch pre-configured dashboards not working troubleshooting

When imported dashboards show no data or variables don't load: (1) Verify data source name in dashboard matches CloudWatch data source; (2) Check dashboard's AWS region setting matches where resources located; (3) Ensure IAM policy grants access to required services (EC2, Lambda, RDS, etc.); (4) Verify resources exist and emitting metrics in selected region.

CloudWatch Application Signals trace links not appearing troubleshooting

When log entries don't show Application Signals trace links: (1) Verify Application Signals data source configured and linked in CloudWatch data source settings; (2) Ensure logs contain @xrayTraceId field; (3) Update log queries to include @xrayTraceId in fields, e.g., fields @message, @xrayTraceId; (4) Configure application to log Application Signals trace IDs per AWS Application Signals documentation.

Enable CloudWatch debug logging

To capture detailed error information for troubleshooting: (1) Set Grafana log level to debug in configuration file: [log] section with level = debug; (2) Review logs in /var/log/grafana/grafana.log (or configured log location); (3) Look for CloudWatch-specific entries that include request and response details; (4) Reset log level to info after troubleshooting to avoid excessive log volume.

Give your agent this brain