observability & logging

21 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Manual tracing and metrics configuration for Lambda

You can configure tracing and metrics manually using TracingInterceptor and TemporalRuntime by setting config.ClientOptions.Interceptors = new[] { new TracingInterceptor() } and config.ClientOptions.Runtime = new TemporalRuntime(new TemporalRuntimeOptions { Telemetry = new TelemetryOptions { Metrics = new MetricsOptions(new OpenTelemetryOptions("http://collector:4317")), }, }).

Temporalio.Extensions.Aws.Lambda.OpenTelemetry package purpose

The Temporalio.Extensions.Aws.Lambda.OpenTelemetry NuGet package provides OpenTelemetry integration with defaults configured for the AWS Distro for OpenTelemetry (ADOT) Lambda layer. With this enabled, the Worker emits SDK metrics and distributed traces for Workflow and Activity executions. The ADOT Lambda layer collects this telemetry and can forward traces to AWS X-Ray and metrics to Amazon CloudWatch.

LambdaWorkerOpenTelemetry.ApplyDefaults configuration

ApplyDefaults configures Temporal tracing with TracingInterceptor, creates an OTLP trace exporter and tracer provider, configures Core SDK OTLP metrics, uses AWS X-Ray-compatible trace IDs, and registers a per-invocation shutdown hook that force-flushes traces. By default, telemetry is sent to localhost:4317, which is the ADOT Lambda layer's default collector endpoint. The endpoint can be overridden with the OTEL_EXPORTER_OTLP_ENDPOINT environment variable.

LambdaWorkerOpenTelemetryOptions customization parameters

You can customize ApplyDefaults by passing a LambdaWorkerOpenTelemetryOptions object with properties: CollectorEndpoint (string, default http://localhost:4317), ServiceName (string), MetricsExportInterval (TimeSpan).

MetricsExportInterval guidance for Lambda

Core SDK metrics export every 10 seconds by default. Set MetricsExportInterval shorter than your Lambda timeout to increase the chance that at least one metrics export happens during each invocation.

ADOT Collector configuration for Lambda telemetry

The default Collector configuration does not route OpenTelemetry Protocol (OTLP) data to the traces pipeline. You must provide a custom Collector configuration that wires the OTLP receiver to both the traces and metrics pipelines. Bundle a custom otel-collector-config.yaml in your Lambda deployment package with receivers for otlp (grpc and http), exporters for debug, awsxray, and awsemf, and service pipelines for traces and metrics.

Lambda OPENTELEMETRY_COLLECTOR_CONFIG_URI environment variable

Set the OPENTELEMETRY_COLLECTOR_CONFIG_URI environment variable on the Lambda function to /var/task/otel-collector-config.yaml to point the Collector at the bundled config.

Enable X-Ray active tracing on Lambda function

Enable X-Ray active tracing on the Lambda function using: aws lambda update-function-configuration --function-name <your-function-name> --tracing-config Mode=Active

Lambda execution role permissions for telemetry

The Lambda execution role must have permissions to write to X-Ray and CloudWatch. Add xray:PutTraceSegments, xray:PutTelemetryRecords, and cloudwatch:PutMetricData permissions to the execution role. Without these permissions, the Collector fails silently and no telemetry appears.

Leverage Namespace data retention policies for security

Temporal Cloud Namespace has a Retention Period setting for workflow histories (1 to 90 days). Set an appropriate retention period to balance operational needs with security. Shorter retention means completed workflow data (history, payloads) is purged sooner, reducing the amount of sensitive data stored in the cloud at any time. Document your retention choices to align with your company's data retention policies and regulatory requirements. For retention periods over 90 days, these can be exported to your own GCS or S3 buckets.

Logging in Activities

Use activity.logger for logging in Activities instead of standard logging. Activity logging provides context-aware logging within the Activity execution context.

Logging in Workflows

Use workflow.logger.info() for logging in Workflows instead of print(). Workflow logging is replay-safe and only logs on initial execution, not during replays. This provides context-aware logging that respects Workflow determinism constraints.

Monitoring worker affinity pattern implementation

Track Activities stuck in unique queues as an indicator of Worker crash or unavailability. Monitor: (1) Total queue count to detect proliferation, (2) Activities waiting in unique queues beyond schedule_to_start_timeout, (3) Workflow retry rates and which Workers are failing.

Log configuration section

The log section is optional and contains: stdout (boolean, true if output goes to standard out), level (logging level: debug, info, warn, error, or fatal, defaults to info), and outputFile (path to output log file).

Metrics configuration providers

The metrics section configures the Cluster's metric subsystem using specific providers: statsd (not natively supported by Temporal), prometheus, and m3.

Prometheus metrics configuration

The prometheus section supports the following settings: framework (supports opentelemetry and tally, default is tally), listenAddress (address for Prometheus to scrape metrics from), and handlerPath (metrics handler path for scraper, default is /metrics).

Statsd metrics configuration

The statsd section supports: hostPort (host:port of statsd server), prefix (specific prefix in reporting to statsd), flushInterval (maximum interval for sending packets, default 300ms), and flushBytes (maximum UDP packet size, default 1432 bytes). Note: statsd is not supported natively by Temporal.

M3 metrics configuration

The m3 section supports: hostPort (host:port of M3 server), service (service tag that this client emits), queue (M3 reporter queue size, default 4k), and packetSize (M3 reporter max packet size, default 32k).

Metrics prefix and tags configuration

The metrics section supports prefix (applied to all outgoing metrics), tags (key-value pairs reported as part of every metric), and excludeTags (map from tag name to tag values list for excluding tags with unbounded cardinality, with optional whitelisting of specific values).

Pprof configuration

The pprof section has a port parameter. If specified, pprof will be initialized upon process start on the listed port.

WithCustomMetricsReporter server option

WithCustomMetricsReporter sets a custom tally metric reporter for the Temporal Server.

Give your agent this brain