OpenTelemetry tracing configuration in .NET
To configure OpenTelemetry tracing in .NET, use the Temporalio.Extensions.OpenTelemetry extension. The Temporalio.Extensions.OpenTelemetry.TracingInterceptor class can be set as an interceptor in the client options, or provided through a Plugin if building a reusable library. When the Client is connected, spans are created for all Client calls, Activities, and Workflow invocations on the Worker. Spans are created and serialized through the server to give one trace for a Workflow Execution.
LambdaWorkerOpenTelemetry.ApplyDefaults configuration
Call LambdaWorkerOpenTelemetry.ApplyDefaults in the configure callback to enable OpenTelemetry integration. ApplyDefaults configures Temporal tracing with TracingInterceptor, creates an OTLP trace exporter and tracer provider, configures Core SDK OTLP metrics, uses AWS X-Ray-compatible trace IDs, and registers a per-invocation shutdown hook that force-flushes traces. By default, telemetry is sent to localhost:4317, which is the ADOT Lambda layer's default collector endpoint. The endpoint can be overridden with the OTEL_EXPORTER_OTLP_ENDPOINT environment variable.
LambdaWorkerOpenTelemetryOptions customization
You can customize OpenTelemetry defaults by passing a LambdaWorkerOpenTelemetryOptions object with properties: CollectorEndpoint (default localhost:4317), ServiceName, and MetricsExportInterval. Core SDK metrics export every 10 seconds by default. Set MetricsExportInterval shorter than your Lambda timeout to increase the chance that at least one metrics export happens during each invocation.
Temporalio.Extensions.Aws.Lambda.OpenTelemetry package
The Temporalio.Extensions.Aws.Lambda.OpenTelemetry NuGet package provides OpenTelemetry integration with defaults configured for the AWS Distro for OpenTelemetry (ADOT) Lambda layer. With this enabled, the Worker emits SDK metrics and distributed traces for Workflow and Activity executions. Install it with: dotnet add package Temporalio.Extensions.Aws.Lambda.OpenTelemetry
AWS Lambda X-Ray tracing setup
Enable X-Ray active tracing on the Lambda function with: aws lambda update-function-configuration --function-name <your-function-name> --tracing-config Mode=Active. The Lambda execution role must have permissions to write to X-Ray and CloudWatch. Add xray:PutTraceSegments, xray:PutTelemetryRecords, and cloudwatch:PutMetricData permissions to the execution role. Without these permissions, the Collector fails silently and no telemetry appears.
Manual OpenTelemetry configuration on Lambda
You can configure tracing and metrics manually using TracingInterceptor and TemporalRuntime: set config.ClientOptions.Interceptors to include TracingInterceptor(); set config.ClientOptions.Runtime to a new TemporalRuntime with TemporalRuntimeOptions containing TelemetryOptions with MetricsOptions using OpenTelemetryOptions pointing to your collector endpoint.
OpenTelemetry collector configuration for Lambda
The default Collector configuration does not route OpenTelemetry Protocol (OTLP) data to the traces pipeline. You must provide a custom Collector configuration that wires the OTLP receiver to both the traces and metrics pipelines. Bundle an otel-collector-config.yaml file in your Lambda deployment package with receivers for otlp (grpc and http), exporters for debug, awsxray, and awsemf, and service pipelines that route otlp to awsxray and awsemf.
Lambda Worker OpenTelemetry integration with otel sub-package
The lambdaworker/otel sub-package provides OpenTelemetry integration configured for AWS Distro for OpenTelemetry (ADOT) Lambda layer. Call otel.ApplyDefaults(opts, &opts.ClientOptions, otel.Options{}) to configure both metrics and tracing, with telemetry sent to localhost:4317 by default (ADOT Lambda layer's collector endpoint).
Lambda Worker with OpenTelemetry example code
Example Lambda Worker with OpenTelemetry integration: After setting opts.TaskQueue, call if err := otel.ApplyDefaults(opts, &opts.ClientOptions, otel.Options{}); err != nil { return err } to configure metrics and tracing. Then register Workflows and Activities as normal.
otel.ApplyMetrics and otel.ApplyTracing individual functions
If you only need metrics or tracing (not both), use otel.ApplyMetrics or otel.ApplyTracing individually instead of otel.ApplyDefaults.
IAM permissions for Lambda Worker telemetry
The Lambda execution role must have xray:PutTraceSegments, xray:PutTelemetryRecords, and cloudwatch:PutMetricData permissions to write telemetry to X-Ray and CloudWatch. Without these permissions, the Collector fails silently and no telemetry appears.
ADOT Collector configuration example for Lambda Worker
Example otel-collector-config.yaml for Lambda Worker telemetry collection: receivers.otlp with grpc endpoint localhost:4317 and http endpoint localhost:4318; exporters including awsxray (region: us-west-2), awsemf for metrics with namespace TemporalWorkerMetrics and log_group_name /aws/lambda/<function-name>; service.pipelines.traces with otlp receiver and awsxray+debug exporters; service.pipelines.metrics with otlp receiver and awsemf exporter.
ADOT Lambda layer requirements for telemetry collection
To collect Lambda Worker telemetry, attach the ADOT Collector layer to the Lambda function. The layer runs a collector sidecar that receives telemetry on localhost:4317 and forwards traces to X-Ray and metrics to CloudWatch. A custom Collector configuration (otel-collector-config.yaml) must wire the OTLP receiver to both traces and metrics pipelines; the default configuration does not route OTLP data to traces.
Spring Boot tracing integration with OpenTelemetry
To enable tracing in Spring Boot Temporal integration, set up Spring Cloud Sleuth with OpenTelemetry export. The Temporal Spring Boot integration will pick up the OpenTelemetry bean configured by spring-cloud-sleuth-otel-autoconfigure and use it for Temporal traces. Alternatively, define a custom io.opentelemetry.api.OpenTelemetry or io.opentracing.Tracer bean.
OpenTracingWorkerInterceptor for tracing on Worker
Register OpenTracingWorkerInterceptor on the Worker to enable tracing. Use WorkerFactoryOptions.newBuilder().setWorkerInterceptors(new OpenTracingWorkerInterceptor()).build(), optionally passing JaegerUtils.getJaegerOptions(type) for configuration.
OpenTracingClientInterceptor for tracing on Temporal Client
Register OpenTracingClientInterceptor on the Temporal Client to enable tracing. Use WorkflowClientOptions.newBuilder().setInterceptors(new OpenTracingClientInterceptor()).build(), optionally passing JaegerUtils.getJaegerOptions(type) for configuration.
Context propagation over Nexus Operation calls
Nexus does not use standard context propagator header structure. Instead, context is serialized into a Map<String, String> with all keys normalized to lowercase. Because Nexus uses custom format and may involve external systems, the ContextPropagator interface does not apply to Nexus headers. Context must be explicitly propagated through interceptors.
ADOT Lambda layer attachment and configuration
To collect telemetry, attach the ADOT Collector layer to your Lambda function. Java does not need a language-specific ADOT layer because the OTel SDK is included as a dependency of the temporal-aws-lambda module. The default Collector configuration does not route OpenTelemetry Protocol (OTLP) data to the traces pipeline. You must provide a custom Collector configuration that wires the OTLP receiver to both the traces and metrics pipelines.
OpenTelemetry Collector configuration for Lambda
receivers:
otlp:
protocols:
grpc:
endpoint: "localhost:4317"
http:
endpoint: "localhost:4318"
exporters:
debug:
awsxray:
region: us-west-2
awsemf:
namespace: TemporalWorkerMetrics
log_group_name: /aws/lambda/<your-function-name>
region: us-west-2
dimension_rollup_option: NoDimensionRollup
resource_to_telemetry_conversion:
enabled: true
service:
pipelines:
traces:
receivers: [otlp]
exporters: [awsxray, debug]
metrics:
receivers: [otlp]
exporters: [awsemf]
telemetry:
logs:
level: debug
metrics:
address: localhost:8888
This YAML configuration should be bundled in the Lambda deployment package and pointed to via the OPENTELEMETRY_COLLECTOR_CONFIG_URI environment variable.
OtelLambdaWorker granular configuration methods
If you only need metrics or tracing, use OtelLambdaWorker.configureMetrics, OtelLambdaWorker.configureTracing, or OtelLambdaWorker.configureFlushHook individually. To use an application-owned OpenTelemetry provider, call builder.setOpenTelemetry(...) instead. In that path, no exporters are created and the helper only installs the metrics scope, interceptors, and per-invocation flush hook.
OtelLambdaWorker configuration example
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import io.temporal.aws.lambda.LambdaWorker;
import io.temporal.aws.lambda.OtelLambdaWorker;
import io.temporal.common.WorkerDeploymentVersion;
public final class Handler implements RequestHandler<Object, Void> {
private static final RequestHandler<Object, Void> WORKER =
LambdaWorker.run(
new WorkerDeploymentVersion("my-app", "build-1"),
builder -> {
OtelLambdaWorker.configure(builder);
builder.setTaskQueue("serverless-task-queue-1");
builder.registerWorkflowImplementationTypes(SampleWorkflowImpl.class);
builder.registerActivitiesImplementations(new SampleActivitiesImpl());
});
@Override
public Void handleRequest(Object input, Context context) {
return WORKER.handleRequest(input, context);
}
}
This example shows how to integrate OtelLambdaWorker into a Lambda Handler for OpenTelemetry observability.
OtelLambdaWorker.configure behavior
OtelLambdaWorker.configure configures OpenTelemetry with OTLP trace and metric exporters, uses AWS X-Ray-compatible trace ID generation, installs an OpenTelemetry-backed metrics scope, and registers per-invocation flush hooks. By default, telemetry is sent to localhost:4317, which is the ADOT Lambda layer's default collector endpoint. The endpoint can be overridden with the OTEL_EXPORTER_OTLP_ENDPOINT environment variable.
Example: Activity with wrapped OpenAI client
This example shows how to use a wrapped OpenAI client in an Activity:
```python
from temporalio import activity
from braintrust import wrap_openai
from openai import AsyncOpenAI
@activity.defn
async def invoke_model(prompt: str) -> str:
client = wrap_openai(AsyncOpenAI(max_retries=0))
response = await client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": prompt},
],
)
return response.choices[0].message.content
```
Braintrust integration overview
Temporal's integration with Braintrust provides full observability into AI agent Workflows by tracing every LLM call, managing prompts without code deploys, and tracking costs across models. Every Workflow and Activity becomes a span in Braintrust, and every LLM call is traced with inputs, outputs, tokens, and latency.
Install Braintrust SDK with Temporal support
Install the Braintrust SDK with Temporal support using the command: uv pip install "braintrust[temporal]"
Initialize Braintrust logger before creating Worker
Initialize the Braintrust logger before creating the Temporal client or worker to ensure spans are properly connected. Use init_logger(project=os.environ.get("BRAINTRUST_PROJECT", "my-project")) from the braintrust module.
Add BraintrustPlugin to Worker
Add the BraintrustPlugin to your Worker to trace Workflow and Activity execution in Braintrust. Import BraintrustPlugin from braintrust.contrib.temporal and pass it to the Worker's plugins parameter: Worker(client, task_queue="my-task-queue", workflows=[MyWorkflow], activities=[my_activity], plugins=[BraintrustPlugin()]).
Add BraintrustPlugin to Temporal Client
Add the BraintrustPlugin to your Temporal Client to enable span context propagation, linking client code to the Workflows it starts. Pass the plugin to the Client.connect() call: client = await Client.connect("localhost:7233", plugins=[BraintrustPlugin()]).
Braintrust API key environment variable
Ensure the Worker process has access to the Braintrust API key via the BRAINTRUST_API_KEY environment variable. Set it before running the worker: export BRAINTRUST_API_KEY="your-api-key". Only the Worker process needs API credentials; the client application that starts Workflow Executions does not need the Braintrust API key.
Trace LLM calls with wrap_openai
Wrap your OpenAI client to automatically trace all LLM calls in Braintrust with inputs, outputs, token counts, and latency. Use wrap_openai(AsyncOpenAI(max_retries=0)) from braintrust. Set max_retries=0 because Temporal handles retries.
Add custom spans for business context
Add custom spans to capture business-level context like user queries, workflow inputs, and final outputs using start_span() from braintrust. Use span.log() to record input and output data.
Example: Custom span with workflow execution
This example shows how to add a custom span for business context:
```python
from braintrust import start_span
import uuid
async def run_research(query: str):
with start_span(name="research-request", type="task") as span:
span.log(input={"query": query})
result = await client.execute_workflow(
ResearchWorkflow.run,
query,
id=f"research-{uuid.uuid4()}",
task_queue="research-task-queue",
)
span.log(output={"result": result})
return result
```
Manage prompts with load_prompt
Braintrust allows managing prompts in a UI and deploying changes without code deploys. The workflow is: 1) Develop prompts in code and see results in Braintrust traces, 2) Create a prompt in the Braintrust UI from the best version, 3) Evaluate different versions using Braintrust's eval tools, 4) Deploy by pointing code at the Braintrust prompt, 5) Iterate in the UI with changes going live without code deploys.
Example: Load prompt from Braintrust in Activity
This example shows how to load a prompt from Braintrust in an Activity:
```python
import braintrust
import os
from temporalio import activity
from braintrust import wrap_openai
from openai import AsyncOpenAI
@activity.defn
async def invoke_model(prompt_slug: str, user_input: str) -> str:
# Load prompt from Braintrust
prompt = braintrust.load_prompt(
project=os.environ.get("BRAINTRUST_PROJECT", "my-project"),
slug=prompt_slug,
)
# Build returns the full prompt configuration
built = prompt.build()
# Extract system message
system_content = None
for msg in built.get("messages", []):
if msg.get("role") == "system":
system_content = msg["content"]
break
client = wrap_openai(AsyncOpenAI(max_retries=0))
response = await client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": system_content},
{"role": "user", "content": user_input},
],
)
return response.choices[0].message.content
```
Fallback prompt for Braintrust resilience
Provide a fallback prompt in code for resilience. If Braintrust is unavailable, the Workflow continues with the hardcoded prompt. Wrap load_prompt() in a try-except block and log a warning before using the default prompt.
Example: Fallback prompt implementation
This example shows how to implement a fallback prompt:
```python
DEFAULT_SYSTEM_PROMPT = "You are a helpful assistant."
try:
prompt = braintrust.load_prompt(project="my-project", slug="my-prompt")
system_content = extract_system_message(prompt.build())
except Exception as e:
activity.logger.warning(f"Failed to load prompt: {e}. Using fallback.")
system_content = DEFAULT_SYSTEM_PROMPT
```
Braintrust trace hierarchy structure
When running a Workflow with Braintrust tracing, the trace hierarchy shows: client span at the top, then temporal.workflow.{WorkflowName}, then temporal.activity.{ActivityName}, then LLM calls like Chat Completion. Example: my-workflow-request (client span) → temporal.workflow.MyWorkflow → temporal.activity.invoke_model → Chat Completion (gpt-4o).
Deep Research sample demonstrates Braintrust integration
The deep research sample from Braintrust cookbook demonstrates a complete AI agent that plans research strategies, generates search queries, executes web searches in parallel, and synthesizes findings into comprehensive reports. It shows all integration patterns: wrapped OpenAI client, BraintrustPlugin on Worker and Client, custom spans, and prompt management with load_prompt().
Set up OpenTelemetry tracing in Python SDK
To configure tracing in Python, install opentelemetry dependencies with 'pip install temporalio[opentelemetry]'. Then use the temporalio.contrib.opentelemetry.TracingInterceptor class as an interceptor argument to Client.connect(). Spans are created for all Client calls, Activities, and Workflow invocations on the Worker, and are serialized through the server to give one trace for a Workflow Execution.
Lambda execution role requires X-Ray and CloudWatch permissions
The Lambda execution role must have permissions to write to X-Ray and CloudWatch. Attach the AWSXRayDaemonWriteAccess managed policy, or add xray:PutTraceSegments, xray:PutTelemetryRecords, and cloudwatch:PutMetricData permissions. Without these permissions, the Collector fails silently and no telemetry appears.
Individual telemetry configuration options
If you only need metrics or tracing (not both), use build_metrics_telemetry_config or apply_tracing individually instead of apply_defaults.
Lambda serverless worker with OpenTelemetry example
Example Lambda handler with OpenTelemetry: from activities import hello_activity; from temporalio.common import WorkerDeploymentVersion; from temporalio.contrib.aws.lambda_worker import LambdaWorkerConfig, run_worker; from temporalio.contrib.aws.lambda_worker.otel import apply_defaults; from workflows import TASK_QUEUE, SampleWorkflow; def configure(config: LambdaWorkerConfig) -> None: config.worker_config["task_queue"] = TASK_QUEUE; config.worker_config["workflows"] = [SampleWorkflow]; config.worker_config["activities"] = [hello_activity]; apply_defaults(config); lambda_handler = run_worker(WorkerDeploymentVersion(deployment_name="my-app", build_id="build-1"), configure)
Enable X-Ray active tracing on Lambda function
Enable X-Ray active tracing on the Lambda function using the AWS CLI command: aws lambda update-function-configuration --function-name <your-function-name> --tracing-config Mode=Active
lambda_worker.otel module provides OpenTelemetry integration
The lambda_worker.otel module provides OpenTelemetry integration with defaults configured for the AWS Distro for OpenTelemetry (ADOT) Lambda layer. With this enabled, the Worker emits SDK metrics and distributed traces for Workflow and Activity executions. The ADOT Lambda layer collects this telemetry and can forward traces to AWS X-Ray and metrics to Amazon CloudWatch.
apply_defaults configures both metrics and tracing
The apply_defaults function from temporalio.contrib.aws.lambda_worker.otel configures both metrics and tracing. By default, telemetry is sent to localhost:4317, which is the ADOT Lambda layer's default collector endpoint.
ADOT Python Lambda layer must be attached
To collect OpenTelemetry telemetry, attach the ADOT Python Lambda layer to your Lambda function. The layer includes both auto-instrumentation and an OpenTelemetry Collector that receives telemetry on localhost:4317 and forwards traces to AWS X-Ray and metrics to Amazon CloudWatch.
Default Collector configuration does not route OTLP to traces pipeline
The default Collector configuration does not route OpenTelemetry Protocol (OTLP) data to the traces pipeline. You must provide a custom Collector configuration that wires the OTLP receiver to both the traces and metrics pipelines.
OpenTelemetry Collector configuration for Lambda workers
Bundle an otel-collector-config.yaml file in your Lambda deployment package with OTLP receivers (gRPC on localhost:4317 and HTTP on localhost:4318), exporters for awsxray and awsemf (with namespace: TemporalWorkerMetrics, log group: /aws/lambda/<your-function-name>, region: us-west-2, dimension_rollup_option: NoDimensionRollup, resource_to_telemetry_conversion enabled), and service pipelines routing otlp traces to awsxray and debug exporters, and otlp metrics to awsemf exporter.
Set OPENTELEMETRY_COLLECTOR_CONFIG_FILE environment variable
Set the OPENTELEMETRY_COLLECTOR_CONFIG_FILE environment variable to /var/task/otel-collector-config.yaml on the Lambda function.
Enable OpenTelemetry tracing in Ruby client
To enable OpenTelemetry tracing for clients, activities, and workflows, use `Temporalio::Contrib::OpenTelemetry::TracingInterceptor`. When creating a client, pass the interceptor in the `interceptors` array. Spans are created for all Client calls, Activities, and Workflow invocations on the Worker and are serialized through the server to give one trace for a Workflow Execution.
OpenTelemetry tracing setup example in Ruby
Example of setting up OpenTelemetry tracing with a Ruby client:
```ruby
require 'opentelemetry/api'
require 'opentelemetry/sdk'
require 'temporalio/client'
require 'temporalio/contrib/open_telemetry'
my_tracer = my_otel_tracer_provider.tracer('my-otel-tracer')
my_client = Temporalio::Client.connect(
'localhost:7233', 'my-namespace',
interceptors: [Temporalio::Contrib::OpenTelemetry::TracingInterceptor.new(my_tracer)]
)
```
OpenTelemetry interceptors package
The TypeScript SDK comes with an optional interceptor package that adds tracing with OpenTelemetry available as @temporalio/interceptors-opentelemetry. An example implementation is available in the interceptors-opentelemetry code sample.
makeOtelPlugin for Workflow code bundling
When pre-bundling Workflow code with OpenTelemetry, pass the plugin from makeOtelPlugin() so that Workflow interceptor modules are included in the bundle. Example: import { makeOtelPlugin } from '@temporalio/lambda-worker/otel'; const { plugin } = makeOtelPlugin(); const { code } = await bundleWorkflowCode({ workflowsPath: require.resolve('./workflows'), plugins: [plugin] });
OpenTelemetry integration for Lambda Worker
The @temporalio/lambda-worker/otel module provides OpenTelemetry integration with defaults configured for AWS Distro for OpenTelemetry (ADOT) Lambda layers. The applyDefaults function registers Temporal SDK interceptors for tracing and configures the Core SDK to export metrics via OpenTelemetry Protocol (OTLP). By default, telemetry is sent to localhost:4317, which is the ADOT Lambda layer's default collector endpoint.
ADOT Lambda layers for Temporal telemetry
To collect telemetry from a Lambda Worker, attach two ADOT Lambda layers: (1) The ADOT JavaScript layer for Node.js-side auto-instrumentation and trace export, and (2) The ADOT Collector layer (aws-otel-collector-amd64) to run the OTel Collector as a Lambda extension, receiving telemetry via OTLP on localhost:4317 and forwarding traces to X-Ray and metrics to CloudWatch.
Lambda function configuration for OpenTelemetry
Set the OPENTELEMETRY_COLLECTOR_CONFIG_URI environment variable to /var/task/otel-collector-config.yaml on the Lambda function. Enable X-Ray active tracing on the Lambda function with: aws lambda update-function-configuration --function-name <your-function-name> --tracing-config Mode=Active. The Lambda execution role must have permissions: xray:PutTraceSegments, xray:PutTelemetryRecords, and cloudwatch:PutMetricData.
ADOT Collector configuration for Temporal metrics and traces
The default ADOT Collector configuration does not route OTLP data to the traces pipeline. You must provide a custom Collector configuration that wires the OTLP receiver to both traces and metrics pipelines. The otel-collector-config.yaml must define receivers for OTLP (gRPC on localhost:4317 and HTTP on localhost:4318), exporters for awsxray and awsemf, and service pipelines that route otlp receiver data to both awsxray and awsemf exporters.