observability

45 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Emit metrics from Go Client with Prometheus

To emit metrics from the Temporal Client in Go, create a metrics handler from Client Options and specify a listener address for Prometheus. Use `client.Options{MetricsHandler: sdktally.NewMetricsHandler(newPrometheusScope(prometheus.Configuration{ListenAddress: "0.0.0.0:9090", TimerType: "histogram"}))}`. The Go SDK provides metrics handlers for Tally and OpenTelemetry.

Configure OpenTelemetry counters as monotonic

To represent Temporal SDK counters as monotonic `Int64Counter` instruments instead of the default `Int64UpDownCounter`, set `UseMonotonicCounters` to `true` when creating the OpenTelemetry metrics handler. This is available in `go.temporal.io/sdk/contrib/opentelemetry` version 0.8.0 and later. Monotonic counters let exporters and metrics backends classify Temporal SDK counters correctly. The `MetricsCounter` contract defines counters as ever-increasing, so only pass non-negative values to `client.MetricsCounter.Inc`; negative values can produce invalid or backend-dependent metric data when `UseMonotonicCounters` is enabled.

Set up OpenTelemetry metrics handler in Go

Create an OpenTelemetry metrics handler with monotonic counters enabled and register it with the client: `metricsHandler := temporalotel.NewMetricsHandler(temporalotel.MetricsHandlerOptions{Meter: otel.GetMeterProvider().Meter("temporal-sdk-go"), UseMonotonicCounters: true}); temporalClient, err := client.Dial(client.Options{MetricsHandler: metricsHandler})`.

Available tracing interceptors in Go SDK

The Go SDK provides tracing interceptors for OpenTelemetry, OpenTracing, and Datadog. Create a tracing interceptor with `opentelemetry.NewTracingInterceptor()`, `opentracing.NewInterceptor()`, or `tracing.NewTracingInterceptor()` and register it by passing it to `ClientOptions` in the `Interceptors` field.

Register tracing interceptor with client

Create a tracing interceptor and register it by passing it to ClientOptions: `c, err := client.Dial(client.Options{Interceptors: []interceptor.ClientInterceptor{tracingInterceptor}})`.

How tracing interceptors propagate trace spans

Each tracing interceptor uses its library's native propagation mechanism to serialize trace spans into Temporal headers. For example, OpenTelemetry uses its `TextMapPropagator` with the W3C TraceContext format. The SDK carries these headers across Workflow, Activity, and Child Workflow boundaries so the tracing library can reconstruct the call graph.

Get logger in workflow code

In Workflow Definitions, use `workflow.GetLogger(ctx)` to write logs. This returns a logger that can be used to emit log messages from within workflow code.

Example workflow logging

Use `logger := workflow.GetLogger(ctx)` to get a logger, then call `logger.Info("message", "key", value)` to log messages from within workflow code.

Custom logger setup with slog in Go

The Go SDK supports custom loggers via `log.NewStructuredLogger()`, which wraps Go's standard `slog.Logger` (Go 1.21+). Most modern logging libraries (zap, zerolog, logrus, etc.) can back a `slog.Handler`, making slog a universal bridge to third-party loggers. Set the custom logger in `client.Options{Logger: logger}`.

Custom logger setup with slog JSON handler

To use slog with JSON output: `slogHandler := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo}); logger := log.NewStructuredLogger(slog.New(slogHandler)); clientOptions := client.Options{Logger: logger}; temporalClient, err := client.Dial(clientOptions)`.

Bridge zap logger through slog

To use zap as a backend through slog: `zapLogger, _ := zap.NewProduction(); handler := zapslog.NewHandler(zapLogger.Core()); logger := log.NewStructuredLogger(slog.New(handler)); clientOptions := client.Options{Logger: logger}; temporalClient, err := client.Dial(clientOptions)`.

Search Attributes value types in Go

Search Attributes are represented as `map[string]interface{}` and values must correspond to the Search Attribute's value type: Bool = `bool`, Datetime = `time.Time`, Double = `float64`, Int = `int64`, Keyword = `string`, Text = `string`.

Set custom Search Attributes when starting workflow

Provide key-value pairs in `StartWorkflowOptions.SearchAttributes` when starting a workflow. Example: `searchAttributes := map[string]interface{}{"CustomerId": payload["customer"], "MiscData": payload["miscData"]}; options := client.StartWorkflowOptions{SearchAttributes: searchAttributes}; we, err := c.Client.ExecuteWorkflow(ctx, options, app.YourWorkflow, payload)`.

Upsert Search Attributes in workflow

`UpsertSearchAttributes` is used to add or update Search Attributes from within Workflow code. It will merge attributes to the existing map in the Workflow. If the same key is upserted twice, the last update wins.

Example of upserting Search Attributes

Call `workflow.UpsertSearchAttributes(ctx, attr1)` to add or update Search Attributes. For example: `attr1 := map[string]interface{}{"CustomIntField": 1, "CustomBoolField": true}; workflow.UpsertSearchAttributes(ctx, attr1); attr2 := map[string]interface{}{"CustomIntField": 2, "CustomKeywordField": "seattle"}; workflow.UpsertSearchAttributes(ctx, attr2)` results in a map with CustomIntField=2 (last update wins), CustomBoolField=true, and CustomKeywordField="seattle".

Remove Search Attribute from workflow

There is no direct support for removing a Search Attribute field. To achieve a similar effect, set the field to some placeholder value. For example, set `CustomKeywordField` to `impossibleVal`, then search with `CustomKeywordField != 'impossibleVal'` to match Workflows without the field set.

Query workflow executions with List Filter

Use `ListWorkflow()` to retrieve a list of Workflow Executions matching Search Attributes via a List Filter. Create a `workflowservice.ListWorkflowExecutionsRequest` with a Query field, such as `request := &workflowservice.ListWorkflowExecutionsRequest{Query: "CloseTime = missing"}` to return only open Workflows. Call `resp, err := temporalClient.ListWorkflow(ctx.Background(), request)` and iterate through `resp.Executions`.

Emit metrics with Prometheus or OpenTelemetry

Workers can emit metrics using telemetry options provided to Runtime.install(). Common options are: metrics.prometheus.bindAddress (address on the Worker host for Prometheus scraping) and metrics.otel.url (gRPC OpenTelemetry collector URL). Example: telemetryOptions: { metrics: { prometheus: { bindAddress: '0.0.0.0:9464' } }, logging: { forward: { level: 'DEBUG' } } }

TypeScript SDK tracing with OpenTelemetry

Use the opentelemetry-interceptors package to set up tracing of Workflows and Activities. The built-in tracing uses protobuf message headers to propagate tracing information from client to Workflow and from Workflow to successors (Continued As New), children, and Activities. All executions are linked with a single trace identifier and have proper parent-to-child span relations.

Configure global OpenTelemetry propagator with Jaeger

To extend the default Trace Context and Baggage propagators to include the Jaeger propagator, run 'npm i @opentelemetry/propagator-jaeger' and at the top level of Workflow code add: import { propagation } from '@opentelemetry/api'; import { CompositePropagator, W3CBaggagePropagator, W3CTraceContextPropagator } from '@opentelemetry/core'; import { JaegerPropagator } from '@opentelemetry/propagator-jaeger'; propagation.setGlobalPropagator(new CompositePropagator({ propagators: [new W3CTraceContextPropagator(), new W3CBaggagePropagator(), new JaegerPropagator()] }));

Activity Context logger usage

Activities can use the Activity Context logger provided by the SDK. Import log from '@temporalio/activity' and use log.info(), log.debug(), etc. The Activity Context logger funnels messages to the Runtime's logger with attributes from the current Activity context automatically included as metadata. Example: import { log } from '@temporalio/activity'; export async function greet(name: string): Promise<string> { log.info('Log from activity', { name }); return `Hello, ${name}!`; }

Workflow Context logger limitations

Workflows cannot use regular Node.js loggers because Workflows run in a sandboxed environment with no I/O capability and may be replayed at any time, causing duplicated log messages. The Temporal SDK provides a Workflow Context logger that funnels messages to the Runtime's logger.

Workflow Context logger usage

Import log from '@temporalio/workflow' and use log.info(), log.debug(), etc. The Workflow Context logger automatically includes attributes from the current Workflow context as metadata. Example: import { log } from '@temporalio/workflow'; export async function myWorkflow(name: string): Promise<string> { log.info('Log from workflow', { name }); return `Hello, ${name}!`; }

Workflow logging serialization limitation

Workflow logging uses Sinks internally and is subject to the same limitations as Sinks. Logged objects must be serializable using the V8 serialization.

Runtime's Logger routes all messages

A Temporal Worker routes all log messages to a single logger object called the Runtime's Logger. This includes messages from the Workflow Context Logger, Activity Context Logger, the TypeScript SDK Worker itself, and the underlying Temporal Core SDK (native code). By default, the Runtime's Logger writes to console (process STDOUT).

Customize Runtime Logger with DefaultLogger

Register a custom Runtime Logger when the SDK Runtime is instantiated using Runtime.install() function. Example: import { DefaultLogger, Runtime } from '@temporalio/worker'; const logger = new DefaultLogger('WARN', ({ level, message }) => { console.log(`Custom logger: ${level} — ${message}`); }); Runtime.install({ logger });

Capture Temporal Core SDK logs with telemetry filter

To capture log messages from the underlying Temporal Core SDK (native code), use telemetryOptions with a logging filter in Runtime.install(). Example: telemetryOptions: { logging: { filter: makeTelemetryFilterString({ core: 'INFO', other: 'INFO' }), forward: {} } }. The filter determines verboseness of messages from 'core' (Temporal Core SDK) and 'other' native libraries.

Log to file with Winston for collector services

Use DefaultLogger with winston to write log messages to a file for collection by services like Datadog Agent. Example: const logger = winston.createLogger({ level: 'info', format: winston.format.json(), transports: [new transports.File({ filename: '/path/to/worker.log' })] }); Runtime.install({ logger: new DefaultLogger('INFO', (entry) => { logger.log({ label: entry.meta?.activityId ? 'activity' : entry.meta?.workflowId ? 'workflow' : 'worker', level: entry.level.toLowerCase(), message: entry.message, timestamp: Number(entry.timestampNanos / 1_000_000n), ...entry.meta }); }) });

Accumulate logs for testing

To collect logs for testing and reporting, use DefaultLogger with a LogLevel and push entries to an array. Example: import { DefaultLogger, LogEntry, LogLevel } from '@temporalio/worker'; const logs: LogEntry[] = []; const logger = new DefaultLogger(LogLevel.TRACE, (entry) => logs.push(entry)); logger.debug('hey', { a: 1 }); logger.info('ho'); logger.warn('lets', { a: 1 }); logger.error('go');

Log levels in increasing order of severity

The TypeScript SDK log levels in increasing order of severity are: TRACE, DEBUG, INFO, WARN, ERROR. The Temporal SDK core normally uses WARN as its default logging level. The Worker comes with a default logger that defaults to log messages with level INFO and higher to STDERR using console.error.

Query Workflow Executions with ListWorkflowExecutions

Use WorkflowService.listWorkflowExecutions to query Workflow Executions by Search Attributes. Example: import { Connection } from '@temporalio/client'; const connection = await Connection.connect(); const response = await connection.workflowService.listWorkflowExecutions({ query: `ExecutionStatus = "Running"` }); where query is a List Filter.

Set custom Search Attributes when starting Workflow

After creating custom Search Attributes in your Temporal Service, set their values when starting a Workflow using WorkflowOptions.searchAttributes. Example: const handle = await client.workflow.start(example, { taskQueue: 'search-attributes', workflowId: 'search-attributes-example-0', searchAttributes: { CustomIntField: [2], CustomKeywordListField: ['keywordA', 'keywordB'], CustomBoolField: [true], CustomDatetimeField: [new Date()], CustomTextField: ['text'] } }); The type of searchAttributes is Record<string, string[] | number[] | boolean[] | Date[]>.

Upsert Search Attributes from Workflow code

Inside a Workflow, read from WorkflowInfo.searchAttributes and call upsertSearchAttributes to add or update Search Attributes. Example: export async function example(): Promise<SearchAttributes> { const customInt = (workflowInfo().searchAttributes.CustomIntField?.[0] as number) || 0; upsertSearchAttributes({ CustomIntField: [customInt + 1], CustomBoolField: [], CustomDoubleField: [3.14] }); return workflowInfo().searchAttributes; }. Set Search Attribute to empty array [] to delete it.

Remove Search Attribute from Workflow

To remove a Search Attribute that was previously set, set it to an empty array: []. Example: upsertSearchAttributes({ CustomIntField: [] }); Alternatively, set it to null: upsertSearchAttributes({ CustomIntField: null });

Workflow Sinks enable one-way export

Sinks enable one-way export of logs, metrics, and traces from the Workflow isolate to the Node.js environment. Sinks are written as objects with methods, registered on the Worker, then proxied in Workflow code, and it helps to share types between both.

Sinks vs Activities differences

Sinks differ from Activities in important ways: (1) A sink function doesn't return any value back to the Workflow and cannot be awaited. (2) A sink call isn't recorded in the Event History of a Workflow Execution (no timeouts or retries). (3) A sink function always runs on the same Worker that runs the Workflow Execution it's called from.

Declare sink interface with Sinks type

Explicitly declare a sink's interface for type safety. Example: import type { Sinks } from '@temporalio/workflow'; export interface CustomLoggerSinks extends Sinks { customLogger: { info(message: string): void; } }

Implement and inject Sink function into Worker

Implement sinks by injecting the Sink function into a Worker using WorkerOptions. Example: import { InjectedSinks, Worker } from '@temporalio/worker'; import { MySinks } from './workflows'; const sinks: InjectedSinks<MySinks> = { alerter: { alert: { fn(workflowInfo, message) { console.log('sending SMS alert!', { workflowId: workflowInfo.workflowId, workflowRunId: workflowInfo.runId, message }); }, callDuringReplay: false } } }; const worker = await Worker.create({ workflowsPath: require.resolve('./workflows'), taskQueue: 'sinks', sinks });

Sink callDuringReplay option controls replay execution

Specify whether an injected Sink function should be called during Workflow replay by setting the callDuringReplay option. The default is false.

InjectedSinkFunction features

The InjectedSinkFunction interface has these features: (1) The first argument is a workflowInfo object containing useful metadata about the Workflow. (2) Remaining arguments are copied between sandbox and Node.js using structured clone algorithm. (3) Sink functions cannot return values to the Workflow to prevent breaking determinism.

Sink functions contribute to Workflow Task duration

The injected sink function contributes to the overall Workflow Task processing duration. Long-running sink functions that communicate with external services may cause Workflow Task timeouts. The effect is multiplied when using callDuringReplay: true and replaying long Workflow histories because the Workflow Task timer starts when the first history page is delivered to the Worker.

Default Search Attributes in Workflow Executions

Default Search Attributes like WorkflowType, StartTime, and ExecutionStatus are automatically added to Workflow Executions.

Custom Search Attributes contain domain-specific data

Custom Search Attributes can contain domain-specific data such as customerId or numItems, allowing you to query Workflow Executions based on application-specific properties beyond default attributes.

Temporal Web tracing tracks Activity Execution

Temporal Web's tracing capabilities mainly track Activity Execution within a Temporal context. For custom tracing specific to your use case, use context propagation to add tracing logic accordingly.

Log level guidance for development vs production

During development or troubleshooting, use debug or trace logging levels. In production, use info or warn to avoid excessive log volume. An appropriate logging level depends on your specific needs.

Give your agent this brain