architecture/workers

117 notes in this subject, read out of this brain and free to use. This is page 2 of 2.

Serverless Worker invocation flow

A Task is submitted to the Matching Service, which attempts a sync match to route the Task directly to an available Worker. If a Worker is available, the Task is routed to it. If no Worker is available, the Matching Service pushes a signal to the WCI, which triggers the configured compute provider to start a Worker. The Serverless Worker starts, creates a Temporal Client, begins polling the Task Queue, and processes Tasks.

Task Queue configuration when constant cannot be shared

When it is not possible to define the Task Queue name in a shared constant, such as when the Client is running on a different system or is implemented in a different programming language, alternative approaches must be used to ensure the Task Queue names match between client and worker.

Task Queue identification and matching

The Temporal Service maintains a set of Task Queues, which Workers poll to receive work. Each Task Queue is identified by a name, which is provided to the Temporal Service when launching a Workflow Execution. The Task Queue name must match between the client that starts the workflow and the worker that processes it.

Task Queue mismatch creates separate queues

Task Queues are created dynamically when first used. A mismatch between the Task Queue name specified when starting a workflow and the Task Queue name the worker listens on does not result in an error. Instead, it creates two different Task Queues. Consequently, the Worker will not receive any tasks from the Temporal Service and the Workflow Execution will not progress.

Task Routing flow control

A Worker that consumes from a Task Queue asks for an Activity Task only when it has available capacity, so it is never overloaded by request spikes. If Activity Tasks get created faster than Workers can process them, they are backlogged in the Task Queue.

Task Routing definition

Task Routing is when a Task Queue is paired with one or more Workers, primarily for Activity Task Executions. This could also mean employing multiple Task Queues, each one paired with a Worker Process.

Task Routing for host-specific processing

When an Activity Task must be routed to a specific Worker Process or Worker Entity, such as in file processing or machine learning model training, use Task Routing with dedicated Task Queues. For example, if a download Activity executes on a host, subsequent processing and upload Activities that depend on that file must run on the same host.

Task Routing for heterogeneous workers

Workers might exist on different types of hardware, such as GPU boxes versus non-GPU boxes. In this case, each type of box would have its own Task Queue and a Workflow can pick one to send Activity Tasks.

Task Routing for cached data

Some Activities load large datasets and cache them in the process. Activities that rely on those datasets should be routed to the same process by assigning a unique Task Queue for each Worker Process involved.

Task Routing for multiple priorities

If a use case involves more than one priority, you can create one Task Queue per priority with a Worker pool per priority. Alternatively, you can use Task Queue Priority, which lets you assign priority levels to Tasks within a single Task Queue, avoiding the overhead of managing multiple Task Queues while ensuring higher-priority Tasks are processed first.

Task Routing for versioning

Task Routing is the simplest way to version code. If you have a new backward-incompatible Activity Definition, start by using a different Task Queue. Alternatively, you can use Worker Versioning, which lets you tag Workers with a version and route Workflow and Activity Tasks to specific versions without requiring separate Task Queues.

Worker Session definition

A Worker Session is a feature provided by some SDKs that provides a straightforward API for Task Routing to ensure that Activity Tasks are executed with the same Worker without requiring you to manually specify Task Queue names. It also includes features like concurrent session limitations and Worker failure detection.

Task Routing for specific environments

To send Activity Tasks to a dedicated environment, use a dedicated Task Queue. This is useful when you need to execute Activities in specific environments or on specific hosts.

Task Routing throttling

The rate at which each Activity Worker polls for and processes Activity Tasks is configurable per Worker. Workers do not exceed this rate even if it has spare capacity. There is also support for global Task Queue rate limiting that works across all Workers for the given Task Queue, frequently used to limit load on a downstream service that an Activity calls into.

Go vs Core SDK behavior when task timeout passes with running Activity

Go and Core SDKs behave differently when task timeout passes and the Activity or Local Activity is still running. In Go, the shutdown completes but the Activity will continue to run and use a slot. In Core, the Worker shutdown will not complete while the Activity completes.

Long or hung Local Activities may block shutdown

Long or hung Local Activities may block Worker shutdown unless they fail early. It is recommended that Local Activities should generally be used for short Activities to avoid blocking shutdown.

Activities and Local Activities must honor context cancellation

Developers should ensure that Activities and Local Activities honor context cancellation or other shutdown signals to facilitate proper Worker shutdown.

Worker shutdown stops polling and begins shutdown sequence

When a Worker shuts down, it stops polling for new tasks and begins the shutdown sequence. In-flight Workflow Tasks may fail if they are not completed in time after exhausting Retry Policy attempts.

Two types of Worker shutdown behavior

Worker shutdown behavior depends on whether graceful shutdown is configured. The two types are: graceful shutdown (which configures how much time a Worker has to complete its current task) and non-graceful period shutdown (which occurs when no graceful period is specified or the shutdown exceeds the configured graceful period).

Graceful shutdown configuration parameters by SDK

Graceful shutdown configures how much time a Worker has to complete its current task before shutting down. Core SDKs use 'graceful_shutdown_period', Go uses 'WorkerStopTimeout', and Java uses 'shutdown()' followed by 'awaitTermination(timeout, unit)'.

Activity context indicates Worker shutdown

An Activity is able to determine that the Worker it is running on is being shut down through the Activity context.

In-flight Workflow Tasks during graceful shutdown

During graceful shutdown, any in-flight Workflow Tasks are attempted to be completed. The only reason they may not complete immediately is if Workflow code is incorrectly blocking or because of Local Activities.

Activities completion during graceful shutdown period

Activities are allowed to complete during the graceful shutdown period.

Local Activities during graceful shutdown

Because Local Activities run within a Workflow Task, current and future Local Activities within the same Workflow Task are allowed to run and complete during graceful shutdown, assuming there is no additional command to yield to.

Non-graceful shutdown behavior

During non-graceful period shutdown, the Activity context is canceled and the Worker finishes shutdown when the current Workflow Task completes with either success or failure.

Local Activities cancel signal during non-graceful shutdown

During non-graceful period shutdown, the Local Activity is sent a cancel signal, then the Workflow Task heartbeats stop, and no new Local Activities will be retried or started. The Worker still waits for the current Workflow Task to complete, meaning it can eventually hit the Workflow Task or execution timeout unless another Worker is spun up.

Nexus Task completion and retry behavior

A Nexus Task Execution completes when a Worker responds to the Temporal Service with either a RespondNexusTaskCompleted or RespondNexusTaskFailed call, or when the Task times out. The Temporal Service interprets the outcome and determines whether to retry the Task or record the progress in a History Event: either NexusTaskCompleted or NexusTaskFailed.

Task definition and types

A Task is a unit of work for Workers. The Temporal Service places Tasks on Task Queues, and Workers poll for and process them to advance Workflows, run Activity attempts, or handle Nexus requests. There are three types of Tasks: Workflow Task, Activity Task, and Nexus Task.

Nexus Task definition

A Nexus Task delivers one Nexus request to start or cancel a Nexus Operation.

Nexus Task Execution definition and lifecycle events

A Nexus Task Execution occurs when a Worker uses the context provided from the Nexus Task and executes an action associated with a Nexus Operation which commonly includes starting a Nexus Operation using its Nexus Operation handler plus many additional actions that may be performed on a Nexus Operation. The NexusOperationScheduled Event corresponds to when the Temporal Service records the Workflow's intent to schedule an operation. The NexusOperationStarted Event corresponds to when the Worker picks up the Nexus Task from the Task Queue, starts an asynchronous Nexus Operation, and returns an Operation token to the caller indicating the asynchronous Nexus Operation has started. Either NexusOperationCompleted or one of the other Closed Nexus Operation Events corresponds to when the Nexus Operation has reached a final state.

Nexus Task idempotency and effectively once semantics

A Nexus Operation Execution appears to the caller Workflow as a single RPC, while under the hood the Temporal Service may issue several Nexus Tasks to attempt to start the Operation. A Nexus Operation Handler implementation should be idempotent. The WorkflowRunOperation provided by the SDK leverages Workflow ID based deduplication to ensure idempotency and provide an 'effectively once' experience.

Worker terminology: Program, Entity, Process

In Temporal documentation, the term 'Worker' can refer to three distinct concepts: Worker Program (static code defining Worker constraints), Worker Entity (individual worker within a process listening to a specific task queue), or Worker Process (process that polls task queues and executes tasks). Documentation aims to be explicit about which is meant in each context.

Worker Program definition

A Worker Program is the static code that defines the constraints of a Worker Process. It is developed using the APIs of a Temporal SDK.

Worker Entity listens to single Task Queue

A Worker Entity is an individual worker within a Worker Process that listens to and polls on a single Task Queue. A Worker Entity contains a Workflow Worker and/or an Activity Worker, which makes progress on Workflow Executions and Activity Executions respectively.

Worker scalability: handling many open executions

A single Worker can handle millions of open Workflow Executions despite cache size or thread count limitations. Workers are stateless, so blocked Workflow Executions can be safely removed and resurrected later on the same or different Worker. The trade-off is added latency, but this allows a Worker to handle update rates with high concurrency.

Worker Identity default format

By default, Temporal SDKs set a Worker Identity to ${process.pid}@${os.hostname()}, combining the Worker's process ID and the hostname of the machine running the Worker. The Worker Identity is visible in Event History and in the list of pollers on a Task Queue, and helps identify specific Worker instances for debugging.

Worker Identity limitations in Docker and cloud environments

The default Worker Identity format has limited usefulness in cloud environments: Docker containers typically have process ID 1, cloud environments like Amazon ECS use random hostnames, and ephemeral IP addresses may change over time. These factors make the default identity format unreliable for uniquely identifying Worker instances.

Worker Identity best practices

For reliable Worker identification, ensure the Worker Identity can be linked back to the corresponding machine, process, execution context, or log stream. Recommended approaches include: using environment-specific identifiers (e.g., ECS Task ID), including relevant context like deployment environment or region, ensuring uniqueness within the system, and keeping it concise and readable.

Worker Process definition and protocols

A Worker Process is responsible for polling a Task Queue, dequeueing a Task, executing code in response to the Task, and responding to the Temporal Service with results. Formally, a Worker Process implements the Task Queue Protocol and the Task Execution Protocol. It can be a Workflow Worker Process (implements Workflow Task Queue Protocol and Workflow Task Execution Protocol) or Activity Worker Process (implements Activity Task Queue Protocol and Activity Task Processing Protocol).

Worker Process capabilities

A Workflow Worker Process can listen on an arbitrary number of Workflow Task Queues and execute an arbitrary number of Workflow Tasks. An Activity Worker Process can listen on an arbitrary number of Activity Task Queues and execute an arbitrary number of Activity Tasks. A single Worker Process can be both a Workflow Worker Process and an Activity Worker Process.

Worker Processes are external to Temporal Service

Worker Processes are external to a Temporal Service. Temporal Application developers are responsible for developing Worker Programs and operating Worker Processes. The Temporal Service does not execute Workflow and Activity Definitions on its own machines; it only orchestrates State Transitions and provides Tasks to available Worker Entities. A production Temporal Application typically has a fleet of Worker Processes running on hosts external to the Temporal Service.

Multiple Worker Entities per Worker Process

Many SDKs support the ability to have multiple Worker Entities in a single Worker Process. Each Worker Entity can listen to only a single Task Queue, but if a Worker Process has multiple Worker Entities, the Worker Process can listen to multiple Task Queues.

Activity Worker resource requirements

Worker Processes executing Activity Tasks must have access to any resources needed to execute the actions defined in Activity Definitions, such as: network access for external API calls, credentials for infrastructure provisioning, and specialized GPUs for machine learning utilities.

Worker Process failures do not cause Workflow Definition failures

You do not handle Worker Process failure or restarts in a Workflow Definition. Workflow Function Executions are completely oblivious to the Worker Process in terms of failures or downtime. The Temporal Platform ensures that the state of a Workflow Execution is recovered and progress resumes if there is an outage of either Worker Processes or the Temporal Service itself. The only reason a Workflow Execution might fail is due to the code throwing an error or exception.

Workflow Cache definition and purpose

A Workflow Cache is an in-memory LRU (least recently used) cache maintained by Workers that stores the state of Workflow Executions they have processed. When a Worker picks up a Workflow Task, it caches the Workflow's state in memory. This allows the Worker to continue processing subsequent Tasks for that Workflow without having to fetch the full Event History from the server and replay it from scratch.

Workflow Cache and Sticky Execution relationship

The Workflow Cache caching mechanism is closely tied to Sticky Execution. The Temporal Service directs future Workflow Tasks to the same Worker that cached the Workflow, via a dedicated Sticky Queue. If the cached Workflow is evicted, to make room for another for example, the Worker must replay the Event History to restore its state before continuing.

Temporalio.Extensions.Aws.Lambda NuGet package purpose

The Temporalio.Extensions.Aws.Lambda NuGet package lets you run a Temporal Serverless Worker on AWS Lambda. You deploy Worker code as a Lambda function, and Temporal Cloud invokes it when Tasks arrive. Each invocation starts a Worker, polls for Tasks, then gracefully shuts down before a configurable invocation deadline.

Create Lambda handler with TemporalLambdaWorker.CreateHandler

Use TemporalLambdaWorker.CreateHandler to create a Lambda handler that runs a Temporal Worker. Pass a WorkerDeploymentVersion and a configure callback that registers your Workflows and Activities. Assign the result to a static field so the handler is created once during Lambda cold start and reused across invocations.

Lambda Worker configuration file resolution order

The Temporalio.Extensions.Aws.Lambda package automatically loads Temporal client configuration from a TOML config file and environment variables. The location of the config file is resolved in the following order: 1. TEMPORAL_CONFIG_FILE environment variable, if set. 2. temporal.toml in $LAMBDA_TASK_ROOT (typically /var/task). 3. temporal.toml in the current working directory. The file is optional. If absent, only environment variables are used.

Lambda Worker defaults table

The Temporalio.Extensions.Aws.Lambda package applies conservative defaults suited to short-lived Lambda invocations. Settings and their Lambda defaults: MaxConcurrentActivities: 2, MaxConcurrentWorkflowTasks: 10, MaxConcurrentLocalActivities: 2, MaxConcurrentNexusTasks: 5, MaxConcurrentWorkflowTaskPolls: 2, MaxConcurrentActivityTaskPolls: 1, MaxConcurrentNexusTaskPolls: 1, MaxCachedWorkflows: 30, GracefulShutdownTimeout: 5 seconds, DisableEagerActivityExecution: Always true, ShutdownDeadlineBuffer: 7 seconds.

ShutdownDeadlineBuffer controls Lambda worker shutdown time

ShutdownDeadlineBuffer is specific to the Temporalio.Extensions.Aws.Lambda package. It controls the time reserved after the worker run budget for worker shutdown and hooks. The default is 7 seconds. If your Worker handles long-running Activities, increase GracefulShutdownTimeout, ShutdownDeadlineBuffer, and the Lambda invocation deadline (--timeout) together.

Temporalio.Extensions.Aws.Lambda.OpenTelemetry package integration

The Temporalio.Extensions.Aws.Lambda.OpenTelemetry NuGet package provides OpenTelemetry integration with defaults configured for the AWS Distro for OpenTelemetry (ADOT) Lambda layer. With this enabled, the Worker emits SDK metrics and distributed traces for Workflow and Activity executions. The ADOT Lambda layer collects this telemetry and can forward traces to AWS X-Ray and metrics to Amazon CloudWatch.

LambdaWorkerOpenTelemetry.ApplyDefaults configuration

Call LambdaWorkerOpenTelemetry.ApplyDefaults in the configure callback to enable OpenTelemetry integration. ApplyDefaults configures Temporal tracing with TracingInterceptor, creates an OTLP trace exporter and tracer provider, configures Core SDK OTLP metrics, uses AWS X-Ray-compatible trace IDs, and registers a per-invocation shutdown hook that force-flushes traces. By default, telemetry is sent to localhost:4317, which is the ADOT Lambda layer's default collector endpoint. The endpoint can be overridden with the OTEL_EXPORTER_OTLP_ENDPOINT environment variable.

LambdaWorkerOpenTelemetryOptions customization

You can customize the OpenTelemetry defaults by passing a LambdaWorkerOpenTelemetryOptions object to ApplyDefaults. The object accepts CollectorEndpoint (string), ServiceName (string), and MetricsExportInterval (TimeSpan). Core SDK metrics export every 10 seconds by default. Set MetricsExportInterval shorter than your Lambda timeout to increase the chance that at least one metrics export happens during each invocation.

ADOT Lambda layer requirements for Temporal Worker

To collect telemetry, attach the ADOT Collector layer to your Lambda function. The default Collector configuration does not route OpenTelemetry Protocol (OTLP) data to the traces pipeline. You must provide a custom Collector configuration that wires the OTLP receiver to both the traces and metrics pipelines. Bundle an otel-collector-config.yaml in your Lambda deployment package and set the OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml environment variable on the Lambda function.

Enable X-Ray active tracing on Lambda function

Enable X-Ray active tracing on the Lambda function by running: aws lambda update-function-configuration --function-name <your-function-name> --tracing-config Mode=Active

Manual tracing and metrics configuration on Lambda

You can also configure tracing and metrics manually using TracingInterceptor and TemporalRuntime. Set config.ClientOptions.Interceptors = new[] { new TracingInterceptor() } and config.ClientOptions.Runtime = new TemporalRuntime(new TemporalRuntimeOptions { Telemetry = new TelemetryOptions { Metrics = new MetricsOptions(new OpenTelemetryOptions("http://collector:4317")), }, });

Give your agent this brain