architecture/workers

117 notes in this subject, read out of this brain and free to use. This is page 1 of 2.

Worker Workflow Task handling: replay and Commands

When a Worker receives a Workflow Task, it replays the Workflow's Event History and runs the Workflow code until it either completes or reaches a point where it must wait (for example, waiting for an Activity or a Timer). It then sends Commands back to the Server that describe what should happen next, such as 'schedule this Activity' or 'start this Timer.'

Worker polling Task Queues

A Worker is the application component responsible for running Workflow and Activity code, built with a Temporal SDK. Workers can be long-running processes deployed on bare metal, virtual machines, or in containers, or can run as Serverless Workers on serverless cloud infrastructure such as AWS Lambda. The Worker connects to the Server and polls Task Queues managed by the Matching Service to ask 'do you have work for me?'

Worker Activity Task handling

When a Worker receives an Activity Task, it calls the Activity function or method and then sends the outcome (success or failure and any additional result data) back to the Server, which records this as Events in the Event History.

Global Namespace failover and task dispatch

The Global Namespace feature enables Workflow Executions to progress through another Cluster in the event of a failover. Only the active Cluster dispatches Tasks. For a failover to be successful, Worker Processes must be polling for Tasks for the Global Namespace on all Clusters.

Global Namespace worker behavior on standby clusters

Worker Processes on the standby Clusters are idle until a failover occurs and their Cluster becomes active.

Nexus Endpoint Worker registration

Workers handle Nexus requests by registering one or more Services and polling the Endpoint's target Task Queue. Multiple Endpoints can target different Task Queues in the same Namespace.

Nexus SDK metrics emitted by Nexus Worker

Nexus Workers emit the following SDK metrics: nexus_poll_no_task, nexus_task_schedule_to_start_latency, nexus_task_execution_failed, nexus_task_execution_latency, and nexus_task_endtoend_latency.

Services are registered in Workers

Services are registered in a Worker that polls the Endpoint's target Task Queue.

Multiple Services can run in the same Worker

Multiple Services can run in the same Worker.

Nexus uses queue-based Worker architecture

Nexus uses the same queue-based Worker architecture as the rest of Temporal. Handler Workers poll the Endpoint's target Task Queue for Nexus Tasks. If a Nexus Service is down, caller Workflows continue to schedule Operations—they process when the service is back up. Load balancing is automatic and no bespoke service deployments are needed.

Worker Service responsibilities

The Worker Service runs background processing for the replication queue, system Workflows, and (in versions older than 1.5.0) the Kafka visibility processor. It uses port 6939 for membership-related communication.

Worker stop timeout vs shutdown deadline buffer tuning effects

The Worker stop timeout controls how long the Worker waits for in-flight Tasks to finish after it stops polling. The shutdown deadline buffer controls how much time before the invocation deadline the Worker stops polling for Tasks. Raising only the shutdown deadline buffer makes the Worker stop polling earlier but does not give in-flight Tasks any more time to complete. Raising only the Worker stop timeout does not make the Worker stop polling earlier, which means the compute provider might terminate the Worker before the full stop timeout completes, and in-flight Activities may not get the full stop timeout to finish.

AWS Lambda autoscaling is event-driven and reactive

The Lambda autoscaling algorithm is event-driven and reactive. Sync match failure is the primary control signal, and backlog aids sizing. When the WCI needs more capacity, it calls the Lambda InvokeFunction API to start new Workers. Each call is a discrete action (invoke N more functions), not a target state. The WCI does not manage a fleet of instances.

AWS Lambda scale-out behavior on sync match failure

On sync match failure, the WCI invokes new Lambda functions. Because Lambda cold start is sub-second to low single-digit seconds, reactive-only control does not create meaningful backlog overshoot. The WCI can scale from zero with low latency.

AWS Lambda scale-in is automatic without active drain

Scale-in is automatic. Each Lambda invocation runs until the Worker has finished processing available Tasks or approaches the 15-minute execution time limit, then shuts down. There is no drain logic or stabilization window. The WCI does not need to actively remove capacity.

AWS Lambda invocation model is stateless

Each Lambda invocation is independent. The Worker starts, creates a fresh client connection, processes multiple Tasks until near the execution time limit, and then shuts down gracefully. There is no shared state across invocations.

Map Temporal Worker Deployment Versions to AWS Lambda function versions

For production workloads, map each Temporal Worker Deployment Version to exactly one AWS Lambda function version, and configure the compute provider with the qualified versioned ARN for that Lambda version (for example, arn:aws:lambda:us-east-1:123:function:my-worker:5). A Worker Deployment Version is an immutable build identifier, and the Lambda function code it invokes should also be immutable for production.

Unqualified Lambda ARNs point to $LATEST and cause non-determinism risk

An unqualified ARN (no version suffix) points at $LATEST, which changes on every redeploy. Without a versioned ARN, deploying replay-unsafe code causes non-determinism errors for in-flight Workflows, even for Workflows annotated as Pinned.

Pinned vs Auto-Upgrade behavior with versioned Lambda ARNs

When using versioned Lambda ARNs: With Pinned versioning behavior, existing Workflows stay on their original Lambda function version until they complete. With Auto-Upgrade versioning behavior, existing Workflows move to the new Worker Deployment Version and its new Lambda function version at the next Workflow Task after the Current Version is updated.

Pinned vs Auto-Upgrade behavior with unqualified Lambda ARNs

When using unqualified Lambda ARNs ($LATEST): With Pinned versioning behavior, existing Workflows stay on their original Worker Deployment Version, but the underlying Lambda code has already changed since $LATEST updated at redeploy, requiring replay-compatible code. With Auto-Upgrade versioning behavior, the Lambda redeploy already changed the code for all versions, and setting the Current Version only changes routing, not which code runs.

Serverless Worker invocation lifecycle has three phases

A single Serverless Worker invocation has three phases: init, work, and shutdown. During init, the Worker initializes and establishes a client connection to Temporal. During work, the Worker polls the Task Queue and processes Tasks. During shutdown, the Worker stops polling, waits for in-flight Tasks to finish, and runs any shutdown hooks (such as OpenTelemetry telemetry flushes). Shutdown begins before the invocation deadline so the Worker can exit cleanly before the compute provider forcibly terminates the execution environment.

Configuration for long-running Activities in Serverless Workers

For Workers handling long-running Activities, set three values together: (1) Worker stop timeout must be greater than the longest Activity runtime to give in-flight Activities enough time to finish after polling stops; (2) Shutdown deadline buffer must be greater than Worker stop timeout plus shutdown hook time to ensure the drain and shutdown hooks complete before the compute provider terminates the environment; (3) Invocation deadline (set on the compute provider) must be greater than longest Activity runtime plus shutdown deadline buffer to give each invocation enough total runtime. For example, if the longest Activity runtime is 5 minutes and shutdown hooks take 3 seconds: set Worker stop timeout to more than 5 minutes, shutdown deadline buffer to more than 303 seconds (5 minutes + 3 seconds), and invocation deadline to at least 10 minutes and 3 seconds (5 minutes + 303 seconds).

Version strings do not need to follow semver

Versions in Worker Versioning can be any arbitrary string and do not need to follow semantic versioning or any other semantic versioning scheme. Version strings can be any value up to 255 characters.

Worker Versioning legacy documentation status

The 2023 draft of Worker Versioning is deprecated and was not made available in Temporal Cloud. The 2024 draft was available in Cloud on an opt-in basis. Users should refer to the current Worker Versioning documentation instead.

Worker Versioning purpose and Build ID

Worker Versioning simplifies deployment of changes to Workflow Definitions by letting you define sets of versions that are compatible with each other and assigning a Build ID to the code that defines a Worker. The Temporal Server uses the Build ID to determine which versions of a Workflow Definition a Worker can process.

Worker Versioning manages nondeterministic changes

Worker Versioning helps manage nondeterministic changes by ensuring that Workers with different Workflow and Activity Definitions operating on the same Task Queue don't attempt to process Workflow Tasks and Activity Tasks that they can't successfully process. It does this by assigning Build IDs to Workers and specifying which Build IDs are compatible with each other through version sets stored by the Temporal Server.

Decommissioning old Workers with versioning

You can decommission old Workers after you archive all open Workflows using their version. If you have no need to query closed Workflows, you can decommission them when no open Workflows remain at that version. For short-lived Workflows, a good strategy is to assign a new Build ID to every new Worker build and add it as the new overall default, knowing you won't need to keep older Workers running for long after the new version is deployed.

Version sets size limits

Version sets have a maximum size limit of 100 Build IDs across all sets by default, configurable via the `limit.versionBuildIdLimitPerQueue` dynamic config property. Operations to add new Build IDs to the sets fail if they exceed this limit. There is also a limit on the number of Version Sets, which defaults to 10, configurable via the `limit.versionCompatibleSetLimitPerQueue` dynamic config property. There is also a limit on the size of each Build ID or version string, which defaults to 255 characters, configurable via the `limit.workerBuildIdSize` dynamic config property.

Unversioned and versioned Task Queue matching

By default, both Task Queues and Workers are in an unversioned state. Unversioned Workers can poll unversioned Task Queues and receive tasks. To use Worker Versioning, both the Task Queue and the Worker must be associated with Build IDs. If you run a Worker using versioning against a Task Queue that has not been set up to use versioning or is missing that Worker's Build ID, it won't get any tasks. An unversioned Worker polling a Task Queue with versioning won't work either.

Version sets example: initial setup

When you add a version '1.0' to a Task Queue as the new default, the version sets are: set 1 (default) contains 1.0 (default). All new Workflows started on the Task Queue have their first tasks assigned to version 1.0. Workers with their Build ID set to 1.0 receive these Tasks. If Workflows without an assigned version are still running, Workers without a version take those tasks.

Version sets example: adding incompatible version

When you add version '2.0' to the sets as the new default, the version sets become: set 1 with 1.0 (default), and set 2 (default) with 2.0 (default). All new Workflows have their first tasks assigned to version 2.0. Existing 1.0 Workflows keep generating tasks targeting 1.0. Each deployment of Workers receives their respective Tasks.

Version sets example: adding compatible version for bug fix

When you add version '2.1' to the sets marked as compatible with '2.0', the version sets become: set 1 with 1.0 (default), and set 2 (default) with 2.0 and 2.1 (default). All new Workflow Tasks generated for Workflows whose last Workflow Task completion was on version 2.0 are now assigned to version 2.1. Because 2.1 is compatible with 2.0, Temporal Server assumes Workers with this version can process the existing Event Histories successfully.

Version sets example: reverting to older compatible set

You can perform an update targeting an existing compatible version and setting its set as the current default. For example, if you have sets: set 1 with 1.0 (default), set 3 (default) with 3.0 (default), and set 2 with 2.0 and 2.1 (default), you can make set 2 the overall-default set, resulting in: set 1 with 1.0 (default), set 3 with 3.0 (default), and set 2 (default) with 2.0 and 2.1 (default). Now new Workflows start on 2.1.

Permitted operations on version sets

Requests to change version sets can: (1) Add a version to the sets as the new default version in a new overall-default compatible set. (2) Add a version to an existing set that's compatible with an existing version, optionally making it the default for that set and optionally making that set the overall-default set. (3) Promote a version within an existing set to become the default for that set. (4) Promote a set to become the overall-default set.

Cannot explicitly delete versions

You can't explicitly delete versions in Worker Versioning. This helps you avoid the situation in which Workflows accidentally become stuck with no means of making progress because the version they're associated with no longer exists. However, a background process will automatically delete IDs and sets that are no longer needed after no open Workflows are using that version.

Unversioned Workers definition

Unversioned Workers refer to Workers that have not opted into the Worker Versioning feature in their configuration. They receive tasks only from Task Queues that do not have any version sets defined on them, or that have open Workflows that began executing before versions were added to the queue.

Cloud Run compute configuration does not name a revision

The compute configuration names a project, region, and Worker Pool but does not name a Cloud Run revision. Temporal runs whichever revision the pool serves at the time. This ties a pool to one build, and a new build needs a new pool.

Deploying new image to live Worker Pool creates replay-unsafe code risk

Deploying a new image into a pool that a live Worker Deployment Version points at creates a new revision, and Cloud Run promotes it to every instance by default. While the version does not change, the code behind it does. Deploying replay-unsafe code this way causes non-determinism errors for in-flight Workflows, including Pinned ones.

Serverless Workers on Cloud Run run in Worker Pools

On Cloud Run, a Serverless Worker runs in a Worker Pool, which is a set of long-lived instances that poll the Task Queue continuously. Each instance runs standard Worker code and processes Tasks for its whole lifetime. The Worker Controller Instance (WCI) controls how many instances run, and each instance manages its own polling and Task processing.

Cloud Run instances are stopped by scale-in, not by code logic

An instance's lifetime is bounded by scale-in. The WCI decides when to remove an instance from Task Queue activity, not based on what individual instances are doing. It waits a set amount of time after the most recent sync match failure and applies a cooldown between successive reductions before lowering the target instance count. It does not track how long an instance has been running or whether it is mid-Activity, so the instance Cloud Run stops may be one still executing work.

WCI autoscaling combines immediate and periodic mechanisms

The Worker Controller Instance uses two mechanisms for autoscaling: immediately bringing up new instances when a Task arrives and no Worker is free (sync match failure), and periodically resizing the Worker Pool based on incoming work. The immediate reaction absorbs bursts without waiting on the evaluation cycle.

WCI measures task arrival rate and worker processing rate

For periodic resizing, the WCI measures two rates over time: how fast Tasks arrive and how fast a single Worker processes them. From these, it calculates how many Workers are needed to handle the incoming work and adjusts the pool's target instance count accordingly as the arrival rate changes.

Cloud Run Worker Pool target utilization is 80% by default

The WCI sizes the pool to a target utilization of 80% by default, so there is room to pick up newly arriving Tasks right away instead of queuing them. When Tasks are already waiting in the backlog, the WCI adds more instances on top to work through them faster.

Cloud Run scale-out provisions capacity ahead of demand

When arrivals rise or backlog builds, the WCI raises the pool's target instance count and Cloud Run starts more instances. Because the WCI keeps a utilization buffer, the pool adds capacity ahead of demand rather than waiting until Tasks pile up. This prevents backlog growth during the startup window when new instances take time to start and connect.

Cloud Run scale-in is more conservative than scale-out

When arrivals and backlog fall, the WCI lowers the target instance count and Cloud Run stops surplus instances. Scale-in is more conservative than scale-out: the WCI holds capacity while sync match failures are still occurring, applies a cooldown before reducing the pool, and avoids removing instances that are about to be needed again. The pool can scale to zero when there is no work.

Cloud Run Serverless Workers require Worker Versioning

Serverless Workers on Cloud Run require Worker Versioning. You must create one Worker Pool for each Worker Deployment Version and carry the Build ID in the pool name. An older version's pool should be kept in place while Pinned Workflows are still running on it; the pool can sit at zero instances and the WCI scales it back up when a Task arrives for that version.

Sticky Execution timeout and fallback

If a Worker fails to start a Workflow Task in the Sticky Queue within five seconds of scheduling (the default timeout), the Temporal Service disables stickiness for that Workflow Execution and reschedules the Workflow Task in the original Task Queue, allowing any Worker to pick it up.

Sticky Execution performance benefit

The main benefit of Sticky Execution is improved performance by caching the Workflow state in memory and directing tasks to the same Worker, reducing the need to reconstruct the Workflow from its Event History for every Task. This is particularly useful for latency-sensitive Workflows.

Sticky Execution applies only to Workflow Tasks

Sticky Execution is the default behavior of the Temporal Platform and only applies to Workflow Tasks. Since Event History is associated with a Workflow, the concept of Sticky Execution is not relevant to Activity Tasks.

StickyScheduleToStartTimeout configuration

The timeout for how long a Worker has to start a Workflow Task in the Sticky Queue after scheduling can be configured using the StickyScheduleToStartTimeout setting on a Worker. The default value is five seconds.

Sticky Execution definition

Sticky Execution is a performance optimization in Temporal that directs Workflow Tasks to the same Worker that previously processed tasks for a specific Workflow Execution. This allows Workers to cache the Workflow state locally, reducing the need to reconstruct the Workflow from its Event History for every Task.

How Sticky Execution works

When a Workflow Task is picked up by a Worker from the original Task Queue, that Worker begins polling an additional Worker-specific Sticky Queue with an automatically-generated name that is shared exclusively with that Worker. The Worker caches the Workflow state in memory. As the Workflow Execution progresses, the Temporal Service schedules additional Workflow Tasks into this Sticky Queue instead of the original queue.

Sticky Execution invalidation on task failure

If a Workflow Task fails, the Worker removes that Workflow Execution from its cache because the state is now unknown, which invalidates the Sticky Execution. The Workflow Task is then put back into the original Task Queue.

Serverless and long-lived Workers on same Task Queue

Serverless Workers can share a Task Queue with long-lived Workers. Serverless Workers are only invoked on sync match failure, so they pick up Tasks that no long-lived Worker was available to handle. In practice, Serverless Workers act as spillover capacity for the long-lived fleet. Do not enable dynamic scaling on long-lived Workers if both groups are on the same Task Queue, as they cannot coordinate scaling and may both handle the same Tasks.

WCI Workflow identification and inspection

WCI Workflow IDs follow the pattern `temporal-sys-worker-controller-instance:<deployment-name>:<build-id>`. You can list WCI Workflows with: `temporal workflow list --namespace <NAMESPACE> --query 'TemporalNamespaceDivision = "TemporalWorkerControllerInstance"'`. You can inspect a WCI Workflow's history with: `temporal workflow show --namespace <NAMESPACE> --workflow-id 'temporal-sys-worker-controller-instance:<DEPLOYMENT_NAME>:<BUILD_ID>'`.

Serverless Worker definition and lifecycle management

A Serverless Worker is a Temporal Worker that runs on serverless compute instead of a long-lived process. Temporal manages the Worker's lifecycle rather than the user running a Worker process. Serverless Workers use the same Temporal SDKs as traditional long-lived Workers and register Workflows and Activities the same way. The lifecycle differs based on the compute provider: AWS Lambda runs short-lived invocations, while GCP Cloud Run runs a pool of long-lived instances.

Serverless Worker constraints table

Activity duration: Depends on compute provider. AWS Lambda Activities must finish within the invocation limit (15 minutes maximum) minus shutdown deadline buffer. GCP Cloud Run instances are long-lived, so no per-invocation limit applies. Workflow duration: No limit; Workflows of any duration work. Worker code: Same Temporal SDK Worker code using the serverless Worker package for your SDK. Versioning: Worker Versioning is required; each Workflow must have AutoUpgrade or Pinned behavior. High Availability: On failover of a Namespace with Multi-region or Multi-cloud Replication, the WCI keeps invoking Workers in the original region unless you manually repoint the compute provider.

Serverless Worker versioning requirements

Serverless Workers require Worker Versioning, and the compute provider must invoke a stable, immutable build for each Worker Deployment Version. How Worker Deployment Versions map to compute provider primitives differs by provider (AWS Lambda and GCP Cloud Run have different versioning approaches).

Serverless Worker prerequisites and compute providers

Serverless Workers require Worker Versioning and must be associated with a Worker Deployment Version that has a compute provider configured. Two compute providers are supported: AWS Lambda, where Temporal assumes an IAM role to invoke a Lambda function, and GCP Cloud Run, where Temporal scales a Cloud Run Worker Pool through the Cloud Run admin API.

Worker Controller Instance system workflow

The Worker Controller Instance (WCI) is a system Workflow that scales Serverless Workers based on Task Queue conditions. One WCI Workflow runs per Worker Deployment Version that has a compute provider configured, running in the same Namespace as the Worker Deployment. The WCI adjusts Worker count based on sync match failures and Task Queue backlog, adding Workers by triggering the compute provider and removing capacity as work drains.

Give your agent this brain