workers/basics

378 notes in this subject, read out of this brain and free to use. This is page 6 of 7.

Temporal Lambda Worker with @temporalio/lambda-worker

The @temporalio/lambda-worker package enables running a Temporal Serverless Worker on AWS Lambda. It is deployed as a Lambda function that Temporal Cloud invokes when tasks arrive. Each invocation starts a Worker, polls for tasks, and gracefully shuts down before a configurable invocation deadline. Workflows and Activities are registered the same way as with standard Workers.

Temporal config file resolution for Lambda Worker

The @temporalio/lambda-worker package automatically loads Temporal client configuration from a TOML config file and environment variables. The location is resolved in order: (1) TEMPORAL_CONFIG_FILE environment variable if set, (2) temporal.toml in $LAMBDA_TASK_ROOT (typically /var/task), (3) temporal.toml in the current working directory. The file is optional; if absent, only environment variables are used.

runWorker function for Lambda handler

Use the runWorker function to create a Lambda handler that runs a Temporal Worker. Pass a deployment version and a configure callback that sets up Workflows and Activities. Example: export const handler = runWorker({ deploymentName: 'sdk-demo', buildId: 'v1' }, (config) => { config.workerOptions.taskQueue = TASK_QUEUE; config.workerOptions.workflowBundle = { codePath: require.resolve('./workflow-bundle.js') }; config.workerOptions.activities = activities; });

Java SDK Poller Autoscaling configuration

In the Java SDK, enable Poller Autoscaling for Workflow Tasks, Activity Tasks, and Nexus Tasks by setting the corresponding options in WorkerOptions: WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs(); WorkflowClient client = WorkflowClient.newInstance(service); WorkerFactory factory = WorkerFactory.newInstance(client); WorkerOptions workerOptions = WorkerOptions.newBuilder() .setWorkflowTaskPollersBehavior(new PollerBehaviorAutoscaling()) .setActivityTaskPollersBehavior(new PollerBehaviorAutoscaling()) .setNexusTaskPollersBehavior(new PollerBehaviorAutoscaling()) .build(); Worker worker = factory.newWorker("my-task-queue", workerOptions); See Java SDK docs at https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/tuning/PollerBehaviorAutoscaling.html

Python SDK Poller Autoscaling configuration

In the Python SDK, enable Poller Autoscaling for Workflow Tasks, Activity Tasks, and Nexus Tasks by setting the corresponding options in Worker: worker = Worker( client, task_queue="my-task-queue", workflows=[MyWorkflow], activities=[my_activity], workflow_task_poller_behavior=PollerBehaviorAutoscaling(), activity_task_poller_behavior=PollerBehaviorAutoscaling(), nexus_task_poller_behavior=PollerBehaviorAutoscaling(), ) See Python SDK docs at https://python.temporal.io/temporalio.worker.PollerBehaviorAutoscaling.html

TypeScript SDK Poller Autoscaling configuration

In the TypeScript SDK, enable Poller Autoscaling for Workflow Tasks, Activity Tasks, and Nexus Tasks by setting the corresponding options in Worker: const worker = await Worker.create({ connection, taskQueue: 'my-task-queue', workflowsPath: require.resolve('./workflows'), activities, workflowTaskPollerBehavior: PollerBehavior.autoscaling(), activityTaskPollerBehavior: PollerBehavior.autoscaling(), nexusTaskPollerBehavior: PollerBehavior.autoscaling(), }); See TypeScript SDK docs at https://typescript.temporal.io/api/interfaces/proto.temporal.api.sdk.v1.WorkerConfig.IAutoscalingPollerBehavior

.NET SDK Poller Autoscaling configuration

In the .NET SDK, enable Poller Autoscaling for Workflow Tasks, Activity Tasks, and Nexus Tasks by setting the corresponding options in TemporalWorkerOptions: using var worker = new TemporalWorker( client, new TemporalWorkerOptions("my-task-queue") { WorkflowTaskPollerBehavior = new PollerBehavior.Autoscaling(), ActivityTaskPollerBehavior = new PollerBehavior.Autoscaling(), NexusTaskPollerBehavior = new PollerBehavior.Autoscaling(), } .AddWorkflow<MyWorkflow>() .AddActivity(MyActivities.MyActivity) ); See .NET SDK docs at https://dotnet.temporal.io/api/Temporalio.Worker.Tuning.PollerBehavior.Autoscaling.html

Ruby SDK Poller Autoscaling configuration

In the Ruby SDK, enable Poller Autoscaling for Workflow Tasks, Activity Tasks, and Nexus Tasks by setting the corresponding options in Worker: worker = Temporalio::Worker.new( client, 'my-task-queue', workflows: [MyWorkflow], activities: [MyActivity], workflow_task_poller_behavior: Temporalio::Worker::PollerBehavior::Autoscaling.new, activity_task_poller_behavior: Temporalio::Worker::PollerBehavior::Autoscaling.new, nexus_task_poller_behavior: Temporalio::Worker::PollerBehavior::Autoscaling.new, ) See Ruby SDK docs at https://ruby.temporal.io/Temporalio/Worker/PollerBehavior/Autoscaling.html

Workflow Cache shared between Workers on a single host

A Workflow Cache is created and shared between all Workers on a single host. It is designed to limit the resources used by the cache for each host/process.

Java SDK Cache options on WorkerFactoryOptions

For the Java SDK, Workflow cache options are defined on WorkerFactoryOptions: - WorkerFactoryOptions#workflowCacheSize: defines the maximum number of cached Workflow Executions. Each cached Workflow contains at least one Workflow thread and its resources (memory, etc.). - maxWorkflowThreadCount: defines the maximum number of Workflow threads that may exist concurrently at any time. These cache options limit the resource consumption of the in-memory Workflow cache. Workflow cache options are shared between all Workers because the Workflow cache is tightly integrated with the resource consumption of the entire host, including memory and total thread count.

Go SDK Workflow Cache configuration

For the Go SDK, use SetStickyWorkflowCacheSize to configure the Workflow Cache size.

Python SDK Workflow Cache configuration

For the Python SDK, use the max_cached_workflows Worker option to configure the Workflow Cache size.

Drawbacks of using excessively large values

Specifying excessively large values for Worker configuration options without monitoring with SDK and system metrics leads to constant resource contention and stealing. This decreases total throughput and increases latency jitter of the system. As with any multithreading system, using large values everywhere without proper monitoring is a pitfall.

Java SDK Worker configuration invariant 1: workflowCacheSize and maxWorkflowThreadCount

In the Java SDK, workflowCacheSize should be less than or equal to maxWorkflowThreadCount, since each Workflow has at least one Workflow thread.

Java SDK Worker configuration invariant 3: poller counts vs executor counts

In the Java SDK, maxConcurrentWorkflowTaskPollers should be significantly less than maxConcurrentWorkflowTaskExecutionSize, and maxConcurrentActivityTaskPollers should be significantly less than maxConcurrentActivityExecutionSize. The number of pollers should always be lower than the number of executors.

Java SDK Worker configuration invariant 2: maxConcurrentWorkflowTaskExecutionSize and maxWorkflowThreadCount

In the Java SDK, maxConcurrentWorkflowTaskExecutionSize should be less than or equal to maxWorkflowThreadCount. It is recommended that maxWorkflowThreadCount be at least 2x of maxConcurrentWorkflowTaskExecutionSize. Having more Worker slots than the Workflow cache size will lead to resource allocation issues between executors and cause unpredictable delays.

Worker Options configuration at instantiation

Each Worker can be configured by providing custom Worker options (WorkerOptions) at instantiation. Options are specific to individual Workers and do not affect other members of your fleet.

Executor slot options: maxConcurrentWorkflowTaskExecutionSize and maxConcurrentActivityExecutionSize

The maxConcurrentWorkflowTaskExecutionSize and maxConcurrentActivityExecutionSize options define the number of total available Workflow Task and Activity Task slots for a Worker. Worker tuners supersede the existing maxConcurrentXXXTask style Worker options. Using both styles will cause an error at Worker initialization time.

Poller Autoscaling feature for automatic poller selection

Temporal SDKs support Poller Autoscaling, which automatically selects an appropriate number of pollers based on need. This feature results in more efficient poller usage, better throughput, and schedule-to-start latency improvements. You can enable this feature by setting the *_task_poller_behavior options to PollerBehaviorAutoscaling. Poller Autoscaling will be the default configuration in future versions of Temporal SDKs.

Poller Autoscaling requires Temporal Server v1.28.0 or later

PollerBehaviorAutoscaling is only enabled in Temporal Server v1.28.0 and later.

Manual poller configuration options

Manual poller configuration options are available but not recommended for production use cases. The following options are available: maxConcurrentWorkflowTaskPollers (in Java SDK: workflowPollThreadCount) and maxConcurrentActivityTaskPollers (in Java SDK: activityPollThreadCount). These options define the maximum count of pollers performing poll requests on Workflow and Activity Task Queues, respectively.

Go SDK Poller Autoscaling configuration

In the Go SDK, enable Poller Autoscaling for Workflow Tasks, Activity Tasks, and Nexus Tasks by setting the corresponding options in worker.Options: w := worker.New(c, "my-task-queue", worker.Options{ WorkflowTaskPollerBehavior: worker.NewPollerBehaviorAutoscaling(worker.PollerBehaviorAutoscalingOptions{}), ActivityTaskPollerBehavior: worker.NewPollerBehaviorAutoscaling(worker.PollerBehaviorAutoscalingOptions{}), NexusTaskPollerBehavior: worker.NewPollerBehaviorAutoscaling(worker.PollerBehaviorAutoscalingOptions{}), }) See Go SDK docs at https://pkg.go.dev/go.temporal.io/sdk/worker#PollerBehaviorAutoscalingOptions

Custom Slot Suppliers

Custom Slot Suppliers hand out slots based on custom logic that you define. Use this approach when you need complete control over when Workers accept and execute Tasks. Implementation details are available in the documentation on implementing Custom Slot Suppliers.

Worker Task Slot definition

A Worker Task Slot represents the capacity of a Temporal Worker to execute a single concurrent Task. Slots are used for both Workflow and Activity Tasks. When a Worker starts processing a Task, it occupies one slot. The number of available slots directly affects how many tasks a Worker can handle simultaneously.

Slot Supplier definition and purpose

A Slot Supplier defines a strategy to provide slots for a Worker, increasing or decreasing the Worker's slot count. The supplier determines when it's acceptable to begin a new Task. Each supplier manages one slot type. There are slot types for Activity, Workflow, Nexus, or Local Activity Tasks. An available slot determines whether or not a Worker is willing to poll for and execute a new Task of that type.

Fixed Size Slot Suppliers

Fixed Size Slot Suppliers hand out slots up to a preset limit. This approach is useful if you have a concrete idea of how many resources your tasks are going to consume and can easily determine an upper bound on how many should run at once. To achieve optimal performance with minimal overhead, calculate the maximum number of slots you can support without oversubscribing or hitting out-of-memory conditions by reviewing hardware and environment characteristics.

Resource-Based Slot Suppliers

Resource-Based Slot Suppliers hand out slots based on real-time CPU and memory usage. You set target utilization for both CPU and memory and the Slot Supplier tries to reach those values without exceeding them under load. A resource-based supplier accounts for memory limits imposed in containerized environments and dynamically adjusts the number of available slots for different task types with respect to current system resources. In containerized environments, all SDKs use cgroups for both CPU and memory, with CPU accounted for at the container level.

Resource-based supplier limitations

You cannot guarantee that the targets for resource-based suppliers won't ever be exceeded. Resources consumed during a task cannot be known ahead of time.

Worker tuners supersede maxConcurrentXXXTask options

Worker tuners supersede the existing maxConcurrentXXXTask style Worker options. Using both styles will cause an error at Worker initialization time.

Worker tuning definition

Worker tuning is the process of defining customized slot suppliers for the different task slots of a Worker to fine-tune its performance. You use special types called Worker tuners that assign slot suppliers to various Task Types, including Worker, Activity, Nexus, and Local Activity Tasks.

Task Pollers role

A Worker's Task Pollers play a crucial role in the Temporal architecture by efficiently ingesting work to Workers to support scalable, resilient Workflow Execution. Pollers create long-polling connections to the Temporal Service and actively poll a Task Queue for Tasks to process. When a Task Poller receives a Task, it delivers the Task to the appropriate Executor Slot for processing.

Poller Autoscaling

Temporal SDKs implement support for Poller Autoscaling, which dynamically adjusts the number of pollers in use to maximize throughput for a given number of workers and the size of the task backlog. Temporal recommends using Poller Autoscaling for the majority of use cases, as manually setting the number of pollers too high or too low for your workload will result in decreased performance.

Worker heartbeat and visualization in UI

The Temporal SDK includes a heartbeat for Worker processes that is sent to the Server, carrying information such as available task slots, CPU usage, and Worker configuration. The Server exposes this data through APIs that surface Worker details in the Temporal UI. You can see a list of all running Workers by selecting Workers in the left navigation menu. You can also view the list of Workers assigned to the Workflow Task Queue and inspect Worker details by selecting the Workers tab in the Workflow details page.

Worker heartbeat requirements

Worker Heartbeating requires Temporal Server 1.30 or higher with API version 1.62 or higher, and is also available in Temporal Cloud. Minimum SDK versions are specified in the Worker Health documentation.

Three important metric groups for Worker performance tuning

Performance tuning uses three important SDK metric groups: slot availability metrics, latency metrics, and cache metrics.

Python SDK composite tuner example

Creating a composite tuner in Python that combines different slot supplier types with poller autoscaling: resource_based_options = ResourceBasedTunerConfig(0.8, 0.9) tuner = WorkerTuner.create_composite( workflow_supplier=FixedSizeSlotSupplier(10), activity_supplier=ResourceBasedSlotSupplier( ResourceBasedSlotConfig(), resource_based_options, ), local_activity_supplier=ResourceBasedSlotSupplier( ResourceBasedSlotConfig(), resource_based_options, ), ) worker = Worker( client, task_queue="foo", tuner=tuner, workflow_task_poller_behavior=PollerBehaviorAutoscaling(), activity_task_poller_behavior=PollerBehaviorAutoscaling() )

.NET SDK resource-based tuner example

Creating a resource-based tuner in .NET C#: var worker = new TemporalWorker( Client, new TemporalWorkerOptions("my-task-queue") { Tuner = WorkerTuner.CreateResourceBased(0.8, 0.9), });

.NET SDK composite tuner example

Creating a composite tuner in .NET C# that combines different slot supplier types: var resourceTunerOptions = new ResourceBasedTunerOptions(0.8, 0.9); var worker = new TemporalWorker( Client, new TemporalWorkerOptions("my-task-queue") { Tuner = new WorkerTuner( new FixedSizeSlotSupplier(10), new ResourceBasedSlotSupplier( new ResourceBasedSlotSupplierOptions(), resourceTunerOptions), new ResourceBasedSlotSupplier( new ResourceBasedSlotSupplierOptions(), resourceTunerOptions)), });

Worker Tuner manages slot suppliers

A Worker Tuner instance exists per-Worker and provides slot suppliers for different slot types: Activity, Workflow, Nexus, or Local Activity Tasks. A tuner assigns different suppliers to each slot type, for example using a fixed assignment slot supplier for Workflows and a resource-based supplier for Activities.

Three types of slot suppliers: fixed-size, resource-based, custom

Temporal offers three types of slot suppliers. For most workloads, Temporal recommends fixed-size slot suppliers. A fixed-size tuner with appropriately chosen values delivers better performance and more predictable behavior than a resource-based tuner.

Resource-based slot suppliers for fluctuating workloads

Resource-based slot suppliers work well when each Task consumes few resources but may run for a long time, such as HTTP calls or other blocking I/O that spend most of their time waiting on external events. They also provide protection from out-of-memory and over-subscription when Tasks have unpredictable per-task resource consumption.

Custom slot suppliers for fine-tuned task allocation

Custom slot suppliers let you tailor the logic of how slots are allocated based on your system requirements, providing flexibility to optimize for specific use cases that fixed assignment and resource-based suppliers do not fully address. They offer the highest level of control over slot allocation.

Custom Slot Supplier interface methods

Custom Slot Suppliers must implement four functions: reserveSlot (called before polling for new tasks, can block and must return a Slot Permit), tryReserveSlot (called for slot reservations like eager activity processing, must not block), markSlotUsed (called when a slot is about to be used for a task with task information), and releaseSlot (called when a slot is no longer needed).

Slot Permit represents right to use a slot

Slot Suppliers issue SlotPermits, which represent the right to use a slot of a specific type: Workflow, Activity, Local Activity, or Nexus. By issuing or withholding permits, you control whether a Worker can perform certain tasks.

rampThrottle controls slot assignment rate in resource-based suppliers

The rampThrottle setting defines the minimum time the Worker will wait between handing out new slots after passing the minimum slots number. A higher rampThrottle trades off performance for safety by allowing the Worker to assess resource usage changes before accepting more tasks.

Go SDK resource-based tuner example

Creating a resource-based tuner in Go: func resourceBasedTuner() (worker.Options, error) { tuner, err := worker.NewResourceBasedTuner(worker.ResourceBasedTunerOptions{ TargetMem: 0.8, TargetCpu: 0.9, InfoSupplier: sysinfo.SysInfoProvider(), }) if err != nil { return worker.Options{}, err } return worker.Options{ Tuner: tuner, }, nil }

Go SDK composite tuner example

Creating a composite tuner in Go that mixes fixed-size and resource-based slot suppliers: func compositeTuner() (worker.Options, error) { options := worker.DefaultResourceControllerOptions() options.MemTargetPercent = 0.8 options.CpuTargetPercent = 0.9 options.InfoSupplier = sysinfo.SysInfoProvider() controller := worker.NewResourceController(options) wfSS, err := worker.NewFixedSizeSlotSupplier(10) if err != nil { return worker.Options{}, err } actSS, err := worker.NewResourceBasedSlotSupplier(controller, worker.DefaultActivityResourceBasedSlotSupplierOptions()) if err != nil { return worker.Options{}, err } laSS, err := worker.NewResourceBasedSlotSupplier(controller, worker.DefaultActivityResourceBasedSlotSupplierOptions()) if err != nil { return worker.Options{}, err } nexusSS, err := worker.NewFixedSizeSlotSupplier(10) if err != nil { return worker.Options{}, err } compositeTuner, err := worker.NewCompositeTuner(worker.CompositeTunerOptions{ WorkflowSlotSupplier: wfSS, ActivitySlotSupplier: actSS, LocalActivitySlotSupplier: laSS, NexusSlotSupplier: nexusSS, }) if err != nil { return worker.Options{}, err } return worker.Options{ Tuner: compositeTuner, }, nil }

Java SDK resource-based tuner example

Creating a resource-based tuner in Java: WorkerOptions.newBuilder() .setWorkerTuner( ResourceBasedTuner.newBuilder() .setControllerOptions( ResourceBasedControllerOptions.newBuilder(0.8, 0.9).build()) .build()) .build()

Java SDK composite tuner example

Creating a composite tuner in Java that combines different slot supplier types: SlotSupplier<WorkflowSlotInfo> workflowTaskSlotSupplier = new FixedSizeSlotSupplier<>(10); SlotSupplier<ActivitySlotInfo> activityTaskSlotSupplier = ResourceBasedSlotSupplier.createForActivity( resourceController, ResourceBasedTuner.DEFAULT_ACTIVITY_SLOT_OPTIONS); SlotSupplier<LocalActivitySlotInfo> localActivitySlotSupplier = ResourceBasedSlotSupplier.createForLocalActivity( resourceController, ResourceBasedTuner.DEFAULT_ACTIVITY_SLOT_OPTIONS); SlotSupplier<NexusSlotInfo> nexusSlotSupplier = new FixedSizeSlotSupplier<>(10); WorkerOptions.newBuilder() .setWorkerTuner( new CompositeTuner( workflowTaskSlotSupplier, activityTaskSlotSupplier, localActivitySlotSupplier, nexusSlotSupplier)) .build()

TypeScript SDK resource-based tuner example

Creating a resource-based tuner in TypeScript: const resourceBasedTunerOptions: ResourceBasedTunerOptions = { targetMemoryUsage: 0.8, targetCpuUsage: 0.9, }; const workerOptions = { tuner: { tunerOptions: resourceBasedTunerOptions, }, };

TypeScript SDK composite tuner example

Creating a composite tuner in TypeScript that mixes different slot supplier types: const resourceBasedTunerOptions: ResourceBasedTunerOptions = { targetMemoryUsage: 0.8, targetCpuUsage: 0.9, }; const workerOptions = { tuner: { activityTaskSlotSupplier: { type: 'resource-based', tunerOptions: resourceBasedTunerOptions, }, workflowTaskSlotSupplier: { type: 'fixed-size', numSlots: 10, }, localActivityTaskSlotSupplier: { type: 'resource-based', tunerOptions: resourceBasedTunerOptions, }, }, };

Resource-based supplier cannot perfectly respect thresholds

Auto-tuned suppliers may diverge from requested thresholds because the resources a given Task will use cannot be known ahead of time. There is a fundamental tradeoff between how quickly a slot supplier is willing to accept Tasks and how well it can respect the defined thresholds.

Python SDK resource-based tuner example

Creating a resource-based tuner in Python with poller autoscaling: tuner = WorkerTuner.create_resource_based( target_memory_usage=0.5, target_cpu_usage=0.5, ) worker = Worker( client, task_queue="foo", tuner=tuner, workflow_task_poller_behavior=PollerBehaviorAutoscaling(), activity_task_poller_behavior=PollerBehaviorAutoscaling() )

Visibility API rate limits for Task Queue metrics

Visibility API rate limits apply to Task Queue performance data requests.

Using backlog metrics to manage Worker fleet

ApproximateBacklogAge shows how long Tasks have been waiting to be dispatched; if this time grows too long, more Workers can boost Workflow efficiency. Calculate the demand per Worker by dividing ApproximateBacklogCount by the number of Workers to determine if task processing rate is acceptable. Use per-Worker demand, the backlog consumption rate (TasksDispatchRate), and dispatch latency (ApproximateBacklogAge) to assess performance. The BacklogIncreaseRate shows changing demand on Workers over time; as it increases, add more Workers until demand and capacity are balanced; as it decreases, reduce your Worker fleet.

Known accuracy limitations of backlog metrics

ApproximateBacklogCount and ApproximateBacklogAge have three main sources of inaccuracy. First, overcount from invalid or expired Tasks: Tasks belonging to cancelled, terminated, completed, or timed out Workflows and Activities stay in the count until processed and discarded at the head of the queue. Second, reset to zero on idle Task Queue unload: if a Task Queue sees no activity for approximately 5 minutes (no Worker polls, new Tasks, or other Task Queue calls), the Temporal Service unloads it from memory and ApproximateBacklogCount reports zero until reloaded, even if backlogged Tasks exist. Third, sticky queue exclusion: Sticky queues are not included in these values.

ApproximateBacklogCount and ApproximateBacklogAge metrics

ApproximateBacklogCount represents the approximate count of Tasks currently backlogged in a Task Queue. The number may include expired Tasks as well as active Tasks, but it will eventually converge to the correct count over time. ApproximateBacklogAge returns the approximate age of the oldest Task in the backlog, based on the creation time of the Task at the head of the queue. Both values can be relied upon when making scaling decisions.

BacklogIncreaseRate calculation and interpretation

BacklogIncreaseRate approximates the net Tasks per second added to the backlog, averaged over the most recent 30 seconds. It is calculated as TasksAddRate minus TasksDispatchRate. Positive values of X indicate the backlog is growing by about X Tasks per second. Negative values of X indicate the backlog is shrinking by about X Tasks per second. While individual add and dispatch rates may be inaccurate due to Eager and Sticky Task Queues, BacklogIncreaseRate reliably reflects the rate at which the backlog is shrinking or growing for backlogs older than a few seconds.

Task Queue types and Task types

The Temporal Service dynamically creates different Task Queue types including Activity Task Queues, Workflow Task Queues, and Nexus Task Queues. Worker Entities poll the queue for Tasks and retrieve Tasks to work on. Tasks are contexts that a Worker progresses using a specific Workflow Execution, Activity Execution, or a Nexus Task Execution. Each Task Queue type offers its Tasks to compatible Workers for Task completion.

schedule_to_start latency tuning approach

If you notice high schedule_to_start metrics, follow these tuning steps in recommended order: (1) Provision additional Worker hosts if currently provisioned hosts are fully utilized with near full CPU usage or high load average. (2) Adjust Worker Executor Slots sizing if Worker hosts are underutilized and available slots are frequently depleted. (3) Increase the poller count if Worker hosts are underutilized and a significant percentage of slots are available. (4) Check and remove or adjust rate limiting if previous steps don't resolve elevated schedule_to_start.

Give your agent this brain