workers & task queues

97 notes in this subject, read out of this brain and free to use. This is page 1 of 2.

Priority Task Queues pattern overview

The Priority Task Queues pattern assigns a PriorityKey to Workflows, Activities, and Child Workflows so that time-sensitive work executes ahead of lower-priority work within a single Task Queue. The Temporal matching service maintains a sub-queue for each priority level and exhausts all tasks at a given level before dispatching to the next.

Activity and Child Workflow priority inheritance

Activities and Child Workflows inherit the parent Workflow's priority unless they explicitly set their own PriorityKey.

Priority Task Queues enabled by default

Priority is enabled by default in Temporal Cloud and self-hosted Temporal.

Set Workflow priority at start in Python

In Python, set Workflow priority using the `priority` parameter in `client.start_workflow()` with `Priority(priority_key=N)` where N is 1–5.

PriorityKey range and default value

PriorityKey is an integer from 1 to 5, where 1 is the highest priority and 5 is the lowest priority. Tasks default to priority 3 when no key is set.

Set Workflow priority at start in Go

In Go, set Workflow priority in `client.StartWorkflowOptions` with `Priority: temporal.Priority{PriorityKey: N}` where N is 1–5.

Set Workflow priority at start in Java

In Java, set Workflow priority using `WorkflowOptions.newBuilder().setPriority(Priority.newBuilder().setPriorityKey(N).build())` where N is 1–5.

Common pitfall: assigning priority 1 to all work by default

When every caller sets the highest priority, the feature provides no ordering benefit. Establish an explicit policy for which work types qualify for each priority level.

Common pitfall: neglecting low-priority starvation

Under sustained high load, priority-5 tasks may wait indefinitely. Use `ScheduleToStartTimeout` on low-priority activities to surface starvation as a visible failure.

Common pitfall: changing priority after scheduling

PriorityKey is evaluated when a task enters the queue and cannot be changed while it waits. To re-prioritize an already-queued task, cancel it and reschedule with the new priority.

Common pitfall: assuming hard isolation between priority levels

Priority controls dispatch order, not Worker capacity allocation. A priority-5 task may still consume a Worker slot that is then unavailable for a priority-1 task arriving a moment later.

Temporalio.Extensions.Aws.Lambda NuGet package purpose

The Temporalio.Extensions.Aws.Lambda NuGet package lets you run a Temporal Serverless Worker on AWS Lambda. You deploy your Worker code as a Lambda function, and Temporal Cloud invokes it when Tasks arrive. Each invocation starts a Worker, polls for Tasks, then gracefully shuts down before a configurable invocation deadline. You register Workflows and Activities the same way you would with a standard Worker.

TemporalLambdaWorker.CreateHandler usage

Use TemporalLambdaWorker.CreateHandler to create a Lambda handler that runs a Temporal Worker. Pass a WorkerDeploymentVersion and a configure callback that registers your Workflows and Activities. Assign the result to a static field so the handler is created once during Lambda cold start and reused across invocations.

Lambda config file resolution order

The Temporalio.Extensions.Aws.Lambda package automatically loads Temporal client configuration from a TOML config file and environment variables. The location of the config file is resolved in the following order: 1) TEMPORAL_CONFIG_FILE environment variable, if set; 2) temporal.toml in $LAMBDA_TASK_ROOT (typically /var/task); 3) temporal.toml in the current working directory. The file is optional. If absent, only environment variables are used.

Lambda worker default concurrency and timeout settings

The Temporalio.Extensions.Aws.Lambda package applies conservative defaults: MaxConcurrentActivities=2, MaxConcurrentWorkflowTasks=10, MaxConcurrentLocalActivities=2, MaxConcurrentNexusTasks=5, MaxConcurrentWorkflowTaskPolls=2, MaxConcurrentActivityTaskPolls=1, MaxConcurrentNexusTaskPolls=1, MaxCachedWorkflows=30, GracefulShutdownTimeout=5 seconds, DisableEagerActivityExecution=always true, ShutdownDeadlineBuffer=7 seconds.

ShutdownDeadlineBuffer purpose on Lambda

ShutdownDeadlineBuffer is specific to the Temporalio.Extensions.Aws.Lambda package. It controls the time reserved after the worker run budget for worker shutdown and hooks. The default is 7 seconds.

AWS Lambda .NET CA loading issue workaround

Some AWS Lambda .NET images override the SSL_CERT_FILE environment variable in a way that prevents the SDK's Rust-based runtime from loading system root CAs. If you encounter TLS certificate errors on Lambda, see the AWS Lambda .NET CA loading workaround in the SDK README.

Serverless Workers definition

Serverless Workers run on ephemeral, on-demand compute rather than long-lived processes. Temporal invokes the Worker when Tasks arrive, and the Worker shuts down when the work is done.

.NET SDK serverless workers AWS Lambda support

The .NET SDK supports AWS Lambda as a serverless provider through the Temporalio.Extensions.Aws.Lambda NuGet package, which enables running a Worker as a Lambda function with setup, configuration, Lambda-tuned defaults, observability, and invocation lifecycle support.

Max concurrent workflow tasks configuration

For Entity Workflows, set max_concurrent_workflow_tasks higher than the default because Entity Workflows spend most time blocked on wait_condition, allowing one Worker to host many concurrent instances.

Python Worker configuration for Entity Workflows

Configure Worker with high max_concurrent_workflow_tasks (e.g., 200) and max_concurrent_activities (e.g., 100) since Entity Workflows spend most time blocked on wait_condition. Use a ThreadPoolExecutor for activity_executor to handle I/O-bound Activity work.

Task Queue rate limiting configuration options

The Python SDK provides four rate limiting controls for Task Queues: 1. max_task_queue_activities_per_second (global): Limits Activity dispatch across ALL Workers on this queue. Enforced by Temporal Server. Best for API rate limits. Last value wins. 2. max_activities_per_second (per-worker): Limits Activities per Worker. Can be combined with global limit for finer control. 3. max_concurrent_activities (concurrency): Limits concurrent executions. Use when API has concurrent connection limits (for example, database pool size). 4. disable_eager_activity_execution (Client configuration): Set to True when starting Workflows to prevent Activities from being eagerly assigned to the Workflow Worker, ensuring they go through the rate-limited Task Queue instead.

Dynamic Task Queue rate limit adjustment with CLI

Operators can adjust Task Queue rate limits dynamically without redeploying workers using the command: temporal task-queue config set --queue-rps-limit 99 --task-queue sendgrid-api. This allows operators to adjust rate limits in response to API provider limit changes or operational needs.

Recommended practice for rate-limited APIs: separate Task Queue per API

Create one Task Queue per rate-limited API to isolate rate limits. This ensures Activities calling external APIs never exceed their rate limits, preventing 429 errors and account issues. The global rate limit applies across all Workers on that queue, enabling independent scaling without exceeding API limits.

Task Queue backlog draining considerations during throttling

During throttling events, Task Queues can grow significantly. Operators need to switch to draining mitigation mode when backlogs occur. Critical considerations include: determining acceptable SLA/SLO for draining (hours or days), and understanding that simply increasing downstream API or Temporal Cloud rate limits may move the bottleneck elsewhere, potentially overwhelming the next component in line, causing cascading failures.

Mitigation strategies for Task Queue backlogs in order of consideration

When Task Queue backlogs occur during throttling, consider these mitigation strategies in order: (1) Request downstream API rate limit increases from API providers, (2) Request Temporal Cloud rate limit increases if using Temporal Cloud, (3) Scale Worker pools by adding more Workers or adjusting Worker configuration, (4) Increase internal resources by scaling infrastructure, (5) Identify the next bottleneck to determine what will become throttled next to prevent cascading failures.

Temporal Server version requirement for max_task_queue_activities_per_second

Temporal Server v1.17+ is required for max_task_queue_activities_per_second support for global rate limiting across all Workers on a queue.

Python SDK Activity routing to rate-limited Task Queues example

When executing Activities in Workflows, specify the task_queue parameter to route to rate-limited queues: await workflow.execute_activity('send_email', email_data, task_queue=SENDGRID_API_QUEUE, start_to_close_timeout=timedelta(minutes=2), retry_policy=workflow.RetryPolicy(...)). This ensures the Activity is dispatched through the specific rate-limited Task Queue rather than executing eagerly on the Workflow Worker.

Python SDK Worker configuration with rate limiting example

Create a Worker with rate limiting using: Worker(client, task_queue=SENDGRID_API_QUEUE, activities=[send_email, send_batch_email], max_task_queue_activities_per_second=1.5, disable_eager_activity_execution=True, max_activities_per_second=0.5, max_concurrent_activities=20). The max_task_queue_activities_per_second applies globally across all workers on the queue, while max_activities_per_second limits per-worker, and max_concurrent_activities limits concurrent executions.

Python SDK client-side disable_eager_activity_execution example

When starting Workflows, disable eager execution to ensure activities go through rate-limited Task Queues: await client.start_workflow(NotificationWorkflow.run, request, id=f'notification-{i}', task_queue='workflows', disable_eager_activity_execution=True).

Deployment guidance for rate-limited Task Queues

Deploy 2-5 Workers per API-specific Task Queue. The rate limit is enforced globally by Temporal Service across all Workers on that queue. More Workers increase fault tolerance, but the rate limit still applies globally. Rate-limited Workers typically have low CPU/memory utilization and monitor accordingly.

Task Queues are dynamically created in Temporal

Task Queues in Temporal are dynamically created when first referenced. Rate limiting is configured at the Worker level and enforced by the Temporal Server.

Worker execution affinity pattern: Shared + Unique queues

To ensure all Activities in a Workflow execute on the same Worker, implement a two-tier Task Queue pattern: Each Worker polls both a shared queue and a unique queue (generated per Worker instance using UUID). The Workflow calls an Activity on the shared queue to discover an available Worker's unique queue name, then routes all subsequent Activities to that Worker's unique queue. All Activities then execute on the same Worker, maintaining data locality.

Unique Task Queue naming convention

Generate unique queue names using UUIDs to avoid collisions across Worker instances. The recommended pattern is `{base-name}-{uuid.uuid4()}`, for example `file-processing-abc123`. Alternatively, use the hostname if running in a containerized environment.

Worker Sessions API in Go SDK

The Go SDK has a built-in Worker Sessions API that handles Worker-specific routing automatically. Other SDKs such as Python and TypeScript must implement the worker affinity pattern manually using unique Task Queue names, which provides the same guarantees as Go's Sessions API.

Worker failure handling with Schedule-to-Start Timeout

If a Worker crashes while processing Activities on its unique queue, the Schedule-to-Start Timeout detects the failure by recognizing that the Worker will no longer dequeue Tasks. Set this timeout to a short duration (for example, 5 minutes for file processing) to quickly detect Worker failures. This timeout limits the maximum amount of time that a Task may remain enqueued before being marked as failed.

Heartbeat Timeout for detecting mid-execution worker crashes

Activities send periodic heartbeats to signal they are alive. If a Worker crashes mid-execution, no heartbeats are sent and the Activity fails within the heartbeat_timeout duration (for example, 30 seconds) rather than waiting for the full Start-to-Close Timeout. Recommended practice is to set a short heartbeat_timeout of 30 seconds to detect crashes quickly.

Worker failure recovery strategy in workflows

When a Worker crashes, activities scheduled to its unique queue will timeout. Implement failure detection and recovery using: (1) Heartbeat Timeout for fastest detection of mid-execution crashes (30s), (2) Schedule-to-Start Timeout for detection of crashes before picking up tasks (5 min), (3) Workflow-level retries with a loop that catches all exceptions and retries the entire sequence on a different Worker. After detecting failure, the Workflow should request a new unique queue from a healthy Worker rather than retrying on the failed Worker's queue.

Worker-specific Task Queue pattern use cases

Worker execution affinity is required for: (1) File processing where files are downloaded to a Worker's local disk and must be available for subsequent Activities without re-downloading multi-GB files, (2) ML model caching where a large model is loaded into memory once and reused across multiple inference Activities, (3) Database connection pooling where expensive connections are established and reused across multiple Activities in the same Workflow Execution.

Data locality benefits of worker affinity

Using worker-specific Task Queues ensures files downloaded in one Activity are immediately available to subsequent Activities on the same Worker, eliminating multi-GB network transfers and reducing execution time by 80% or more for Workflows processing large files. This also reduces network egress costs from transferring large files between Workers.

Resource efficiency with worker affinity

Expensive resources such as ML models and database connections can be loaded once per Workflow instead of per Activity, reducing memory usage and initialization overhead. This avoids duplicate resource initialization when Activities execute on different Workers.

Python SDK get_available_task_queue activity implementation

The get_available_task_queue Activity returns this Worker's unique task queue name as a string. The Workflow calls this Activity on the shared queue to discover a Worker's unique queue, then routes all subsequent Activities to that queue. Example: activity.logger.info(f"Returning unique queue: {UNIQUE_WORKER_TASK_QUEUE}") and returns the queue name.

Python SDK worker configuration for affinity pattern

Configure two Workers in the same process: (1) A shared_worker that polls the shared queue (FILE_PROCESSING_SHARED_QUEUE) and registers the get_available_task_queue activity. (2) A unique_worker that polls the unique queue (UNIQUE_WORKER_TASK_QUEUE) and registers the actual processing activities (download_file, process_file, upload_file). Run both Workers concurrently using asyncio.gather(shared_worker.run(), unique_worker.run()). Set max_concurrent_activities based on disk I/O limits.

Queue lifecycle and cleanup considerations

Each Worker creates a unique queue that persists until activities complete. Monitor total queue count to avoid proliferation. Consider setting a reasonable schedule_to_close_timeout on Activities to bound how long work tied to a unique queue can remain active, or implement explicit cleanup jobs to remove stale unique queues when Workers terminate. When scaling down, ensure Workers complete in-progress Workflows before termination.

Public client configuration

The publicClient is a required section describing configuration for a worker to connect to Temporal server for background server maintenance. It contains hostPort (IPv4 host port or DNS name to reach Temporal frontend) and supports dns:/// prefix for round-robin between IP addresses for DNS names.

Global membership broadcastAddress configuration

The broadcastAddress in the membership section is used by the gossip protocol to communicate with other hosts in the same Cluster for membership info. Use an IP address that is reachable by other hosts in the same Cluster. If there is only one host in the Cluster, you can use 127.0.0.1. Only IPv4 is supported via net.ParseIP syntax.

Global maxJoinDuration default

The maxJoinDuration parameter controls the amount of time the service will attempt to join the gossip layer before failing. Default is 10s.

Services configuration roles

The services section contains configuration keyed by service role type. There are four supported service roles: frontend, matching, worker, and history.

Service RPC configuration section

Each service has an rpc section (required) containing: grpcPort (port on which gRPC listens), membershipPort (port for membership info communication with other hosts, each service should use different port), bindOnLocalHost (determines whether uses 127.0.0.1 as listener address), and bindOnIP (binds service on specific IP or 0.0.0.0, mutually exclusive with bindOnLocalHost, only IPv4 supported). Port values are currently expected to be consistent among role types across all hosts.

RESOURCE_EXHAUSTED on respond operations blocks task slot release

When the Temporal Service is throttling Workers reporting task results, the SDK retries automatically, but a delayed respond call has a compounding cost: the Worker holds the Task slot until the call succeeds, and the Service-side task stays in-flight until the response lands. Every in-flight task still holds its slot, so you have less concurrency for new work.

Poll throttling often symptom rather than cause

Throttling on poll operations is often a symptom rather than a cause. The Temporal Service throttles poll operations before it throttles respond or user-facing operations, so throttling here can be the first visible sign of pressure that has nothing to do with your Workers.

Over-polling risks in worker configuration

A large number of Workers each configured with many concurrent pollers can exceed the Namespace poller limit without processing any more work. Check your configured poller counts against Worker performance guidance before assuming the limit is too low.

Self-hosted Temporal Service persistence latency on UpdateWorkflowExecution

In self-hosted Temporal Service environments, slow persistence on `UpdateWorkflowExecution` is the usual root cause of throttling on respond operations. Check persistence latency filtered to that operation.

Self-hosted Temporal Service frontend.namespaceCount poller limit

In self-hosted Temporal Service deployments, if poll operations are being throttled at scale, you may need to raise the Namespace concurrent poller limit through `frontend.namespaceCount` or `frontend.globalNamespaceCount`. Scale Worker capacity first if schedule-to-start latency is the real problem.

RESOURCE_EXHAUSTED on poll operations increments temporal_long_request_failure

When the Temporal Service is throttling Worker poll calls on PollWorkflowTaskQueue or PollActivityTaskQueue, these increment `temporal_long_request_failure` rather than `temporal_request_failure`, because poll operations are long-poll requests.

Throttled workers lower effective poll rate

When the Temporal Service throttles Worker poll calls, Workers back off and poll less frequently, which lowers the effective poll rate for the Task Queue even when every Worker is healthy. That shows up as rising schedule-to-start latency and, if it persists, as a growing Task backlog.

High temporal_request_failure_total causes and diagnosis

temporal_request_failure_total metric counts number of RPC requests made by Temporal Client that failed. Causes: Network Issues (problems with network connection between Client and Server), Client Errors (misconfiguration or resource exhaustion in Client), Operation Errors (operations like SignalWorkflowExecution or TerminateWorkflowExecution fail acting on closed Workflow Execution that no longer exists because it completed and was removed from persistence at Namespace retention time), Rate Limiting (rate of requests exceeds configured limits on server, often indicated by ResourceExhausted status code), Request Size Limit (Worker tries to return Activity response larger than blob size limit of 2MB, service rejects it), and Server Errors (Temporal Server experiencing issues fails to respond correctly). Diagnosis: check status or code tag of metric to see error types, look at operation tag to see which operations failing, monitor Temporal Server logs and Client logs for error messages and warnings, and check network connection between Client and Server.

High temporal_request_latency causes and diagnosis

temporal_request_latency measures latency of gRPC requests made by Temporal Client. High values caused by: Network Latency (physical distance and network conditions between Client and Server affect latency), Network Transfer Time (larger payloads take longer to transfer, especially large payloads in RespondWorkflowTaskCompleted and when Workflows schedule multiple activities with large inputs), Resource Exhaustion (running out of CPU or memory on client or server causes delays), Client Configuration (improper configuration like thread pool sizes set too aggressively or memory constraints too low for allocated threads cause Tasks to overwhelm client), and Server Load (Temporal Server under heavy load takes longer to respond). Diagnosis and remediation: monitor temporal_request_latency metric to identify when and where spikes occur, check network connection between Client and Server, monitor resource usage on both Client and Server, review Temporal Client configuration for optimization, and if using Temporal Cloud check if Cloud's service-latency metric spikes and contact Temporal Support.

rate(temporal_long_request_total{operation="PollWorkflowTaskQueue"})

The rate(temporal_long_request_total{operation="PollWorkflowTaskQueue"}) expression measures per-second average rate of PollWorkflowTaskQueue long poll requests over a time period. PollWorkflowTaskQueue is operation where Workers poll for Workflow Tasks from Task Queue. temporal_long_request_total metric counts number of these long poll requests. Applying rate() function in Prometheus calculates per-second average rate over time range specified in query. This helps understand load on Temporal service and how often Workers poll for Workflow Tasks.

High temporal_long_request_failure causes and diagnosis

temporal_long_request_failure metric counts number of failed RPC long poll requests for PollWorkflowTaskQueue, PollActivityTaskQueue, and GetWorkflowExecutionHistory (when polling new events). High values caused by: Network Issues (problems with network connection between Temporal Client and Server including firewalls and proxies), Rate Limiting (rate of requests exceeds configured limits on server or Temporal Cloud, often indicated by ResourceExhausted status code), and Server Errors (Temporal Server experiencing issues fails to respond correctly to long poll requests). Diagnosis: check operation and status or code tag of temporal_long_request_failure metric to see error types, if ResourceExhausted status code review rate limits configured on server or contact Temporal Support for Cloud, and check network connection between Client and Server.

Give your agent this brain