workers

109 notes in this subject, read out of this brain and free to use. This is page 2 of 2.

TypeScript SDK Temporal Cloud environment variables

Required environment variables to connect to Temporal Cloud: TEMPORAL_NAMESPACE (Namespace and Account ID in format <namespace_id>.<account_id>), TEMPORAL_ADDRESS (gRPC endpoint), TEMPORAL_API_KEY (for API key authentication), TEMPORAL_TLS_CLIENT_CERT_DATA or TEMPORAL_TLS_CLIENT_CERT_PATH (for mTLS authentication), TEMPORAL_TLS_CLIENT_KEY_DATA or TEMPORAL_TLS_CLIENT_KEY_PATH (for mTLS authentication).

Update API key on existing connection

To update an API key on an existing Connection object, use the setApiKey method: connection.setApiKey(<APIKey>).

Rotate mTLS certificate on running Worker

Starting with @temporalio/worker v1.15.0, you can replace a running Worker's connection to rotate an mTLS client certificate without restarting the Worker. Create a new NativeConnection with the new certificate and assign it to worker.connection. The Worker starts using the new connection for subsequent calls while already in-flight calls on the old connection finish normally.

Task Queue is the only required Worker option

The taskQueue is the only required option when creating a Worker. For any code to execute, a Worker Process must be running that contains a Worker Entity that is polling the same Task Queue name.

Worker requires three main configurations

The Worker needs: (1) taskQueue - the Task Queue to poll (only required argument), (2) activities - optional imported and supplied directly to the Worker, and (3) Workflow bundle - either specify workflowsPath pointing to workflows.ts file for Webpack processing, or pass a prebuilt bundle to workflowBundle.

How Workers bundle and execute code

Workers bundle Workflow code and node modules using Webpack v5 and execute them inside V8 isolates. Activities are directly required and run by Workers in the Node.js environment.

Connection class for Temporal Application and Activity connections

To connect to a development Temporal Service from a Temporal Application or from within an Activity, import the Connection class from @temporalio/client and use Connection.connect() to create a Connection object. If connection options are left empty, the SDK defaults to connecting to 127.0.0.1:7233 in the default Namespace.

Set fairness weight overrides via CLI

Override the weights of up to 1000 keys through the config API using temporal task-queue config set. When an override is set, the weight on the Task is ignored and the overridden weight is used: ``` temporal task-queue config set \ --task-queue my-task-queue \ --task-queue-type activity \ --namespace my-namespace \ --fairness-key-weight premium=5.0 \ --fairness-key-weight basic=1.0 ``` To unset a single key's override, pass `key=default`. To clear all overrides, use `--fairness-key-weight-clear-all`.

Fairness weight overrides per Task Queue

Weight overrides are stored per Task Queue, including type, so they must be set for both Workflow and Activity Task Queues to take effect for both.

Fairness limitations: multiple fairness weights on same key

There should only be one fairness weight assigned to each fairness key within a Task Queue. Having multiple fairness weights on a fairness key will result in unspecific behavior.

Fairness limitations: accuracy with many keys

There isn't a limit on the number of fairness keys you can use, but their accuracy can degrade as you add more.

Fairness limitations: partition imbalance

Fairness is enforced within a single Task Queue partition. When a Task Queue's partitions are imbalanced, Fairness may not appear to hold, since it applies only within individual partitions. Depending on your use case, you can reach out to Temporal Support to get your Task Queues set to a single partition.

Fairness limitations: weight applied at schedule time

The fairness weight applies at schedule time, not at dispatch time. So it only affects newly-scheduled Tasks, not currently backlogged ones. This means if you need to throttle a single fairness key in the existing backlog, you won't be able to.

Fairness limitations: Worker Versioning interactions

When you use Worker Versioning and you're moving Workflows from one version to another, Priority still applies between versions. Fairness isn't guaranteed between versions. Fairness is only guaranteed when Tasks are originally queued on the same Worker version. There might be discrepancies on Tasks moved between versions.

Fairness limitations: server restart and state preservation

During server restarts, Temporal preserves fairness state for the top 100 keys. Other keys rebuild their fairness state as new Tasks arrive, which can temporarily distort weighted dispatch. Fairness pass dithering reduces this distortion by spreading keys' initial positions according to their weights. It can reduce FIFO-like ordering among equal-weight keys. To enable fairness pass dithering, contact Temporal Support.

Fairness limitations: doesn't consider already-dispatched executions

Fairness doesn't consider Task executions that have already been dispatched to Workers. As a result, fair dispatch may not be immediately visible in the mix of Tasks currently running on Workers.

Task Queue Priority range and defaults

Priority keys are set to an integer value in the range [1, 5], where lower values indicate higher priority (1 is highest). If no priority is specified, Tasks default to priority 3. Priority is enabled by default in both Temporal Cloud and self-hosted Temporal.

Task Queue Priority: partition-level enforcement

Priority is enforced within a single Task Queue partition. A Task Queue by default uses multiple partitions and randomly distributes Tasks across them, so partitions are usually balanced and priority ordering closely approximates the Task Queue as a whole. When partitions become imbalanced, lower-priority Tasks on a lighter partition can dispatch ahead of higher-priority Tasks waiting on a heavier partition.

FIFO ordering within priority levels

Tasks are scheduled by default to run in first-in-first-out (FIFO) order within each priority level. If you need greater control of task ordering within a priority level, such as preventing large tenants from overwhelming small tenants, use Fairness.

When to use Priority: real-time vs batch separation

Use Priority to manage the execution order of Tasks when you need to differentiate between Tasks, like batch and real-time Tasks, so that you can use a single pool of Workers for efficient resource allocation, while ensuring real-time Tasks are processed ahead of batch Tasks.

When to use Priority: urgent task override

Use Priority as a way to run urgent Tasks immediately and override others. For example, in an e-commerce platform, you may want to process payment-related Tasks before less time-sensitive Tasks like internal inventory management.

Enabling or disabling Fairness with active backlog: existing queue dispatch order

When Fairness is enabled on a Namespace with existing Tasks, the existing queued Tasks are dispatched first in their original priority + FIFO order. Fairness keys on Tasks already in the backlog do not retroactively affect their dispatch order. When Fairness is disabled, the existing fairness-ordered backlog is dispatched first in its original fairness order. After the backlog drains, Tasks dispatch in priority + FIFO order. In both directions, the existing backlog is dispatched before any new Tasks queued under the new mode. New Tasks dispatch only after the backlog fully drains.

Task Queue Fairness: virtual queues and round-robin dispatch

Task Queue Fairness lets you distribute Tasks based on fairness keys and fairness weights. Each fairness key creates a virtual queue organized into logical groups like tenants or applications. These virtual queues operate using round-robin dispatch, cycling through each fairness key in turn to prevent any single fairness key from monopolizing Worker capacity.

Fairness weight: default and customization

By default, each fairness key is weighted equally with a fairness weight of 1.0. This can be customized by assigning a different fairness weight to a key. For example, Tasks with a fairness weight of 2.0 will be dispatched twice as often as keys with weight 1.0.

Fairness weight scaling with rate limits

The per-fairness-key rate limit is scaled by the fairness weight assigned to that key. If the per-fairness-key limit for a queue is set to 10, then all keys with the default weight (1.0) will have a limit of 10 tasks/second. But if a key is given a weight of 2.5, then the per-key rate limit for that key will be 25 tasks/second.

Unkeyed Tasks in Fairness: implicit empty-string key

Tasks that do not have a fairness_key set are grouped together under an implicit empty-string key. All unkeyed Tasks share this single default bucket and participate in the same round-robin dispatch alongside named fairness keys, with a default weight of 1.0. This means Fairness adoption can be incremental.

Priority and Fairness combined: three-rule dispatch order

When Priority and Fairness are used together, Tasks are dispatched by walking three rules in order: 1) Priority tier (strict) - Tasks at higher priority always dispatch before lower priority tasks, regardless of fairness keys or weights. 2) Fairness key within a tier (weighted) - Within a priority tier, each fairness key is dispatched proportional to its weights. 3) FIFO within a key - Tasks sharing a priority tier and fairness key dispatch in the order they were enqueued. These rules apply within a Task Queue partition.

When to use Priority alone vs Fairness alone vs both

Use Priority alone when you need strict priority ordering (e.g., separating real-time Tasks from batch Tasks). Use Fairness alone when you need tier or tenant isolation so no group is starved, but you don't need to preempt any group ahead of another. Use both when you have a tiered SLA hierarchy - Priority for the broad tier (e.g., paid vs. free) and Fairness for per-tenant equity within a tier.

Enable Fairness in Temporal Cloud

To enable Fairness for a Namespace in Temporal Cloud, navigate to the Namespace's Overview page in the UI and activate the Fairness toggle. Fairness is a paid feature in Temporal Cloud.

Enable Fairness in self-hosted Temporal

To enable Fairness in self-hosted Temporal, set `matching.enableFairness` to `true` in the dynamic config on the relevant Task Queues or Namespaces.

Disable Priority in self-hosted Temporal

To disable Priority in self-hosted Temporal, set the dynamic config `matching.useNewMatcher` to `false` on a Task Queue, Namespace, or globally.

Set Task Queue rate limits via CLI

Within a Task Queue, set dispatch rate limits using the temporal task-queue config set command. Use `--queue-rps-limit` for the whole queue limit and `--fairness-key-rps-limit-default` for per-fairness-key limits: ``` temporal task-queue config set \ --task-queue my-task-queue \ --task-queue-type activity \ --namespace my-namespace \ --queue-rps-limit 500 \ --queue-rps-limit-reason "overall limit" \ --fairness-key-rps-limit-default 33.3 \ --fairness-key-rps-limit-reason "per-key limit" ```

Whole queue rate limits and per-fairness-key rate limits

Whole queue rate limits apply to the entire queue regardless of fairness key. This is the same setting exposed through Worker Options in SDKs; when set via API, it takes precedence over the limit set through Worker Options. Per-fairness-key rate limits work in conjunction with Task Queue Fairness, applying to each individual virtual queue. The whole queue limit and per-fairness-key limit may be set independently.

Register Nexus Service handler in Worker

Register a Nexus Service handler in a Worker by passing it in the `nexusServices` array when creating the Worker: `const worker = await Worker.create({ connection, namespace, taskQueue, workflowsPath, nexusServices: [helloServiceHandler] })`. The handler must be registered alongside the workflows and activities it uses.

Worker tuning resource constraints

Worker performance is constrained by three primary resources: Compute (CPU-bound operations and concurrent task execution), Memory (workflow cache and thread pools), and IO (network calls to Temporal Service and polling).

Compute settings configure concurrent task execution

Compute settings control how many tasks a worker can execute concurrently through: MaxConcurrentWorkflowTaskExecutionSize (maximum concurrent workflow tasks), MaxConcurrentActivityTaskExecutionSize (maximum concurrent activity tasks), MaxConcurrentLocalActivityTaskExecutionSize (maximum concurrent local activities), and MaxWorkflowThreadCount/workflowThreadPoolSize (thread pool for workflow execution).

Compute defaults by SDK

Compute configuration defaults vary by SDK: Go has MaxConcurrentWorkflowTaskExecutionSize=1000, MaxConcurrentActivityTaskExecutionSize=1000, MaxConcurrentLocalActivityTaskExecutionSize=1000; Java has 200 for each concurrent execution setting and MaxWorkflowThreadCount=600; TypeScript has MaxConcurrentWorkflowTaskExecutionSize=40, MaxConcurrentActivityTaskExecutionSize=100, MaxConcurrentLocalActivityTaskExecutionSize=100, reuseV8Context=1; Python has 100 for each concurrent execution setting; .NET has 100 for each concurrent execution setting.

Memory settings control workflow cache and thread pools

Memory settings control workflow cache size and thread pool allocation through: MaxCachedWorkflows/StickyWorkflowCacheSize (number of workflows to keep in cache) and MaxWorkflowThreadCount (thread pool size). TypeScript also has reuseV8Context to reuse V8 context for workflows.

Memory defaults by SDK

Memory configuration defaults for workflow cache vary by SDK: Go uses MaxCachedWorkflows=10,000; Java uses StickyWorkflowCacheSize=600; TypeScript uses dynamic sizing (for example, 2000 for 4 GiB RAM); Python uses MaxCachedWorkflows=1,000; .NET uses MaxCachedWorkflows=10,000.

IO settings control pollers and rate limits

IO settings control the number of pollers and rate limits for task queue interactions through: MaxConcurrentWorkflowTaskPollers (number of concurrent workflow pollers), MaxConcurrentActivityTaskPollers (number of concurrent activity pollers), Namespace APS (actions per second limit for namespace), and TaskQueueActivitiesPerSecond (activity rate limit per task queue).

IO defaults by SDK

IO configuration defaults vary by SDK: Go has MaxConcurrentWorkflowTaskPollers=2, MaxConcurrentActivityTaskPollers=2, Namespace APS=400, TaskQueueActivitiesPerSecond=Unlimited; Java has MaxConcurrentWorkflowTaskPollers=5, MaxConcurrentActivityTaskPollers=5; TypeScript has MaxConcurrentWorkflowTaskPollers=10, MaxConcurrentActivityTaskPollers=10; Python has MaxConcurrentWorkflowTaskPollers=5, MaxConcurrentActivityTaskPollers=5; .NET has MaxConcurrentWorkflowTaskPollers=5, MaxConcurrentActivityTaskPollers=5.

Worker failure recovery process

When a worker crashes or experiences a host outage, the following recovery process occurs: the workflow task times out, another available worker picks up the task, the new worker replays the event history to reconstruct state, and execution continues from where it left off.

Compute-related metrics for monitoring

Compute-related metrics for monitoring worker performance include: worker_task_slots_available {worker_type = WorkflowWorker} for MaxConcurrentWorkflowTaskExecutionSize, worker_task_slots_available {worker_type = ActivityWorker} for MaxConcurrentActivityTaskExecutionSize, workflow_active_thread_count (Java only) for MaxWorkflowThreadCount, and workflow_task_execution_latency for CPU-intensive logic. Also monitor machine CPU consumption such as container_cpu_usage_seconds_total in Kubernetes.

Memory-related metrics for monitoring

Memory-related metrics for monitoring worker performance include: sticky_cache_total_forced_eviction, sticky_cache_size, sticky_cache_hit, and sticky_cache_miss for StickyWorkflowCacheSize. Also monitor machine memory consumption such as container_memory_usage_bytes in Kubernetes.

IO-related metrics for monitoring

IO-related metrics for monitoring worker performance include: num_pollers {poller_type = workflow_task} for MaxConcurrentWorkflowTaskPollers, num_pollers {poller_type = activity_task} for MaxConcurrentActivityTaskPollers, and request_latency {namespace, operation} for network latency.

Task Queue metrics

Task queue metrics include poll_success_sync_count (sync match rate showing tasks immediately assigned to workers) and approximate_backlog_count (approximate number of tasks in a task queue). The DescribeTaskQueue API provides: ApproximateBacklogCount, ApproximateBacklogAge, TasksAddRate, TasksDispatchRate, and BacklogIncreaseRate.

Failure metrics for error monitoring

Failure metrics include: long_request_failure (failures for long-running operations like polling and history retrieval) and request_failure (failures for standard operations like task completion responses). Common failure codes are: RESOURCE_EXHAUSTED (rate limits exceeded), DEADLINE_EXCEEDED (operation timeout), and NOT_FOUND (resource not found).

Resource-based slot suppliers

Instead of fixed slot counts, resource-based slot suppliers can automatically adjust available task slots based on CPU and memory utilization. This provides a more dynamic approach to managing worker capacity based on actual resource availability.

Poller autoscaling capability

Poller autoscaling automatically adjusts the number of concurrent polls based on workload, allowing workers to dynamically adapt to changing task queue conditions.

Give your agent this brain