Rate limiting configuration for Task Queue and Worker
If after adjusting poller and executor count you still observe elevated schedule_to_start, underutilized Worker hosts, or high worker_task_slots_available, check rate limiting settings. If server-side rate limiting per Task Queue is set by WorkerOptions#maxTaskQueueActivitiesPerSecond, remove the limit or adjust the value up (available in Go and Java SDKs). If Worker-side rate limiting per Worker is set by WorkerOptions#maxWorkerActivitiesPerSecond, remove the limit (available in Go, TypeScript, and Java SDKs).
Poller count tuning conditions
Consider increasing the number of task pollers if: (1) Worker hosts are underutilized (no bottlenecks on CPU, load average, etc.), (2) worker_task_slots_available metric shows that a significant percentage of Worker slots are available on a regular basis, and (3) the schedule_to_start metric is abnormally long. Increase the number of pollers by adjusting maxConcurrentWorkflowTaskPollers or maxConcurrentActivityTaskPollers, depending on which type of schedule_to_start metric is elevated. Automated poller tuning can handle this automatically.
Worker Executor Slots sizing recommendations
Increase the maximum number of working slots by adjusting maxConcurrentWorkflowTaskExecutionSize or maxConcurrentActivityExecutionSize if both conditions are met: (1) Worker hosts are underutilized (no bottlenecks on CPU, load average, etc.), and (2) the worker_task_slots_available metric from the corresponding Worker type frequently shows a depleted number of available Worker slots. Alternatively, consider using a resource-based slot supplier.
Poll Success Rate calculation and target
Poll Success Rate is calculated as (poll_success + poll_success_sync) / (poll_success + poll_success_sync + poll_timeouts). Poll Success Rate should be greater than 90% in most cases of systems with a steady load. For high volume and low latency systems, try to target greater than 95%. If Poll Success Rate is low and schedule_to_start_latency is low and Worker hosts resource utilization is low at the same time, you might have too many workers and should consider sizing down.
DescribeTaskQueueEnhanced method for retrieving Task Queue metrics
Task Queue metrics can be retrieved using the DescribeTaskQueueEnhanced method in the Go SDK, the Temporal CLI task-queue describe command, or using DescribeTaskQueue through RPC. The Temporal Service reports information separately for each Task Queue type (not aggregated).
Go SDK DescribeTaskQueueEnhanced example
for _, taskQueueName := range taskQueueNames {
resp, err := s.client.DescribeTaskQueueEnhanced(ctx, client.DescribeTaskQueueEnhancedOptions{
TaskQueue: taskQueueName,
ReportStats: true,
})
if err != nil {
log.Printf("Error describing task queue %s: %v", taskQueueName, err)
}
// Get the backlog count from the enhanced response
backlogCount += getBacklogCount(resp)
}
This example shows how to call DescribeTaskQueueEnhanced in the Go SDK to retrieve Task Queue statistics by specifying the Task Queue name and setting ReportStats to true.
Temporal CLI task-queue describe command
Use the command 'temporal task-queue describe --task-queue YourTaskQueueName [additional options]' to display a list of active Workers that have recently polled a Task Queue. This command retrieves poller information, backlog statistics, and task reachability for Task types (available in Temporal Server v1.25.0, Temporal CLI 1.1 and later). Task reachability status is experimental and determining Task reachability incurs a non-trivial computing cost.
LastAccessTime interpretation for Worker fleet capacity
Each Temporal Server records the last time of each poll request, displayed in the temporal task-queue describe output. A LastAccessTime value exceeding one minute may indicate that the Worker fleet is at capacity or that Workers have shut down or been removed. Values under 5 minutes typically suggest the Worker fleet is at capacity, meaning all Workflow and Activity slots are full. Values over 5 minutes since the last poll request usually suggest that Workers have shut down or been removed, as Workers are removed if 5 minutes have passed since the last poll request.
TasksAddRate and TasksDispatchRate metrics
TasksAddRate and TasksDispatchRate report the approximate Tasks-per-second added to or dispatched from a Task Queue. This rate is averaged over the most recent 30-second time interval. The calculations include Tasks added to or dispatched from the backlog as well as Tasks that were immediately dispatched and bypassed the backlog (sync-matched). However, the actual Task delivery count may be significantly higher: Eager dispatch allows Activities to be requested by an SDK within a Workflow Task completion response without passing through Task Queues, and Tasks passed to Sticky Task Queues are not included in these returned values.
maxWorkflowThreadCount and workflow_active_thread_count are Java SDK only
The maxWorkflowThreadCount and workflow_active_thread_count parameters are specific to the Java SDK only and are not available in other SDKs.
When to decrease workflow cache limits
If Worker hosts do not have enough free RAM or are close to reasonable thread limits, consider decreasing the workflowCacheSize and maxWorkflowThreadCount limits.
When to increase workflow cache limits
If the Workflow Cache limits (workflowCacheSize and maxWorkflowThreadCount) are hit, and Worker hosts have enough free RAM and are not close to reasonable thread limits, you may choose to increase workflowCacheSize and maxWorkflowThreadCount limits to decrease the overall latency and cost of Replays in the system.
Impact of workflow cache eviction
An evicted Workflow Execution will need to be replayed when it gets any action that may advance it.
Workflow cache eviction triggers
Workflow Executions are evicted from the cache when either the number of cached Workflow Executions reported by sticky_cache_size hits workflowCacheSize, or the number of threads reported by the workflow_active_thread_count metrics gauge hits maxWorkflowThreadCount.
Workflow cache metric behavior in CoreSDK-based SDKs
In CoreSDK-based SDKs, like TypeScript, the workflow cache metric works differently and should be monitored and adjusted on a per Worker and Task Queue basis.
Temporal Cloud authentication methods for Workers
Temporal Cloud supports two secure authentication methods for Workers: mTLS Certificates and API Keys (configured via the UI when creating a namespace). Both options help secure communication between workers and Temporal Cloud.
Worker certificate rotation without restart support
Per-SDK code for rotating a Worker's mTLS client certificate without a restart is documented on each SDK's Temporal Client page, in the 'Connect to Temporal Cloud' section. Go, Java, Python, .NET, Ruby, and TypeScript are all supported for certificate rotation without restart. PHP and Rust are not yet supported. The temporal-worker-cert-rotation reference implementation walks through automating this with cert-manager on Kubernetes.
Lambda worker default settings
Lambda worker packages apply conservative defaults suited to short-lived Lambda invocations.
For .NET (Temporalio.Extensions.Aws.Lambda):
- MaxConcurrentActivities: 2
- MaxConcurrentWorkflowTasks: 10
- MaxConcurrentLocalActivities: 2
- MaxConcurrentNexusTasks: 5
- MaxConcurrentWorkflowTaskPolls: 2
- MaxConcurrentActivityTaskPolls: 1
- MaxConcurrentNexusTaskPolls: 1
- MaxCachedWorkflows: 30
- GracefulShutdownTimeout: 5 seconds
- DisableEagerActivityExecution: always true
- ShutdownDeadlineBuffer: 7 seconds
For TypeScript/JavaScript (@temporalio/lambda-worker):
- maxConcurrentActivityTaskExecutions: 2
- maxConcurrentWorkflowTaskExecutions: 10
- maxConcurrentLocalActivityExecutions: 2
- maxConcurrentNexusTaskExecutions: 5
- workflowTaskPollerBehavior: SimpleMaximum(2)
- activityTaskPollerBehavior: SimpleMaximum(1)
- nexusTaskPollerBehavior: SimpleMaximum(1)
- maxCachedWorkflows: 30
- shutdownGraceTime: 5 seconds
- shutdownDeadlineBufferMs: 7000
- Eager Activities: not supported