workers

47 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Serverless Workers run Temporal Workers on AWS Lambda

Serverless Workers let you run Temporal Workers on serverless compute like AWS Lambda without provisioning or scaling long-lived processes. Temporal Cloud invokes your Worker when Tasks arrive, and the Worker shuts down when the work is done.

Python SDK synchronous vs async Activities

Synchronous Activities are the default and recommended approach: the Python SDK runs synchronous Activities in a ThreadPoolExecutor, which is safer and easier to debug than async Activities. Async Activities are only necessary when the Activity must use async-native libraries throughout its implementation. Configure the Worker with a ThreadPoolExecutor and tune max_workers based on expected concurrent Activity executions. Size the executor based on the ratio of concurrent players to Activity execution time.

Worker Process

A Worker Process is responsible for polling a Task Queue, dequeueing a Task, executing your code in response to a Task, and responding to the Temporal Server with the results.

Worker Program

A Worker Program is the static code that defines the constraints of the Worker Process, developed using the APIs of a Temporal SDK.

Worker Entity

A Worker Entity is the individual Worker within a Worker Process that listens to a specific Task Queue.

Task Routing

Task Routing is when a Task Queue is paired with one or more Worker Processes, primarily for Activity Task Executions.

Worker Session

A Worker Session is a feature provided by some SDKs that provides a straightforward way to ensure that Activity Tasks are executed with the same Worker without requiring you to manually specify Task Queue names.

Sticky Execution

A Sticky Execution is when a Worker Entity caches the Workflow Execution Event History and creates a dedicated Task Queue to listen on.

Workflow cache

A Workflow cache is an in-memory cache on a Worker that holds the state of Workflow Executions it has processed so later Workflow Tasks can avoid a full Event History replay. It is used with Sticky Execution.

Worker registration for PermitSlotWorkflow

Register PermitSlotWorkflow in the workflows list when creating a Worker: worker = Worker(client, task_queue=TEMPORAL_TASK_QUEUE, workflows=[PermitSlotWorkflow], activities=[]). The permit lifecycle is pure Workflow code with no Activities required. For graceful shutdown in production, add a SIGTERM handler to cancel the worker.run() Task so pod restarts and rolling deploys drain in-flight work.

Worker configuration for document approval pattern

The Worker is configured with: task_queue='document-approval' (from environment or default). max_concurrent_workflow_tasks=100 to allow parallel Workflow execution. max_concurrent_activities=50 to limit concurrent Activity executions. activity_executor=ThreadPoolExecutor(max_workers=50) to offload blocking I/O from the async event loop. Workflows registered: DocumentApprovalWorkflow. Activities registered: send_notification, record_audit_entry, store_document, generate_approval_report.

GPU Worker concurrency configuration

Limit GPU Workers to 2-4 concurrent activities due to GPU memory constraints. An NVIDIA T4 with 16GB memory can typically handle 2-4 concurrent inference tasks. Adjust max_concurrent_activities if out-of-memory errors occur during operation.

CPU Worker concurrency configuration

Standard Workers can handle 100+ concurrent activities. Deploy standard CPU Workers (such as c5.xlarge with 4 vCPU and 8GB RAM) on cost-effective instances and tune concurrency based on CPU and memory availability.

High-memory Worker concurrency configuration

High-memory Workers need careful concurrency tuning to avoid out-of-memory errors. An r5.2xlarge instance with 64GB RAM can handle approximately 10 concurrent activities using 5-6GB per activity. Adjust max_concurrent_activities based on per-Activity memory usage.

GPU Worker hardware dependencies

GPU Workers require NVIDIA drivers (version 525.60 or later), CUDA toolkit (12.0 or later), and ML frameworks (PyTorch, TensorFlow) with GPU support. Use container images such as 'nvidia/cuda:12.0-cudnn8-runtime-ubuntu22.04' or 'pytorch/pytorch:2.0.0-cuda11.7-cudnn8-runtime'.

Worker operational monitoring for specialized hardware

Monitor GPU memory, compute utilization, and CPU/memory usage per Worker pool. Track schedule_to_start_latency per Task Queue to detect under-provisioning of specialized hardware. Monitor GPU temperature, driver errors, and CUDA out-of-memory errors. Tag resources by hardware type (GPU, CPU, high-memory) for cost analysis.

Python Worker configuration for CPU queue

Example Python Worker for standard CPU processing with high concurrency: ```python import asyncio import logging from temporalio.client import Client from temporalio.worker import Worker from task_queues import STANDARD_CPU_QUEUE from workflows import MLPipelineWorkflow from activities import validate_data, preprocess_data, store_results logging.basicConfig(level=logging.INFO) async def main(): client = await Client.connect("localhost:7233") worker = Worker( client, task_queue=STANDARD_CPU_QUEUE, workflows=[MLPipelineWorkflow], activities=[validate_data, preprocess_data, store_results], max_concurrent_activities=100, ) logging.info(f"Starting CPU worker on {STANDARD_CPU_QUEUE}") await worker.run() if __name__ == "__main__": asyncio.run(main()) ``` Deploy 10 instances on standard compute instances (c5.xlarge: 4 vCPU, 8GB RAM) with autoscaling based on queue depth and CPU utilization.

Python Worker configuration for GPU queue

Example Python Worker for GPU-intensive ML activities with limited concurrency: ```python import asyncio import logging from temporalio.client import Client from temporalio.worker import Worker from task_queues import GPU_ML_QUEUE from activities import generate_embeddings, run_inference, train_model logging.basicConfig(level=logging.INFO) async def main(): client = await Client.connect("localhost:7233") worker = Worker( client, task_queue=GPU_ML_QUEUE, activities=[generate_embeddings, run_inference, train_model], max_concurrent_activities=2, ) logging.info(f"Starting GPU worker on {GPU_ML_QUEUE}") await worker.run() if __name__ == "__main__": asyncio.run(main()) ``` Deploy 2-3 instances on GPU instances (g4dn.xlarge with NVIDIA T4, or p3.2xlarge with V100). GPU Workers require NVIDIA drivers (version 525.60 or later), CUDA toolkit (12.0 or later), and ML frameworks with GPU support. Monitor GPU memory usage and adjust max_concurrent_activities if OOM errors occur.

Python Worker configuration for high-memory queue

Example Python Worker for high-memory data processing with limited concurrency: ```python import asyncio import logging from temporalio.client import Client from temporalio.worker import Worker from task_queues import HIGH_MEMORY_QUEUE from activities import process_large_dataset, aggregate_analytics, build_large_index logging.basicConfig(level=logging.INFO) async def main(): client = await Client.connect("localhost:7233") worker = Worker( client, task_queue=HIGH_MEMORY_QUEUE, activities=[process_large_dataset, aggregate_analytics, build_large_index], max_concurrent_activities=10, ) logging.info(f"Starting high-memory worker on {HIGH_MEMORY_QUEUE}") await worker.run() if __name__ == "__main__": asyncio.run(main()) ``` Deploy 3-5 instances on memory-optimized instances (r5.2xlarge: 8 vCPU, 64GB RAM). Adjust max_concurrent_activities based on per-Activity memory usage to prevent OOM errors. Monitor memory utilization and swap usage.

Worker registration for capacity provisioning system

The Worker program registers both ProvisionTRUWorkflow and DeprovisionTRUWorkflow using w.RegisterWorkflow(). It creates an Activities struct with the API key from the TEMPORAL_CLOUD_API_KEY environment variable and registers it with w.RegisterActivity(activities). The Worker listens on the capacity-management task queue.

Worker capacity exhaustion symptoms

Worker capacity exhaustion symptoms include exhausted Task slots, disconnected pollers, Task completions dropping to zero, rising schedule-to-start latency, and an empty Sticky Execution cache.

Serverless Workers troubleshooting platforms

Serverless Workers can be deployed on AWS Lambda or GCP Cloud Run. Diagnosis involves tracing the flow from Task Queue to Worker execution.

Lambda invocation loop from deployment name or build ID mismatch

If CloudWatch shows rapid, repeated invocations with no Workflow progress, the deployment name or build ID in the Worker code may not match the Worker Deployment Version configuration. The deployment name and build ID in the Lambda function code must exactly match the values used when creating the Worker Deployment Version. A mismatch causes an invocation loop: the WCI invokes the Lambda, the Worker starts and polls with a different deployment version than the WCI expects, the Task is not processed, and the WCI invokes the Lambda again. To fix, update the deployment name and build ID in the Worker code to match the Worker Deployment Version, then redeploy the Lambda function.

Serverless Worker invocation flow on AWS Lambda

When a Serverless Worker invocation works correctly on AWS Lambda, the following sequence occurs: (1) The Worker function is deployed on Lambda. (2) A Worker Deployment Version is configured with a compute provider, which starts a Worker Controller Instance (WCI) Workflow and a validation invocation of the Lambda function. (3) The Lambda polls the Temporal Service successfully, binding the Task Queue configured on the Worker to the Worker Deployment Version. (4) The WCI continuously monitors the associated Task Queue on a schedule, and the Matching Service notifies the WCI Workflow of sync match failures immediately. (5) A Task arrives on the Task Queue and the WCI detects the backlog. (6) The WCI invokes the Lambda function. (7) The Lambda function starts, the Worker connects to Temporal and polls the Task Queue. (8) The Worker processes Tasks and shuts down gracefully.

Validate Connection in Temporal UI for serverless worker

In the Temporal UI, navigate to Workers > Deployments > select your deployment, open the Actions menu on the version, and click Validate Connection. A successful validation confirms that the Worker Deployment Version has a compute provider configured, that Temporal can assume the invocation role, and that the Lambda function can be invoked.

Validate Connection failure modes for Lambda

Validate Connection reports the following failure modes: (1) No compute provider configured: No Worker Controller Instance (WCI) Workflow exists and the Lambda is never automatically invoked. A common cause is manually invoking the Lambda function before creating the Worker Deployment Version in the UI or CLI. To fix, create or update the Worker Deployment Version with the compute provider flags. (2) Invalid connection: Temporal cannot assume the invocation role or reach the Lambda function. Verify the Lambda function ARN and invocation role ARN are correct, and verify the invocation role was created using the CloudFormation template with the External ID matching the value in the Worker Deployment Version configuration. (3) Task Queue not registered: The Lambda invokes successfully but the Worker errors before registering its Task Queue. Check the Lambda function's CloudWatch logs for configuration or runtime errors such as missing environment variables, incorrect TLS configuration, or missing dependencies.

Check Task Queue bindings via CLI for worker deployment

To check Task Queue bindings for a worker deployment, use the CLI command: temporal worker deployment describe-version --namespace <NAMESPACE> --deployment-name <DEPLOYMENT_NAME> --build-id <BUILD_ID> --report-task-queue-stats

Lambda timeout and graceful shutdown with Activities

If the Lambda function reaches its configured timeout before the Worker finishes processing, AWS terminates the invocation. The Worker begins graceful shutdown before the Lambda deadline. If Activities take longer than the available execution window, the Activities are abandoned mid-execution and retried on the next invocation. For long-running Activities, increase both the Lambda timeout and the Worker's shutdown buffer together.

Check Lambda invocation metrics in AWS Console

To check if a Lambda function is being invoked, go to AWS Console > Lambda > Functions > your function > Monitor and look for recent invocations in the Invocations graph. Alternatively, check CloudWatch > Log groups > /aws/lambda/your-function-name for execution logs.

Common Lambda connection errors for Temporal workers

When checking CloudWatch logs for errors during Worker startup on Lambda, common errors include: (1) Connection failures: The Worker cannot reach the Temporal Service. Check that TEMPORAL_ADDRESS and TEMPORAL_API_KEY environment variables or temporal.toml config file are correctly set on the Lambda function. For self-hosted deployments, verify network reachability. (2) TLS errors: The TLS certificate or key is missing, expired, or does not match the Namespace. (3) Authentication errors: The API key is invalid or does not have access to the Namespace.

Serverless Workers failure locations

When a Serverless Worker is not running Tasks, the failure is almost always in one of three places: the Worker Controller Instance (WCI) is not starting compute, the compute provider is rejecting the request, or the Worker starts but cannot do its job.

Serverless Workers invocation workflow

The Serverless Workers invocation process follows these steps: (1) Deploy the Worker to the compute provider, (2) Create a Worker Deployment Version with a compute configuration, which starts a WCI Workflow, (3) A Worker polls the Temporal Service, which binds its Task Queue to the Worker Deployment Version, (4) The WCI monitors that Task Queue and the Matching Service signals the WCI when a Task arrives with no Worker free to take it, (5) Work arrives, the WCI starts compute, and the Worker processes Tasks.

GCP Cloud Run Serverless Workers status

GCP Cloud Run support for Serverless Workers is in Pre-release and its APIs may change in backwards-incompatible ways. The WCI resizes a long-lived Worker Pool on GCP Cloud Run.

AWS Lambda Serverless Workers status

AWS Lambda support for Serverless Workers is in Public Preview. The WCI invokes a function per unit of work on AWS Lambda.

Cloud Run Worker Pool compute configuration fields

The compute configuration for a Cloud Run Worker Pool must specify project, region, and pool name. Temporal addresses the pool as projects/<PROJECT>/locations/<REGION>/workerPools/<POOL_NAME>. A wrong region or wrong pool name will report the pool as not found.

Cloud Run invoker service account IAM permissions

The invoker service account needs roles/iam.serviceAccountTokenCreator on itself so Temporal can impersonate it. The invoker also needs run.workerPools.get to read the pool and run.workerPools.update to scale it. The role roles/run.developer includes both permissions. Validation only exercises the read permission, so an invoker that can read but not update will validate successfully and then fail to scale.

Cloud Run Temporal Cloud service account IAM permissions

On Temporal Cloud, the Terraform module grants the necessary IAM permissions. On a self-hosted Service, the GCP identity the server runs as needs the permissions to impersonate the invoker service account and access the Worker Pool.

Worker Deployment Version must be set as current

The Worker Deployment Version must be set as current before Tasks route to it. If created through the CLI, setting it as current is a separate step from creation. Verify with temporal worker deployment describe.

Check Task Queue binding for Cloud Run Workers

The WCI scales the pool in response to activity on Task Queues bound to the Worker Deployment Version. If no Task Queue is bound, there is nothing for it to watch. Check which Task Queues are bound using: temporal worker deployment describe-version --namespace <NAMESPACE> --deployment-name <DEPLOYMENT_NAME> --build-id <BUILD_ID> --report-task-queue-stats. If no Task Queues are listed, no Worker has successfully polled under this version.

Cloud Run Worker Pool max_count default and scaling

The max_count setting for a Cloud Run Worker Pool defaults to 30 instances. If the pool stops growing while a backlog builds, raise max_count in the Scaling and Lifecycle settings on the Worker Deployment Version. If the count stalls below max_count, check the project's Cloud Run quotas for the region, as Cloud Run caps instances and CPU per region regardless of what count the WCI requests.

Check Cloud Run Worker Pool logs command

Read Worker Pool logs with: gcloud run worker-pools logs read <POOL_NAME> --region <REGION> --project <YOUR_GCP_PROJECT>. The pool produces no logs while scaled to zero, so run this command while an instance is up.

Cloud Run Worker Pool connection failures troubleshooting

Connection failures where the Worker cannot reach the Temporal Service can be debugged by checking the TEMPORAL_ADDRESS and TEMPORAL_NAMESPACE environment variables on the pool. For self-hosted deployments, verify network reachability from Cloud Run to the Temporal frontend.

Cloud Run Worker deployment name and build ID matching

The deployment name and build ID in the Worker code must match exactly with the Worker Deployment Version. Check these values in your Worker code against the output of temporal worker deployment describe. On a mismatch, the Worker polls under a version the WCI does not manage, so its polls never satisfy the Tasks the WCI is scaling for.

Cloud Run Worker Pool missing secrets troubleshooting

If the instance cannot read the Temporal Cloud API key or TLS material from Secret Manager, the runner service account the pool runs as needs roles/secretmanager.secretAccessor on the secret. Note that this is the service account in spec.template.spec.serviceAccountName, not the invoker service account.

Cloud Run Worker Pool scaling flow sequence

The correct Cloud Run Worker scaling flow follows these steps: (1) Deploy the Worker image to a Cloud Run Worker Pool with an instance count of zero. (2) Create a Worker Deployment Version pointing at that pool, which starts a Worker Controller Instance (WCI) Workflow. (3) An instance starts, the Worker polls the Temporal Service, and the server binds the Worker's Task Queue to the Worker Deployment Version. (4) The WCI monitors that Task Queue, and the Matching Service signals the WCI when a Task arrives with no Worker free to take it. (5) As work arrives, the WCI raises the pool's instance count through the Cloud Run admin API, and as work drains, it lowers the count, potentially to zero.

Cloud Run Worker Pool annotation fields for debugging

Three metadata.annotations fields on a Cloud Run Worker Pool indicate scaling status: run.googleapis.com/manualInstanceCount shows the instance count currently requested (0 means no Worker is running); run.googleapis.com/scalingMode should be 'manual' (the WCI scales the pool by writing its manual instance count); serving.knative.dev/lastModifier shows who last changed the pool (if it is the invoker service account, Temporal is reaching the pool; if it is still the deployment account, Temporal has never successfully written to the pool).

Validate Connection for Cloud Run Worker Pools

The 'Validate Connection' action in Workers > Deployments impersonates the invoker service account and reads the Worker Pool. A successful validation confirms three things: the compute configuration names a pool that exists, Temporal can impersonate the invoker, and the invoker can read the pool. No Worker is started as part of validation.

Give your agent this brain