best-practices

124 notes in this subject, read out of this brain and free to use. This is page 1 of 3.

Problems solved by Temporal best practices framework

Without clearly defined Temporal standards, organizations often struggle with inconsistent Workflow implementations, fragmented best practices, and misaligned development approaches. The best practices framework helps developers establish robust Temporal standards by providing proven foundation principles, standardized implementation patterns, and confidence in alignment with architectural principles.

Target audience for Temporal best practices guide

The best practices documentation is intended for developers responsible for building a Temporal Cloud practice within their organization, anyone building tutorials, courses, onboarding paths, or documentation, and partners or vendors creating Temporal-related learning materials.

Temporal best practices guides available

The best practices documentation includes guides on: Managing a Namespace for configuring and optimizing Temporal Namespaces; Managing Temporal Cloud Access Control for implementing proper access control and user management; Security Controls for Temporal Cloud for protecting deployments; Worker Deployment and Performance for deploying and optimizing Temporal Workers; Cost Optimization for optimizing workload costs while maintaining reliability; Pre-Production Testing for failure injection, load testing, and operational validation; and Knowledge Hub for building internal Temporal knowledge bases.

Temporal best practices documentation scope and purpose

The Temporal best practices guide provides a validated, opinionated framework for using Temporal Cloud. It offers proven foundation principles validated across diverse use cases, standardized implementation patterns for consistent adoption across projects, and confidence in alignment with Temporal's architectural principles and recommended practices.

Ship section content

The Ship section should include: Architecture and standards documenting Namespace conventions, connectivity requirements, and Worker deployment standards; Cost guidance explaining billable Actions, storage tiers, and cost-saving tips; Shared responsibility defining an ownership matrix between Platform and Application teams across IAM, infrastructure, development, deployment, observability, and operations; Design patterns curating Workflow patterns with descriptions and code sample links.

Operate section content

The Operate section should include: Troubleshooting and escalation covering observability tools, runbooks for common issues, escalation paths, SLAs, and example alert definitions; Support and FAQs documenting support tier, ticket submission process, Temporal account contacts, expert-led session types, and frequently asked questions.

Knowledge hub problems it solves

An internal Temporal knowledge hub prevents tribal knowledge from getting siloed within teams, which leads to inconsistent patterns for Workflow design, error handling, and testing. It reduces fragmentation that increases support burden and slows developer onboarding, while helping prevent inadvertently introduced security vulnerabilities or compliance gaps.

Knowledge hub success metrics

Track four key metrics to measure knowledge hub effectiveness: Time to first Workflow (target under 30 minutes, from days-to-weeks baseline); Time to Workflow in production (target under 2 weeks, from weeks-to-months baseline); Support question rate (target fewer than 5 per week, from 20-30+ baseline); Knowledge Hub traffic (track steady or growing page views per month as no baseline exists initially). Declining traffic on a page may indicate it is outdated; high traffic with high bounce rates may indicate the page is not answering the question.

Knowledge hub maintenance practices

Maintain knowledge hub currency by: reviewing each page at least quarterly with an assigned review owner and date to make staleness visible; updating whenever the organization changes its Temporal architecture, updates deployment tooling, or modifies shared responsibility model; pruning aggressively by removing or archiving content that no longer applies because outdated documentation is worse than no documentation since developers follow it and get unexpected results. Capture every question reaching the Platform team as a candidate for new content, and solicit contributions from application teams through lightweight processes like pull request templates.

What not to include in knowledge hub

A knowledge hub is not a mirror of Temporal's official documentation. Avoid duplicating SDK API references, concept explanations, or release notes that Temporal already maintains. When that content changes, a copied version becomes a source of confusion. Instead, link to the official docs and reserve the knowledge hub for organization-specific decisions, conventions, and operational procedures that Temporal's public documentation does not cover.

Evaluate section content

The Evaluate section should include: Temporal overview explaining what Temporal is, why the organization chose it, and business value metrics justifying adoption; and Decision framework providing qualifying questions, good and bad use cases, and alternative recommendations to help developers determine fit for their problem.

Build section content

The Build section should include: Getting started walks developers through a 30-minute quickstart covering environment setup, a starter template, and running a first Workflow locally and on Temporal Cloud; Learning paths provides self-paced courses from foundational to advanced topics tailored by persona with links to Temporal's free training.

Knowledge hub ownership and discoverability

Designate a Platform team or developer experience team as the owner responsible for initial content creation, ongoing maintenance, and reviewing contributions. Make the hub discoverable by registering a short URL (e.g., go/temporal) that redirects to it, pinning the link in Temporal-related communication channels (Slack, Microsoft Teams), and when answering questions, respond with a link to the relevant knowledge hub page instead of re-explaining inline to build the habit of checking the hub first.

Knowledge hub sections by developer journey

A Temporal knowledge hub should be organized into four main sections aligned with where developers are in their journey: Evaluate (understanding Temporal fit), Build (getting started from zero to running Workflow), Ship (architecture standards and production deployment), and Operate (self-service tools and incident response).

Sticky cache size metric and memory pressure

Workers keep a cache of Workflow Executions to improve performance by reducing replay overhead. However, larger caches consume more memory. The temporal_sticky_cache_size metric tracks the size of the cache. If you observe high memory usage for your Workers and high temporal_sticky_cache_size, you can be reasonably sure the cache is contributing to memory pressure. If your Workers are memory-bound and experiencing high temporal_sticky_cache_size, consider reducing the cache size to allow more concurrent executions.

Key Worker metrics to monitor

Monitor these Worker-related metrics on your Worker monitoring dashboard: (1) Worker CPU and memory utilization. (2) workflow_task_schedule_to_start_latency and activity_task_schedule_to_start_latency. (3) worker_task_slots_available. (4) temporal_long_request_failure, temporal_request_failure, temporal_long_request_latency, and temporal_request_latency. When you observe anomalies, correlate across multiple metrics to identify root causes.

Schedule-to-Start latency interpretation

Schedule-to-Start latency measures how long a Task waits in the queue before a Worker starts it. High latency indicates one of three conditions depending on resource metrics: (1) High Schedule-to-Start latency and high CPU/memory means Workers are saturated; scale up your Workers or add more Workers, or Workers may be blocked on Activities. (2) High Schedule-to-Start latency and low CPU/memory means Workers are underutilized; increase the number of pollers, executor slots, or both; if accompanied by high temporal_long_request_latency or temporal_long_request_failure, Workers are struggling to reach the Temporal Service. (3) Low Schedule-to-Start latency and low CPU/memory could be normal; if consistently seeing low memory and CPU usage, you may be over-provisioning Workers and can consider scaling down.

Claim Check pattern for reducing Event History size

A common way to reduce payload and Event History size is the Claim Check pattern, widely used with messaging systems such as Apache Kafka. Instead of passing large data into your function, store that data external to Temporal in a database or file system. Pass an identifier for the data, such as a primary key or path, into the function and use an Activity to retrieve it as needed. If your Activity produces large output, use a similar approach: write the data to an external system and return an identifier that can be used to retrieve it later.

Event History size limits and replay performance

If a Worker goes offline and another Worker picks up the same Workflow Execution, the new Worker must replay the existing Event History to resume the Workflow Execution. If the Event History is too large or has too many Events, replay affects the performance of the new Worker and may even cause timeout errors well before the hard limit of 51,200 Events is reached. It is recommended not to exceed a few thousand Events in a single Workflow Execution.

Event History payload size limits

Input parameters and output values of both Workflows and Activities are stored in the Event History. A Workflow Execution may be terminated if any single payload exceeds 2 MB or if the entire Event History exceeds 50 MB. Avoid passing large amounts of data into and out of Workflows and Activities to avoid hitting these limits.

Patching as fallback alternative to Worker Versioning

If your deployment environment cannot yet support Worker Versioning, you can use Patching to introduce changes to your Workflow code without breaking running Executions. Treat patching as a fallback for environments that cannot adopt versioned worker deployments yet, not as the default recommendation.

Use Continue-As-New to manage Event History growth

The best way to handle Event History growth is to use the Continue-As-New mechanism to continue under a new Workflow Execution with a new Event History, repeating this process as you approach the limits again. All Temporal SDKs provide functions to suggest when to use Continue-As-New. For example, the Python SDK has the is_continue_as_new_suggested() function that returns a bool indicating whether to use Continue-As-New.

Worker Versioning for safe Workflow code deployment

Use Worker Versioning to deploy new Workflow code without breaking running Executions. Worker Versioning lets you map each Workflow Execution to a specific Worker Deployment Version identified by a build ID, which guarantees that pinned Workflows always run on the same Worker version where they started. For most teams, Worker Versioning should be the default approach for evolving Workflow code in production, providing safer rollouts, clearer rollback behavior, and a cleaner operational model than maintaining multiple code paths inside a Workflow definition.

Task Queue name mismatch consequences

Since Task Queues are created dynamically when first used, a mismatch between the Client and Worker Task Queue names does not result in an error. Instead, it creates two different Task Queues, and the Worker never receives Tasks from the Temporal Service because it is polling the wrong queue. Define the Task Queue name as a constant that both the Client and Worker reference to avoid this issue.

Separate Task Queues for distinct workloads

Use separate Task Queues for distinct workloads. This isolation allows you to control rate limiting, prioritize certain workloads, and prevent one workload from starving another. For each Task Queue, ensure you configure at least two Workers to poll the Task Queue. Each microservice or workload type should have its own Task Queue, allowing each service to scale independently based on its workload.

Default Worker settings designed for development not production

Default Worker settings are designed to work across a wide range of use cases primarily for ease in development and testing. In production environments, your Workflow complexity, Activity duration, payload sizes, and infrastructure constraints all influence optimal Worker configuration. Actively tuning your Workers ensures they perform well under your specific workload conditions.

Package Workers as artifacts from CI/CD pipeline

Workers should be artifacts produced by a CI/CD pipeline. Inject all required parameters for connecting to Temporal Cloud or a self-hosted Temporal Service at runtime via environment variables, configuration files, or command-line parameters. This allows for more granularity, easier testability, easier upgrades, scalability, and isolation of Workers.

Worker deployment quick checklist

Before deploying Workers to production, address these key items: (1) Configure each Worker appropriately by actively tuning Worker options based on code, language runtime limits, and system resource constraints rather than relying on defaults designed for development. (2) Deploy enough Workers and monitor performance metrics to scale Workers to meet workload requirements. (3) Separate Task Queues logically by sizing and splitting work across Activity and Workflow Task types and Task Queues based on workload characteristics. (4) Use Worker Versioning to safely deploy new Workflow code without breaking running Executions. (5) Run benchmarks under realistic load to confirm limits and settings are appropriate for your environment.

Multi-stage Docker build pattern for Worker images

Use a multi-stage Dockerfile to create minimal production-ready Worker images. The first stage (builder) compiles the Worker binary using BuildKit cache mounts for go/pkg/mod and go-build caches, then builds a statically linked binary with CGO_ENABLED=0. The second stage (final image) uses a minimal base image like busybox and copies only the compiled binary from the builder stage, then sets the entrypoint to run the Worker process.

Graceful Worker shutdown to avoid task interruption

Before shutting down a Worker, verify that it does not have too many active Tasks. This is especially relevant if your Workers are handling long-running, expensive Activities. If worker_task_slots_available is at or near zero, the Worker is running active Tasks. Shutting it down could trigger expensive retries or timeouts for long-running Activities. Use Graceful Shutdowns to allow the Worker to complete its current Tasks before shutting down. All SDKs provide a way to configure Graceful Shutdowns. For example, the Go SDK has the WorkerStopTimeout option that lets you configure how long the Worker has to complete its current Tasks before shutting down.

Key Worker options to tune for production

Focus on tuning these key Worker options in production: (1) Task slots - limits how many Tasks execute concurrently, set based on Worker CPU, memory, and resource demands of code; can use different Slot Suppliers or implement a custom Slot Supplier to control how Task slots are assigned to different Task types. (2) Sticky cache size - controls the size of the sticky cache for Workflow Executions; larger caches reduce replay overhead but consume more memory. (3) Poller counts - controls the number of pollers for Tasks; use the Poller Autoscaling feature to automatically adjust the number of pollers based on workload.

Execution failure alert - unregistered activity invocation (Go SDK only)

Alert on metric 'unregistered_activity_invocation' with condition by 'activity_type', 'task_queue', and 'workflow_type'. Threshold: any occurrence. Duration: 1 minute. Severity: Critical. Group by namespace, activity_type, task_queue, and workflow_type. This alert is specific to the Go SDK.

Alert dependency chains and root cause analysis

Alerts are chained together: task slots fill up, which drops pollers to zero, which drives schedule-to-start latency up, which drives task completions to zero. Expect them to fire in clusters. When several fire together, use triage pages to identify which is the root cause and which are symptoms rather than responding to all four.

Execution failure alert - local activity latency exceeds heartbeat timeout

Alert on metric 'local_activity_execution_latency' with condition by 'activity_type'. Threshold: p99 above 30 minutes. Duration: 5 minutes. Severity: Critical. Group by namespace and activity_type.

Worker capacity alert - all pollers disconnected

Alert on metric 'num_pollers' with condition 'poller_type' in ['workflow_task', 'workflow_sticky_task', 'activity_task']. Threshold: reaches 0. Duration: 5 minutes. Severity: Critical. Group by namespace and poller_type.

Execution failure alert - workflow task execution latency high

Alert on metric 'workflow_task_execution_latency' with condition by 'task_queue' and 'workflow_type'. Threshold: p99 above 10 seconds. Duration: 5 minutes. Severity: Critical. Group by namespace, task_queue, and workflow_type.

Execution failure alert - gRPC message too large

Alert on metric 'workflow_task_execution_failed' with condition 'failure_reason=GrpcMessageTooLarge'. Threshold: any occurrence. Duration: 1 minute. Severity: Critical. Group by namespace and failure_reason.

Execution failure alert - non-determinism error

Alert on metric 'workflow_task_execution_failed' with condition 'failure_reason=NonDeterminismError'. Threshold: any occurrence. Duration: 1 minute. Severity: Critical. Group by namespace and failure_reason.

Worker capacity alert - sticky cache holding zero entries under load

Alert on metric 'sticky_cache_size' paired with a non-zero workflow task rate on the same worker. Threshold: reaches 0. Duration: 15 minutes. Severity: Warning. Group by namespace. Pair this alert with a demand signal to avoid false positives on idle task queues.

Worker capacity alert - activity schedule-to-start latency severe

Alert on metric 'activity_schedule_to_start_latency' with condition by 'task_queue'. Threshold: p99 above 30 minutes. Duration: 5 minutes. Severity: Critical. Group by namespace and task_queue.

Worker capacity alert - task completions dropped to zero

Alert on metric 'request' with condition 'operation' in ['RespondWorkflowTaskCompleted', 'RespondActivityTaskCompleted'] and the task queue has demand. Threshold: rate reaches 0 while task queue has demand. Duration: 5 minutes. Severity: Critical. Group by namespace and operation. Pair this alert with a demand signal to avoid false positives on idle task queues.

Worker metrics not visible to Temporal Service

Workers emit metrics that the Temporal Service has no view into. These include workflow code failing on replay, workers that have stopped polling, task slots that never free up, and local activities that run past the workflow task heartbeat window. Workers themselves are responsible for emitting these metrics.

Worker capacity alert - worker task slots exhausted

Alert on metric 'worker_task_slots_available' with condition 'worker_type' in ['WorkflowWorker', 'ActivityWorker', 'LocalActivityWorker']. Threshold: reaches 0. Duration: 2 minutes. Severity: Critical. Group by namespace and worker_type. Note: This metric reports meaningful values only with fixed-size slot suppliers and cannot be used with resource-based slot suppliers.

Request failure alerts - request latency high on user-facing operations

Alert on metric 'request_latency' with condition 'operation' in ['StartWorkflowExecution', 'SignalWithStartWorkflowExecution', 'SignalWorkflowExecution', 'UpdateWorkflowExecution', 'ExecuteMultiOperation']. Threshold: p99 above 2 seconds. Duration: 5 minutes. Severity: Critical. Group by namespace and operation.

Request failure alerts - INTERNAL from Temporal Service

Alert on metric 'request_failure' with condition 'status_code=INTERNAL' for any operation. Threshold: any occurrence. Duration: 2 minutes. Severity: Critical. Group by namespace and operation.

Request failure alerts - UNIMPLEMENTED from Temporal Service

Alert on metric 'request_failure' with condition 'status_code=UNIMPLEMENTED' for any operation. Threshold: any occurrence. Duration: 2 minutes. Severity: Critical. Group by namespace and operation.

Request failure alerts - RESOURCE_EXHAUSTED on poll operations

Alert on metric 'long_request_failure' with condition 'status_code=RESOURCE_EXHAUSTED' and 'operation' in ['PollWorkflowTaskQueue', 'PollActivityTaskQueue']. Threshold: any occurrence. Duration: 5 minutes. Severity: Warning. Group by namespace and operation.

Request failure alerts - RESOURCE_EXHAUSTED on respond operations

Alert on metric 'request_failure' with condition 'status_code=RESOURCE_EXHAUSTED' and 'operation' in ['RespondWorkflowTaskCompleted', 'RespondWorkflowTaskFailed', 'RespondActivityTaskCompleted', 'RespondActivityTaskFailed']. Threshold: any occurrence. Duration: 5 minutes. Severity: Critical. Group by namespace and operation.

Request failure alerts - NOT_FOUND on activity heartbeat

Alert on metric 'request_failure' with condition 'status_code=NOT_FOUND' and 'operation=RecordActivityTaskHeartbeat'. Threshold: any occurrence. Duration: 5 minutes. Severity: Warning. Group by namespace and operation.

Request failure alerts - NOT_FOUND on respond operations

Alert on metric 'request_failure' with condition 'status_code=NOT_FOUND' and 'operation' in ['RespondWorkflowTaskCompleted', 'RespondWorkflowTaskFailed', 'RespondActivityTaskCompleted', 'RespondActivityTaskFailed']. Threshold: any occurrence. Duration: 5 minutes. Severity: Critical. Group by namespace and operation.

Idle task queue and autoscaling edge cases

An idle task queue legitimately reports zero task completions and an empty sticky cache, so pair both alerts with a demand signal. Autoscaling to zero legitimately reports no pollers and no free slots. For these cases, either exclude scaled-down workers from those alerts or scope them to task queues you keep warm.

Alert duration tuning during deployments

Several alert conditions appear briefly during deployments. A rolling restart drops pollers to zero on each pod as it cycles, produces NOT_FOUND on respond operations for in-flight tasks, and can produce short-lived non-determinism errors while two worker versions overlap. A Temporal Service upgrade produces the same. Set the 'for' duration longer than a deploy takes to prevent false positives. These conditions clear on their own once the rollout finishes.

Alert threshold tuning principles

Every threshold is a starting point, not a service level objective. Pick thresholds from your own observed p99 during a healthy period, not from suggested values. High-throughput task queues need different values than bursty batch workloads. Two knobs per alert are the threshold (what counts as unhealthy) and the 'for' duration (how long the condition must hold before firing).

Execution failure alert - activity execution failures elevated

Alert on metric 'activity_execution_failed' with condition by 'activity_type'. Threshold: rate above 10 per second. Duration: 2 minutes. Severity: Warning. Group by namespace and activity_type.

Five priority alerts for worker failures

Start with these five alerts as they catch the failure modes that stop workflow executions outright and are the least likely to produce false positives: (1) All pollers disconnected - workers have stopped polling, (2) Non-determinism error - workflow code no longer matches recorded history, (3) gRPC message too large - executions are being terminated and losing work, (4) Workflow task schedule-to-start latency elevated - tasks are backing up, (5) RESOURCE_EXHAUSTED on user-facing operations - starts, signals, and updates are being throttled.

Tag coverage variance across SDKs

Tag coverage varies across SDKs. The 'sticky_cache_size' metric carries 'namespace' only in TypeScript and Java SDKs, and 'task_queue' only in TypeScript. It is necessary to confirm exact metric names and tags in your own metrics endpoint before writing queries.

Metric name prefixes and suffixes by type

Every metric carries a 'temporal_' prefix. Counters pick up a '_total' suffix when scraped through Prometheus (e.g., 'request_failure' becomes 'temporal_request_failure_total'). Histograms pick up '_seconds_bucket' on the bucket series (e.g., 'temporal_workflow_task_schedule_to_start_latency_seconds_bucket'). Gauges take no suffix at all (e.g., 'temporal_num_pollers', 'temporal_worker_task_slots_available', 'temporal_sticky_cache_size'). Adding '_total' to a gauge returns nothing with no error message.

Alert severity adjustment for workload characteristics

Some critical-severity alerts are early warnings, not confirmed stoppage. High workflow task execution latency matters significantly on latency-sensitive namespaces but barely matters on batch workloads where tasks routinely run long. Downgrade latency alerts if your namespace is batch-oriented. Some warning-severity alerts can still lose data - for example, NOT_FOUND on activity heartbeat means the attempt already timed out and will run again. If that activity is not idempotent, treat it as Critical.

Alert severity routing guidance

Page on Critical alerts - these mean workflow executions have stopped or are still running but losing work or duplicating side effects. Send Warning alerts to a channel someone reads during the day - these mean something is degrading but executions are still moving. Adjust defaults based on your deployment characteristics.

Supported Temporal SDKs

Temporal offers SDKs for .NET, Go, Java, PHP, Python, Ruby, and TypeScript. These SDKs enable building robust Temporal applications.

Give your agent this brain