design patterns

80 notes in this subject, read out of this brain and free to use. This is page 1 of 2.

Active/Passive HA pattern definition

Active/Passive is a Worker deployment pattern where Workers run in only one region at a time — the 'active' primary region. The secondary region stays empty with no Workers until a failover occurs. On failover, a fresh Worker fleet is brought up in the secondary region from a 'cold start'. This is the simplest operational model but has the highest recovery time of the three patterns.

Active/Hot-Passive worker behavior on failover

In Active/Hot-Passive, on failover the secondary region's standby Workers — already connected and warm — begin processing immediately with no cold start and no DNS wait. Because the standby fleet is already sized for full load, failover needs no scale-up. The Namespace and Workers fail over together automatically: Temporal Cloud promotes the secondary replica to active.

Active/Hot-Passive HA pattern definition

Active/Hot-Passive is a Worker deployment pattern where a full Worker fleet runs in both the primary and secondary regions simultaneously. The primary region's fleet is active and processes all Workflows; the secondary region's fleet stays connected and warm on standby, doing no work. On failover, the standby fleet in the secondary region begins processing immediately with no cold start. This pattern achieves the lowest recovery time tied with Active/Active but requires the cost of a full standby fleet during normal operation.

Active/Active HA pattern definition

Active/Active is a Worker deployment pattern where Worker fleets run in as many regions as desired, with all regions processing Workflows against the Namespace's single active replica through the Namespace Endpoint. All Worker fleets are active simultaneously. If one region goes down, the others keep processing without interruption. This achieves the lowest recovery time without requiring a dedicated standby fleet.

Namespace High Availability behavior with Worker patterns

A Temporal Cloud Namespace with High Availability always has exactly one active region at a time, regardless of which Worker deployment pattern is chosen. The other region holds a passive replica that receives replicated state. However, since Workers do not need to run in the same region as the active Namespace replica, Temporal Cloud Namespaces can support Active/Active as a Worker deployment pattern even though the Namespace itself has only one active region at any time.

Active/Passive recovery time factors

Recovery time in Active/Passive is dominated by: (1) how quickly the outage is detected, since Workflows make no progress until the operator responds, and (2) how long the secondary-region Worker fleet takes to cold start, including container or VM startup, image pulls, and application warm-up. The Active/Hot-Passive pattern removes both factors by keeping Workers already running and warm in the secondary region.

Active/Passive benefits

Active/Passive offers three main benefits: (1) Easy to reason about — only one region is active at a time, so traffic routing and interactions with systems like databases and queues are simpler to understand, and the pattern pairs naturally with other active/passive systems. (2) Simple to operate — during normal operation it resembles a single-region deployment. (3) Lowest overall architecture cost — the Worker fleet size is simply the capacity needed to operate in one region, with no standby Workers during steady state.

Active/Passive tradeoffs

Active/Passive has two main tradeoffs: (1) Highest overall recovery time of the three patterns due to cold starting the Worker fleet after failover. (2) Depends on tested automation to bring up the secondary-region fleet quickly.

Active/Passive worker failover is operator responsibility

In Active/Passive, the Namespace fails over automatically, but bringing up the Workers in the secondary region is the operator's responsibility. The operator must: (1) detect an outage and decide to fail over by defining failover conditions and signals that trigger them, (2) scale up the Workers by bringing up the secondary-region fleet ideally with tested automation and scaling down the primary region's fleet, and (3) optionally enforce single-region task processing using Active/Hot-Passive if needed.

Active/Passive Namespace Endpoint requirement

In Active/Passive, Workers should connect through the Namespace Endpoint, which always connects to the Namespace in its active region and automatically fails over to the new region. The rationale is that if a Temporal Cloud incident requires the Namespace to fail over while the rest of the primary region is healthy, Workers in the primary region can still connect through the Namespace Endpoint and process Workflows. If Workers use the Regional Endpoint for the primary region, they will not reliably connect to the Namespace during a Temporal Cloud incident in the primary region.

Active/Passive cross-region private connectivity requirement

If using private connectivity in Active/Passive, provide the primary region's Workers a network route to the VPC Endpoint in the other region, so they can reach the active replica after a Namespace-only failover. If cross-region routing cannot be provided, use the Active/Hot-Passive pattern instead, where each region's Workers connect to their local replica.

Active/Hot-Passive benefits

Active/Hot-Passive offers three main benefits: (1) Easy to reason about — only one region is active at a time, making traffic routing and system interactions simpler and pairing naturally with other active/passive systems. (2) Lowest recovery time, tied with Active/Active — secondary-region Workers are already connected and warm, so failover involves no cold start, and the standby fleet is already sized for full load so no scale-up is needed. (3) Low latency during normal operation — Workflow and Activity Tasks are processed only in the active region with no cross-region forwarding of Worker polls, though Client requests originating in the secondary region are still forwarded.

Active/Hot-Passive tradeoff: cost

Active/Hot-Passive has one major tradeoff: highest overall architecture cost — a full standby Worker fleet runs in the secondary region at all times, even during steady state.

Active/Hot-Passive Regional/VPC Endpoint and forwarding requirement

In Active/Hot-Passive, each Worker fleet must connect through its region's Regional Endpoint or VPC Endpoint and forwarding must be disabled for Worker polls. Using the Namespace Endpoint by mistake routes the standby Workers to the active region and defeats the pattern. Disabling forwarding applies to Worker polls only — Client requests such as Start Workflow, Signal, Query, Cancel, and Terminate are always forwarded to the active region, so a Workflow starter in the secondary region keeps working.

Active/Active benefits

Active/Active offers three main benefits: (1) Hands-off Worker failover — with the Namespace Endpoint, Workers in every region follow the Namespace to the new active region automatically with no Worker failover step to run. (2) Lowest recovery time, no standby fleet — surviving regions keep processing with no cold start, giving the same low recovery time as Active/Hot-Passive while spreading capacity across regions instead of parking it in a dedicated standby fleet; surviving regions may need to scale up to absorb the failed region's load. (3) Resilient to losing a region — like spreading across Availability Zones, losing one region's fleet leaves the others running.

Active/Active tradeoffs

Active/Active introduces two main tradeoffs: (1) Workers outside the active region reach it across regions (directly or via forwarding), adding latency that can matter for latency-sensitive Workflows. (2) External systems are harder — Workers are active in multiple regions at once, so any databases and queues they touch need a cross-region consistency story.

Active/Active Namespace Endpoint use by default

In Active/Active, all fleets in any region should connect through the single Namespace Endpoint by default. It always routes to the active region and follows failovers automatically, so every fleet keeps reaching the active Namespace with no reconfiguration — it 'just works,' and Workers in all regions fail over automatically. One endpoint everywhere also keeps configuration and management simple.

Active/Active Regional Endpoint use cases

In Active/Active, use a Regional Endpoint only when you need the lowest recovery time. Connecting each fleet to its region's Regional Endpoint or VPC Endpoint removes the DNS step from the connection path, which can shave time off failover. The tradeoffs are: more setup, and a real risk of misconfiguration such as routing a fleet to the wrong region. Reach for Regional Endpoints only when you absolutely need low recovery time. With Regional Endpoints, keep forwarding enabled so passive-region polls still reach the active replica.

Pattern comparison: recovery time (RTO)

Active/Hot-Passive and Active/Active both achieve the lowest recovery time, because neither requires a cold start after failover. In Active/Hot-Passive, a warm standby Worker fleet in the secondary region begins processing the moment it becomes active. In Active/Active, Workers are already processing in every region, so surviving regions keep running with no gap. Active/Passive must cold start a Worker fleet in the secondary region, giving it the highest recovery time.

Pattern comparison: Worker location and cost

Active/Passive has lowest overall architecture cost with Workers running in one region at a time. Active/Hot-Passive has higher cost with a full standby fleet in the secondary region at all times. Active/Active spreads capacity across regions during normal operation.

Workflow starters and Clients HA deployment

Workflow starters and Clients should be deployed with the same regional pattern as the Workers, since they often share the same in-region dependencies (databases, queues, upstream services) and should fail over alongside them. Point Clients at the Namespace Endpoint so they follow the active region automatically with no configuration change on failover, and use a Regional Endpoint only when a Client must be pinned to a region.

Codec Servers and proxies HA deployment

Codec Servers and proxies must be reachable from every region where Workers connect. In Active/Passive, scale them up in the secondary region as part of a failover. In Active/Hot-Passive and Active/Active patterns, run them in both regions at all times.

Databases and queues HA considerations

Databases and queues remain the application's responsibility, and the right approach depends on the Worker deployment pattern. A single-region-active datastore pairs naturally with Active/Passive and Active/Hot-Passive patterns, while running Workers active in both regions (as in Active/Active) raises consistency questions that must be designed for by the application.

Multi-cloud HA/DR patterns

Multi-region and multi-cloud HA/DR use the same Worker deployment patterns (Active/Passive, Active/Hot-Passive, Active/Active). The secondary region can be any Temporal Cloud region that supports replication from the primary region, whether in the same cloud provider or a different one. The same considerations apply: route Workers through the right endpoints and private connectivity, and give any databases and queues a cross-region (or cross-cloud) consistency story.

Active/Passive multi-cloud private networking recommendation

If using Active/Passive with multi-cloud deployment, it is highly recommended to ensure Workers in the active cloud can reach the Namespace if it fails over to the passive cloud due to a Temporal-specific outage, so they can keep processing Tasks across the cross-cloud path.

Active/Passive private networking cross-region path recommendation

If using Active/Passive with private networking, it is highly recommended to provide a network path for Workers in the active region to reach the VPC Endpoint in the passive region, so they can keep processing Tasks if the Namespace fails over due to a Temporal-specific outage.

HA/DR components requiring failover

To keep Workflows running during a cloud outage, these components need to failover to a healthy region: Temporal Cloud Namespace (achieve by enabling High Availability on the Namespace), Workers (the compute resources that execute Workflows and Activities), Workflow starters and Clients (the applications that start and Signal Workflows), Codec Servers (a critical dependency for Workers, the Web UI, and the CLI), Proxies between Workers and Temporal Cloud (any forward proxy or mTLS terminator), and Datastores (databases, queues, and any other systems that Activities read and write).

Retry Alerting best practice: choose threshold above normal transient noise

Choose a threshold above normal transient noise. If the downstream system occasionally has 1–2 retry attempts under normal conditions, set the threshold at 5 or 10 so the metric only fires for genuinely sustained failures.

Retry Alerting pitfall: setting threshold too low

Do not set the threshold too low. A threshold of 1 means the metric fires on the very first retry—which is normal behavior. Calibrate the threshold to the system's expected transient error rate.

Retry Alerting pitfall: not resetting alerting context on success

Do not fail to reset alerting context on success. If the Activity eventually succeeds after 50 attempts, the high-attempt metric has already fired. Ensure the alerting system can resolve the alert when the metric rate drops to zero.

Retry Alerting pitfall: alerting on total counter value instead of rate

Do not alert on the total counter value instead of the rate. If the counter is cumulative, a single high-attempt event in the past will keep the counter elevated forever. Alert on the increment rate (events per minute) rather than the absolute count.

Retry Alerting pitfall: emitting metric in Workflow instead of Activity

Do not emit the metric in the Workflow instead of the Activity. The Workflow does not have access to the Activity's attempt number without passing it explicitly. The Activity context always has the current attempt number—use it there.

Retry Alerting best practice: combine with Fast/Slow Retries

Combine with Fast/Slow Retries pattern. Emit the metric in the slow-phase Activity of a Fast/Slow Retries pattern to alert when the Workflow has been in the slow phase long enough to be a concern.

Retry Alerting best practice: use SDK metrics scope not third-party library

Use the SDK metrics scope, not a third-party library. The SDK scope integrates with the Worker's existing metrics pipeline and adds default tags such as namespace and task queue automatically.

Retry Alerting best practice: use rate-based alerts not count-based

Set up rate-based alerts, not count-based. A count alert requires resetting or remembering the baseline. A rate alert (for example, "more than 3 increments per minute") fires when the problem is active and clears when it resolves.

Retry Alerting via Metrics pattern overview

The Retry Alerting via Metrics pattern emits a custom metric counter from inside the Activity whenever the attempt number exceeds a threshold. This surfaces silent, persistent failures to on-call teams before they breach an SLA, without modifying retry behavior or adding Workflow-level tracking.

Why Retry Alerting via Metrics solves the problem

When an Activity retries indefinitely, failures are invisible at the system level until something breaks. The Temporal UI shows the current attempt number, but on-call teams do not watch the UI continuously. Without a metric or alert, a downstream system can be down for hours while the Workflow keeps retrying silently, and the first sign of a problem is an SLA breach or a user complaint.

Retry Alerting metric emission through SDK metrics scope

Read the current attempt number from the Activity execution context and emit a counter metric when it exceeds a threshold. The metric is sent through the Temporal SDK's built-in metrics scope—the same pipeline used for SDK-internal metrics—so it flows to whatever metrics backend Workers are already configured to use, such as Prometheus or StatsD, without additional setup.

Retry Alerting workflow configuration - metric independent of retry policy

Configure the Activity in the Workflow with the desired retry policy. The metric emission inside the Activity is independent of the retry configuration. The metric can be emitted while using any retry policy, including indefinite retries with no maximum attempts.

Retry Alerting best practice: emit on every attempt above threshold

Emit on every attempt above the threshold, not only once. Incrementing the counter on each high-attempt invocation allows alerting systems to detect both the onset and the duration of a problem by watching the counter rate.

UI display of External Storage payloads

When a payload is offloaded to external storage, the Temporal UI displays a reference token instead of the actual data. Application code receives the fully decoded result because the SDK transparently retrieves the payload from external storage before returning it to the Workflow or Client.

External Storage pattern overview

External Storage offloads payloads to an external store such as Amazon S3 and passes a small reference token through Event History instead. This implements the claim check pattern.

Why External Storage is needed

External Storage addresses scenarios where payloads exceed the per-payload hard limit, where history size grows quickly due to accumulating Activity input/output data causing degraded Workflow Task latency, data processing pipelines with large blobs, AI agent conversations with long histories, spiky data sizes, migration to Temporal Cloud from self-hosted with higher limits, and data governance requirements to store payload data in controlled infrastructure.

External Storage data conversion pipeline position

During Data Conversion, External Storage sits at the end of the pipeline after both the Payload Converter and Payload Codec. When a payload reaches or exceeds the configured size threshold, the storage driver uploads it to external storage and replaces it with a lightweight reference. Payloads below the threshold stay inline in Event History. The process reverses when the Temporal Service dispatches Tasks to Workers.

External Storage parallelization of uploads and downloads

The SDK parallelizes uploads and downloads to minimize latency. When a single Workflow Task involves multiple payloads that reach the threshold, the SDK uploads or downloads all of them concurrently rather than one at a time, allowing external storage operations to scale well even when a Task carries many large payloads.

External Storage encryption with codec

Because External Storage runs after the Payload Codec, if you use an encryption codec, payloads are already encrypted before upload to your store.

Production storage system requirements

A production storage system should store payload data durably and retain it for the full Workflow lifetime plus the Namespace retention period, be reachable from every Client, Worker, and Codec Server that encodes or decodes payloads, support expected payload sizes, return consistent data immediately after a write completes, meet latency and throughput requirements under realistic load, and provide appropriate controls for authentication, encryption, monitoring, and backup.

Recommended storage systems

Start with an object store such as Amazon S3, Google Cloud Storage, or Azure Blob Storage unless there is a specific reason to use a different system. First-party S3 storage drivers are provided for Go, Python, and TypeScript SDKs. First-party Google Cloud Storage drivers are provided for Go and TypeScript SDKs. Google Cloud Bigtable is low-latency for Google Cloud but payloads must fit within Bigtable's cell and row size limits. Redis is suitable when configured for durability with AOF persistence, not as an evicting cache.

Storage driver operations

A storage driver provides two operations: Store (upload payloads and return a claim, which is a set of key-value pairs the driver uses to locate the payload later) and Retrieve (download payloads using the claims that Store produced). The S3 driver also includes diagnostic metadata in error messages such as the AWS region to help with troubleshooting storage failures.

Multiple storage drivers and selector function

Temporal SDKs allow you to configure multiple storage drivers and use a selector function to route payloads to different drivers based on size, type, or other criteria such as hot and cold storage tiers.

External Storage key configuration settings

Configure External Storage on the Data Converter with these key settings: Size threshold (the driver offloads payloads whose serialized size reaches or exceeds this value, defaults to 256 KiB), Drivers (one or more storage driver implementations), and Driver selector (when using multiple drivers, a function that chooses which driver handles each payload).

Data governance use case with threshold 0 or 1

For data governance where organizations prefer to store payload data in infrastructure they control, lower the offload size threshold to externalize all payloads regardless of size. In the Python and TypeScript SDKs, set the threshold to 0. In the Go SDK, set it to 1, because 0 selects the 256 KiB default.

External Storage TTL and lifecycle management

Temporal does not automatically delete payloads from external store. Payloads can be orphaned if a request fails after upload completes. Configure a lifecycle policy to ensure payloads are eventually cleaned up and provide a grace period for debugging and recovery. The TTL must be long enough: TTL > Maximum Workflow Run Timeout + Namespace Retention Period. For example, if the longest-running Workflow has a Run Timeout of 14 days and Namespace retention period is 30 days, configure the lifecycle rule to expire objects after at least 44 days.

Indefinite Workflow TTL and Continue-as-New

If Workflows run indefinitely with no Run Timeout, there is no finite TTL that guarantees safety. Set a generous TTL based on operational needs. Use Continue-as-New for Workflows that need to run longer. The new run uploads fresh payloads, and the old run's payloads only need to survive through its retention period.

Durable External Storage redundancy

External Storage stores payloads in a single storage backend. If that backend becomes unavailable, Workers cannot retrieve payloads. To protect against regional or provider failures, implement drivers in a way that takes advantage of storage provider's redundancy features such as cross-region replication and multi-region routing.

Storage failure retry behavior within Task attempt

The SDK does not retry a failed Store or Retrieve call within the same Task attempt. If a storage driver returns an error, that failure fails the current Workflow Task or Activity Task attempt. Temporal retries the Task as a whole, and the new attempt retries the storage operation along with the rest of the Task. For Activities, the Retry Policy controls the timing of that retry.

S3 Multi-Region Access Points with Cross-Region Replication

The built-in S3 storage driver supports S3 Multi-Region Access Points (MRAP) as the bucket endpoint. Combined with Cross-Region Replication (CRR), this gives automatic failover across regions: if a bucket or region becomes unavailable, requests route to the closest healthy bucket. Configure the External Storage S3 driver to use the MRAP ARN in place of the bucket name.

Cross-region replication eventual consistency tradeoff

Cross-region replication introduces eventual consistency. After a write, a read in another region may temporarily miss the object. To mitigate: ensure Activities that read from External Storage have appropriate retry policies to recover from transient unavailability caused by replication lag, and if an Activity needs to read a payload immediately after it is written, prefer scheduling it on the same Worker or in the same region to avoid the lag window. By default, S3 CRR has no SLA on replication time. Enable Replication Time Control if stronger guarantees are needed.

Replication and versioning cost considerations

Replication and versioning can significantly increase storage costs. Check your provider's pricing before enabling.

When not to use Parallel Execution pattern

Parallel Execution is not a good fit for operations with dependencies between them, resource-constrained environments (use controlled parallelism instead), operations requiring strict ordering, or a single fast operation where the overhead is not worth it.

Give your agent this brain