high-availability

31 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Zombie Workflow Execution

A zombie Workflow Execution is an Execution that a region can no longer mutate on its own. During conflict resolution after a failover, the competing Execution in the previously active region becomes a zombie Workflow Execution and is terminated once replication informs it of the competing Workflow ID.

Regional Endpoint for demanding Recovery Time

If an application has an extremely demanding Recovery Time, the DNS propagation delay can be eliminated by connecting through a Regional Endpoint instead of the Namespace Endpoint.

Failover definition and purpose

When a Namespace with High Availability is disrupted by an outage, Temporal Cloud can fail over the Namespace from the primary to the replica. This allows in-flight Workflow Executions to continue, new Workflow Executions to start, and closed Workflow Executions to be inspected, all with minimal interruptions or data loss.

Failback definition

Failback is the process of returning control from the replica to the primary region after a failover. After an automatic failover, Temporal automatically fails back to the original region once it is healthy, unless you opt out.

Automatic failover RTO objective

Temporal aims to detect the outage and complete a failover in minutes from when the outage began, according to the stated Recovery Time Objective (RTO).

Automatic failover conditions and detection

Temporal Cloud runs automated Workflows that continuously monitor the health of Temporal Cloud in every region and every cell. If monitored conditions are failing for too long, Temporal Cloud automatically triggers a failover on any Namespaces with High Availability that have a healthy replica. Temporal's on-call engineers may also trigger a failover at their discretion.

Automatic failover monitoring conditions

Conditions monitored for automatic failover include: whether Temporal Cloud's services in the cell are reachable from the Control Plane; the average latency of inbound RPC calls (excluding long-polling APIs) to Temporal services in the cell; the percentage of inbound RPC calls that returned errors related to server health; the average latency of calls from Temporal Cloud's services in the cell to its persistence layer; and the percentage of calls to the persistence layer that returned errors related to persistence health.

Manual failover use cases

Manual failovers are warranted for: testing failover or migrating to a new region (standard way to exercise failover process with Clients and Workers); outages that affect only the user's systems but not Temporal Cloud itself (requiring user-triggered failover); and failing over more aggressively during a regional outage before Temporal detects it. A manual failover does not conflict with automatic failover; whichever happens first takes effect and the later one is a no-op.

Failover process steps

The failover process: (1) During normal operation, the primary asynchronously replicates data to the replica, keeping them in sync. (2) A failover is triggered. For automatic failovers, the majority of time is spent on outage detection. (3) The Namespace becomes active in the replica's region—Temporal Cloud first attempts a graceful failover (pauses traffic, drains in-flight replication, switches to replica with no data conflicts); if graceful attempt does not complete within 10 seconds, a forced failover immediately activates the replica. During the switch, Workflow operations are briefly paused and SDKs receive a retryable 'Service unavailable' error. (4) The Namespace Endpoint redirects via DNS to the active region, a change that can take a few minutes to propagate. (5) Failback occurs automatically if Temporal triggered the failover and the region is healthy, or the Namespace continues until a user triggers another failover.

Graceful failover vs forced failover timing

Temporal Cloud attempts a graceful failover first: it pauses traffic, drains in-flight replication, and switches to the replica with no data conflicts. If the graceful attempt does not complete within 10 seconds, Temporal Cloud falls back to a forced failover, which immediately activates the replica.

Post-failover event recording

After any failover (triggered by user or Temporal), an event appears in both the Temporal Cloud Web UI (on the Namespace detail page) and in audit logs. The audit log entry uses the operation 'FailoverNamespace'. Temporal Cloud notifies via email whenever a failover occurs.

Split-brain scenario definition

A split-brain scenario occurs when a network partition separates the two regions and both regions cannot communicate with each other. If the replica is promoted to active during a network partition, both regions will be active simultaneously, accepting writes independently. When the network partition resolves and regions can communicate again, Temporal's conflict resolution process reconciles the divergent histories and determines which region remains active.

Replication lag and non-graceful failover impact

Namespaces with replicas rely on asynchronous event replication. Updates made to the primary may not immediately be reflected in the replica due to replication lag, particularly during failovers. In a non-graceful failover, operations that had already replicated remain durable in the replica, while operations that had not yet replicated (in the replication backlog) are reconciled when the region recovers according to the conflict resolution process.

Conflict resolution data loss scenario

Conflict resolution can only recover data from a functioning Temporal Service. If the previously active region never recovers, Workflow API calls that fall within the RPO (under one minute) may be permanently lost. Such a case would require the permanent loss of multiple cloud Availability Zones and has never happened in the history of Temporal Cloud.

Graceful failover recovery point

In a graceful failover, Temporal Cloud drains the replication backlog to zero and pauses traffic before switching regions, so the replica holds every acknowledged operation and the Namespace achieves a recovery point of zero.

Activity execution semantics in High Availability

High Availability Namespaces provide at-least-once semantics for execution of Activities. Completed Activities may be re-dispatched in a newly active Namespace, leading to repeated executions. In contrast, non-replicated Namespaces can be configured to provide at-most-once semantics when a retry policy's maximum attempts is set to 0.

Durability boundary for Workflow operations

A StartWorkflowExecution, SignalWorkflowExecution, SignalWithStartWorkflowExecution, or UpdateWorkflowExecution call that returns success is durably committed in the active region and replicated asynchronously to the replica.

Workflow ID uniqueness preservation after forced failover

The Workflow ID uniqueness guarantee—at most one Open Workflow Execution per Workflow ID—is always enforced within the active Namespace, and conflict resolution preserves it across a failover. This guarantee limits how many Executions are Open at the same time. Because the guarantee constrains only concurrency and not how many Run IDs a Workflow ID accumulates over its lifetime, conflict resolution can reconcile a divergence without ever running the same Workflow ID twice concurrently.

Check automatic failback status in Web UI

To determine whether your Namespace will be automatically failed back, check the list of failovers on your Namespace's detail page in the Temporal Cloud Web UI. If the most recent failover was automatic, the Namespace will fail back when the original region is healthy. If user-triggered, automatic failback is not available.

Trigger failover via Web UI

To trigger a failover using the Web UI: visit the Namespace page on the Temporal Cloud Web UI, navigate to your Namespace details page, select the Trigger a failover option from the menu, and confirm your action.

Manual failovers apply only to Multi-region and Multi-cloud Replication

Manual failovers can only be triggered for Multi-region and Multi-cloud Replication Namespaces. Same-region Replication Namespaces fail over automatically between cells and cannot be manually failed over.

Trigger failover via tcld CLI

To manually trigger a failover using tcld, run: tcld namespace failover --namespace <namespace_id>.<account_id> --region <target_region>. The target_region must be a region where the Namespace has a replica in Activated state. When using API key authentication with --api-key flag, add it directly after tcld and before namespace failover.

Trigger failover via Cloud Ops API HTTP

Send a POST request to https://saas-api.tmprl.cloud/cloud/namespaces/<namespace>/failover-region with request body containing: region (required) - the region code to failover to, must be a region where the Namespace has a replica in Activated state (example: aws-us-east-1); asyncOperationId (optional) - a user-defined ID for tracking the async operation, server assigns one if not set.

Trigger failover via Cloud Ops API gRPC

Use the FailoverNamespaceRegion RPC with FailoverNamespaceRegionRequest message containing: namespace (required, string) - the namespace to failover; region (required, string) - the region ID to failover to, must be a region where the namespace is currently available; async_operation_id (optional, string) - the ID to use for this async operation. Returns FailoverNamespaceRegionResponse containing an async operation for tracking failover status.

Terraform provider does not support triggering failovers

The Temporal Cloud Terraform provider does not support triggering failovers. Failovers must be triggered using the Web UI, tcld CLI, or Cloud Ops API.

Failover async operation retries and on-call intervention

Once the failover async operation returns successfully, the Namespace is failed over. Temporal manages retries for the failover Workflow. In the rare event that an internal error prevents the failover from completing, the Temporal on-call team is automatically paged to intervene and force the failover to completion.

Automatic failback after automatic failover

After an automatic failover, Temporal Cloud automatically fails back to the original region once the region is healthy. No action is required. Monitor Temporal's status page for updates on the original region's health.

Opt out of automatic failback

After an automatic failover has completed, you can disable automatic failovers on the Namespace to prevent automatic failback. When ready to return to the original region, trigger a failover to that region and then re-enable automatic failovers.

Fail forward to new region permanently

After an automatic failover, you can trigger a failover to the region that is already active, treating it as your primary for as long as it's healthy. Automatic failovers remain enabled, so Temporal still protects you if the new region has an outage.

No automatic failback after user-triggered failover

If you triggered a failover yourself during an outage, Temporal will not automatically fail back. You must trigger a failover back to the original region when it is healthy. Automatic failback is only available when the most recent failover was automatic.

Check replication lag before failover

Always check replication lag before initiating a failover. A forced failover when there is significant replication lag has a higher likelihood of rolling back Workflow progress.

Give your agent this brain