monitoring

10 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Detect failover by monitoring active region change

The clearest way to detect that a failover has happened is to watch whether your Namespace's active region changed. When Temporal Cloud promotes the replica in the secondary region to active, the active region reported for the Namespace changes, providing a reliable and unambiguous signal that a failover occurred.

Track failovers using FailoverNamespace audit log operation

To track failovers as they happen, look for the FailoverNamespace operation in the audit logs. When Temporal triggers failovers, the audit log updates with details including operation set to FailoverNamespace.

Signals to detect regional outage before failover

To detect a regional outage directly, before or independently of a Namespace failover, watch for: a spike in replication lag between primary and replica, a drop in Workflow throughput such as sudden decline in rate of Workflows started or completed, a spike in errors across overall stack including application errors and failed Activities, a drop in throughput across overall stack such as fewer requests reaching services, and errors or failovers in other cloud systems such as databases and queues.

P95 replication lag target in Temporal Cloud

Temporal Cloud strives to maintain a P95 replication lag of less than 1 minute, where P95 means 95% of updates are processed faster than this limit.

Replication lag risk during forced failover

A forced failover when there is significant replication lag increases the likelihood of rolling back Workflow progress. Always check the replication lag metrics before initiating a failover.

Replication lag metrics emitted by Temporal Cloud

Temporal Cloud emits replication lag metrics as pre-computed percentiles (p50, p95, p99) that are labeled with temporal_namespace.

Action count doubles with replica due to replication

When a Namespace is using a replica, the Action count in temporal_cloud_v1_total_action_count may be 2x what it was before adding a replica. This happens because Actions are replicated and occur on both the primary and the replica.

Unhealthy replica conditions prevent failover

If the replica is unhealthy, Temporal Cloud disables the trigger failover option to prevent failing over to an unhealthy replica. Unhealthy replica conditions include: data synchronization issues where the replica fails to remain in sync with the primary due to network or performance problems, replication lag where the replica falls behind the primary causing it to be out of sync, network issues with loss of communication between replica and primary, and failed health checks.

Asynchronous replication between primary and replica

Temporal Cloud's High Availability features use asynchronous replication between the primary and the replica. Workflow updates in the primary, along with associated History Events, are transmitted to the replica. Replication lag refers to the transmission delay of Workflow updates and history events from the primary to the replica.

Active/Passive requires manual failover detection

With some Worker deployment patterns, most notably Active/Passive, detecting an outage is the responsibility of the operator. Workflows make no progress until an outage is detected and Workers are brought up in the new active region. Fast, reliable detection therefore directly determines recovery time.

Give your agent this brain