design-patterns

826 notes in this subject, read out of this brain and free to use. This is page 5 of 14.

Local Activity exceeding Workflow Task timeout pitfall

If a Local Activity takes longer than the Workflow Task timeout (default 10 seconds), the entire task times out and retries—including any Local Activities that already completed in memory during that task.

Local Activity server round-trips savings

A Local Activity bypasses all server scheduling overhead. The Workflow Task scheduler on the Worker invokes the Activity function directly. The result is folded into the WorkflowTask completion event sent at the end of the task. Each Activity converted saves approximately 50 ms of scheduling overhead on Temporal Cloud.

Local Activity exactly-once semantics pitfall

Local Activities do not have exactly-once semantics. A Local Activity does not get its own persisted history event until the Workflow Task completes. A crashed Worker causes the whole task to re-run. When Local Activities are chained, if a Worker crashes after the third of five sequential Local Activities, all five re-execute on the next attempt. If you need a durable checkpoint between each step, use regular Activities instead.

Local Activities default measurement recommendation

If you haven't measured a latency problem, start with regular Activities—they are easier to debug, rate-limit, and monitor. Replace with Local Activities only after identifying a latency bottleneck.

Local Activities eliminate server scheduling overhead

For a Workflow with three serial Activities, switching all three to Local Activities can save 150 ms or more while keeping the exact same business logic.

Local Activity long retry intervals pitfall

Long retry intervals with Local Activities create server-side timer events for each retry attempt with back-off. For truly short Activities, use a tight `scheduleToCloseTimeout` and allow immediate retries rather than spaced-out back-off.

Local Activity signal and update handler blocking

Avoid blocking signal and update handlers. While a Local Activity executes, the Workflow Task is occupied. Incoming signals and updates accumulate in the server buffer and are not processed until the next task begins.

Local Activity retry policy configuration

Set retry policy carefully for Local Activities. Large `initialInterval` or `maximumInterval` values in a retry policy still cause the SDK to schedule server-side timer events, which partially defeats the latency benefit.

Local Activity vs Regular Activity: benefits and trade-offs comparison

Regular Activities have 2-4 server round-trips per call, ~50 ms latency overhead on Temporal Cloud, support heartbeats, use StartToCloseTimeout, have independent retry semantics per attempt, use a dedicated worker pool, support rate limiting/routing via task queue, and are visible in Temporal UI as full Activity history events. Local Activities have 0 server round-trips, near-zero latency overhead, no heartbeat support, use ScheduleToCloseTimeout, re-execute entire Workflow Task on failure, share the Workflow Task thread, bypass task queue without rate limiting, and are recorded in Workflow Task events without standalone UI Activity tasks.

Local Activity duration recommendation

Keep each Local Activity short. Aim for well under 5 seconds to leave headroom for retries within the same Workflow Task, which has a 10-second default timeout.

Local Activity at-least-once execution design

Design Local Activities for at-least-once execution. If a Workflow Task fails after a Local Activity completes but before the task is persisted, all Local Activities in that task re-execute on the next attempt. Activity logic must tolerate this.

When not to use Local Activities: poor fit

Local Activities are a poor fit for: Activities that may run longer than the Workflow Task timeout (default 10 seconds), Activities that require heartbeating to detect stuck executions, operations with long retry back-off intervals, non-idempotent operations where re-execution on Worker crash would cause harm, and operations that need rate limiting or routing through task queue capacity controls.

Activity Heartbeat pattern overview and purpose

The Activity Heartbeat pattern enables long-running Activities to report progress, handle cancellation gracefully, and resume from the last checkpoint after failures. Heartbeats inform Temporal that the Activity is still alive and allow storing progress details that survive Worker restarts.

Pitfall: Not resuming from heartbeat progress on retry

When an Activity retries, retrieve the last heartbeat details and resume from the last checkpoint instead of restarting from scratch. Failure to do this negates the benefits of checkpointing.

Pitfall: Missing HeartbeatTimeout configuration

Without a HeartbeatTimeout, Temporal cannot detect a stuck or crashed Worker until the StartToCloseTimeout expires, which can be very long. Always set HeartbeatTimeout shorter than StartToCloseTimeout.

Best practice: Test Activity resumption

Verify Activities resume correctly after simulated failures to ensure checkpoint and resumption logic works as expected.

Best practice: Log heartbeat progress

Log heartbeat details for debugging and monitoring to track Activity execution and identify where resumption occurs.

Best practice: Check for cancellation regularly

Heartbeat regularly to detect cancellation quickly. Cancellation is only delivered on the next heartbeat, so infrequent heartbeats delay cancellation detection.

Best practice: Handle idempotency in Activities

Ensure reprocessing the last checkpoint is safe. Activities must be idempotent because the last heartbeated item may be reprocessed on retry.

Pitfall: Catching the wrong exception for cancellation

Cancellation handling is SDK-specific. Inside the Activity, the heartbeat() call throws ActivityCompletionException (Java), cancellation surfaces as CancelledFailure (TypeScript) or asyncio.CancelledError (Python), and the context reports ctx.Err() returning context.Canceled (Go). The CanceledFailure type is what the Workflow observes, not what the Activity body catches.

How Activity Heartbeats work

Activity heartbeats periodically report progress to the Temporal Service. The heartbeat details are persisted and available to retry attempts, enabling resumption from the last checkpoint. Heartbeat timeouts detect stuck Activities faster than execution timeouts.

Pitfall: Heartbeating too infrequently

Cancellation is only delivered on the next heartbeat. If the Activity heartbeats every 5 minutes, cancellation takes up to 5 minutes to propagate.

Best practice: Checkpoint strategically

Save progress at meaningful boundaries such as records, pages, or chunks rather than arbitrary points in processing.

Best practice: Keep heartbeat details small

Store minimal state in heartbeat details (IDs, offsets, counts), not full objects. Heartbeat details have size limits.

Trade-offs of Activity Heartbeats

Trade-offs to consider are that frequent heartbeats increase network traffic. You must implement checkpointing logic and state management. You must handle partial reprocessing of the last checkpoint (idempotency). You need to balance heartbeat frequency between responsiveness and overhead. Heartbeat details have size limits, so you should avoid large objects.

Best practice: Heartbeat at regular intervals

Heartbeat at regular intervals, balancing between responsiveness (every 10-30 seconds) and overhead. Do not heartbeat on every iteration of tight loops.

Best practice: Set heartbeat timeout

Configure heartbeat timeout to 2-3x the expected heartbeat interval.

Benefits of Activity Heartbeats

Heartbeats enable fault tolerance by resuming from the last checkpoint after failures. Heartbeat timeouts detect stuck Activities faster than execution timeouts. They provide visibility into Activity progress in real-time. Activities can handle cancellation gracefully and clean up resources. Completed work is not reprocessed, and Activities can move between Workers.

When NOT to use Activity Heartbeats

Heartbeat pattern is not a good fit for quick operations (under 10 seconds), operations that cannot be checkpointed, Activities requiring exact-once semantics without idempotency, or real-time streaming (use Workflows instead).

When to use Activity Heartbeats

Heartbeat pattern is a good fit for batch processing of large datasets, file uploads and downloads with progress tracking, database migrations or bulk operations, long-running computations (ML training, video encoding), external API polling with multiple attempts, and any Activity running longer than 30 seconds.

Activity Heartbeat with complex progress state

For advanced use cases, heartbeat details can store structured progress state including multiple fields such as processed_count, failed_count, and last_processed_id. On retry, the Activity reconstructs the progress object from heartbeat details and resumes from the appropriate position.

Activity Heartbeat retry flow after Worker crash

When a Worker crashes, the heartbeat timeout expires and Temporal retries the Activity on a new Worker. The new attempt retrieves the last heartbeat details using SDK-specific methods (getHeartbeatDetails in Java, activityInfo().heartbeatDetails in TypeScript, activity.info().heartbeat_details in Python, or GetHeartbeatDetails in Go) and resumes from the checkpoint instead of restarting.

Problems solved by Activity Heartbeats

Without heartbeats, long-running Activities require very long timeouts that delay failure detection, must reprocess entire batches from the beginning on failures, provide no visibility into Activity progress, risk zombie Activities that appear alive but are stuck, and require custom checkpointing and recovery logic.

Non-retryable errors pattern overview

The Non-Retryable Errors pattern marks specific error types so Temporal stops retrying immediately when one is raised. Use it for failures where the root cause is structural — invalid input, a missing record, an authorization problem — where repeating the same call will never produce a different result.

Pitfall: swallowing ActivityError without logging

Non-retryable errors fail fast and silently if you do not catch and log them. Always log the failure before re-raising or returning an error result.

Pitfall: confusing non-retryable errors with Workflow failures

A non-retryable ActivityError fails the Activity and delivers the error to the Workflow. The Workflow itself does not fail unless it re-raises the error without catching it.

Pitfall: using error message instead of type name for RetryPolicy

RetryPolicy.NonRetryableErrorTypes matches on type names, not message strings. Without a type name, the policy cannot identify the error.

Best practice: use both non-retryable mechanisms for defence in depth

Mark the error as non-retryable at the throw site so the Activity is self-describing, and also list the type in the RetryPolicy so the classification is enforced even if the Activity code changes.

Best practice: match non-retryable classification to error nature

Reserve non-retryable classification for truly permanent failures. A rate-limit error (HTTP 429) is transient — the same call will succeed after a delay. A not-found error (HTTP 404) is typically permanent. Match the non-retryable classification to the nature of the error.

Best practice: use specific error type names

Use domain-specific error type names rather than generic names like 'Error' or 'Failure'. For example, use 'OrderNotFoundError' or 'InsufficientFundsError' so the Workflow can distinguish between failure causes.

Best practice: validate input before scheduling Activity

If the Workflow can detect invalid input upfront — using an Update validator or by inspecting the input data — fail fast in the Workflow rather than paying the cost of an Activity execution.

Non-retryable error flow in Temporal

When an Activity raises an error, Temporal inspects whether the error type is non-retryable. If non-retryable — either because the Activity flagged it or the RetryPolicy lists the type — Temporal delivers the ActivityError to the Workflow without delay. If retryable, Temporal schedules another attempt after the configured backoff. The Workflow catches the ActivityError and handles it according to the business logic.

Two mechanisms for marking non-retryable errors

There are two complementary mechanisms for marking errors as non-retryable: (1) Mark the error as non-retryable at the throw site — the Activity explicitly signals that this specific failure should not be retried. (2) Register non-retryable error types in the RetryPolicy — the Workflow declares which error type names should never be retried, regardless of how the Activity raises them. Both mechanisms can be used together.

Why permanent failures should not be retried

Retrying permanent failures wastes time and resources. For example, a transfer to a non-existent account number will fail on attempt 1, 2, and 3 in exactly the same way. An API call with a malformed request body will be rejected every time. A request from a revoked API key will receive an authorization error on every attempt. With the default unlimited retry policy, the Workflow waits through exponential backoff delays — minutes to hours — before eventually delivering the error to the Workflow, when it could have failed in milliseconds.

Pitfall: marking transient errors as non-retryable

Network timeouts and service unavailability are transient errors. Marking them non-retryable removes Temporal's ability to recover automatically.

Eager Workflow Start pattern overview

Eager Workflow Start dispatches the first Workflow Task directly to a co-located Worker, bypassing the Temporal Matching Service. This pattern requires the starter and Worker to share the same process and client connection. It is available in Go, Java, and Python SDKs but not in TypeScript.

Pattern selection for first-response latency optimization

When optimizing primarily for first-response latency, use Early Return + Local Activities. The client gets its response in approximately 160 ms while background work continues independently.

Pattern selection for total workflow latency optimization

When optimizing only for total workflow latency and not first-response time, use Local Activities. If co-location is feasible, add Eager Workflow Start for the maximum latency reduction.

Pattern comparison table for three-Activity transaction workflow on Temporal Cloud

Performance benchmarks for reducing latency are: Baseline (regular Activities): ~850 ms first response, ~850 ms total latency. Early Return: ~265 ms first response, ~850 ms total latency. Local Activities: ~275 ms first response, ~275 ms total latency. Early Return + Local Activities: ~160 ms first response, ~275 ms total latency. Eager Workflow Start + Local Activities: ~265 ms first response, ~265 ms total latency (Go, Java, Python only). Early Return + Local Activities + Eager Start: ~160 ms first response, ~265 ms total latency (Go, Java, Python only). All patterns except those marked with SDK restrictions support all SDKs.

Local Activities pattern overview

Local Activities run Activity functions in-process inside the Workflow Task, eliminating all server scheduling round-trips. This pattern is best for short, idempotent Activities on a latency-sensitive path.

First Response vs Total Latency metrics

First Response is the time until the client receives an actionable result. Total Latency is the time until the Workflow fully completes. These metrics are used to distinguish patterns optimized for client responsiveness versus overall workflow completion speed.

Latency sources and overhead in Workflows

The three main sources of Workflow latency are: (1) Matching Service routing the first Workflow Task, which incurs ~30–50 ms overhead; (2) Activity scheduling round-trip, which incurs ~50 ms per Activity; (3) Client waiting for full workflow completion, which incurs total workflow duration overhead.

Baseline Temporal Workflow latency on Cloud

A default Temporal Workflow implementation using regular Activities scheduled through the Temporal server carries inherent latency. On Temporal Cloud, a typical three-Activity workflow reaches baseline latency of 850 ms or more.

Simplest pattern for getting started with latency optimization

Local Activities is recommended as the simplest pattern to begin with. It requires minimal structural change and provides the most straightforward per-Activity latency improvement.

Frequent polling implementation with heartbeats

For polling intervals of 1 second or faster, implement a loop inside an Activity that calls activity.heartbeat() on each iteration, checks the external service status, and sleeps for 1 second between iterations. The heartbeat reports progress and enables Temporal to detect stuck Activities. The Workflow configures the Activity with a heartbeat timeout shorter than the start-to-close timeout (e.g., heartbeat_timeout=2s, start_to_close_timeout=60s). If the Activity misses a heartbeat, Temporal detects the failure and retries the Activity on another Worker.

Infrequent polling implementation with Activity retries

For polling intervals of 1 minute or slower, implement a single-poll Activity that throws an exception when the external service is not ready. Temporal handles the retry scheduling based on the retry policy. Configure the retry policy with backoff_coefficient=1 (fixed interval) and initial_interval set to the desired polling frequency (e.g., 60 seconds). The start_to_close_timeout should be short (e.g., 2 seconds). Retries do not add events to the Workflow history, keeping it small.

When to use frequent polling

Frequent polling (1 second or faster) is a good fit for real-time status checks, high-priority operations requiring fast response, and short-lived external operations lasting minutes. It is not suitable for long-running operations lasting hours or days, rate-limited APIs, or resource-constrained external services.

Three polling strategies for different frequencies

The Polling External Services pattern implements three distinct strategies optimized for different polling frequencies: frequent polling (1 second or faster) using Activity loops with heartbeats, infrequent polling (1 minute or slower) using Activity retries with fixed backoff, and periodic sequence (complex polling) using Child Workflows with Continue-As-New.

Periodic sequence with Child Workflows and Continue-As-New

For complex polling sequences with changing parameters or multiple Activities between attempts, use Child Workflows with Continue-As-New to prevent unbounded history. The Child Workflow loops up to a maximum number of attempts (e.g., 10), executing an Activity and sleeping between attempts. After reaching the maximum attempts, it calls Continue-As-New with the same parameters to start a fresh execution. The parent Workflow starts the Child Workflow with a specific workflow ID and waits for its result. The parent remains unaware of the child's Continue-As-New calls.

When to use infrequent polling

Infrequent polling (1 minute or slower) is a good fit for batch job completion checks, long-running external processes, rate-limited APIs, and operations that may take hours or days. It is not suitable for sub-minute polling requirements or operations requiring immediate response.

Give your agent this brain