Fast/Slow Retries solves retry policy trade-offs
Conventional retry policies force a choice between three bad options: (1) Low MaximumAttempts recovers from transient errors quickly but abandons requests during longer outages; (2) Unlimited or high MaximumAttempts with short interval floods a degraded downstream system with retries and accumulates noisy failures; (3) Long fixed interval with unlimited retries recovers from outages eventually but is too slow for transient errors. The Fast/Slow Retries pattern handles both scenarios well by using the Workflow as a retry orchestrator.
Fast/Slow Retries pattern overview
The Fast/Slow Retries pattern orchestrates two retry phases in a Workflow: a fast phase with short intervals and bounded attempts for transient errors, followed by a slow phase with long intervals and unlimited retries for extended outages. Use this when a single RetryPolicy should not cover both brief blips and hour-long outages or maintenance windows.
Fast/Slow Retries outer bound with timeout
Phase 2 runs indefinitely by default. If the business process has a maximum wait time, add a ScheduleToCloseTimeout or use a Workflow execution timeout to impose an outer bound.
Fast/Slow Retries pitfall: catching too broadly in phase 1
Catch ActivityError specifically. Catching all exceptions in Phase 1 may swallow errors that should propagate immediately, such as CancelledError in Python or a PanicError in Go.
Fast/Slow Retries pitfall: exponential backoff in phase 2
Do not use exponential backoff in Phase 2. The default BackoffCoefficient is 2.0, which doubles the interval with each attempt. Set BackoffCoefficient=1.0 in the slow phase to keep the interval fixed and predictable.
Best practice: Use idempotency keys for retry safety
Use idempotency keys when retrying. It is vital to have downstream systems detect and discard duplicate calls to avoid duplicate downstream effects.
Pitfall: Ignoring ActivityError in Workflow
Do not ignore the ActivityError in the Workflow. Exhausted retries raise an error in the Workflow. If you do not catch it, the Workflow fails without any compensation or alerting.
Idempotency keys protect against worker crash after API success
If a Worker crashes after the API call succeeds but before the result is recorded, Temporal will not retry — the call is lost. An idempotency key (a stable identifier derived from the Workflow and Activity IDs) lets the downstream system detect and discard duplicates if a retry is needed in future.
Pitfall: No timeout with low attempt cap
Do not set a low attempt cap without a timeout. Without StartToCloseTimeout, a single hanging attempt can block all retries for minutes or hours.
Pitfall: Confusing MaximumAttempts with retry count
Do not confuse MaximumAttempts with allowed retry count. MaximumAttempts=3 means 3 total attempts (1 initial + 2 retries), not 3 retries after the initial attempt.
Best practice: Prefer non-retryable errors for structural failures
Prefer non-retryable errors for structural failures. If the failure is not transient (for example, invalid input), mark it as non-retryable rather than relying solely on maximum_attempts.
Best practice: Catch ActivityError explicitly in Workflow
Catch ActivityError in the Workflow and handle the exhausted-retries case explicitly — log, alert, compensate, or escalate — rather than letting it fail the Workflow silently.
Best practice: Combine MaximumAttempts with StartToCloseTimeout
Combine MaximumAttempts with StartToCloseTimeout. A per-attempt timeout prevents a slow response from consuming the entire retry budget on a single hanging call.
Best practice: Match retry cap to cost model
Match the cap to the cost model. If the API charges per call, set maximum_attempts to the maximum number of calls you are willing to pay for per Workflow execution.
Pitfall: Disabling retries without safeguards
Do not disable retries on operations without safeguards. Setting maximum_attempts=1 means any failure — including a Worker crash after the API responded — results in a permanent gap.
Disable retries with maximum_attempts=1
Set maximum_attempts=1 to disable retries entirely. The Activity starts once and any failure is immediately delivered to the Workflow. This is appropriate when the operation is not idempotent and a second attempt would cause a duplicate side effect such as a double charge or duplicate email.
ActivityError delivered when MaximumAttempts reached
When the MaximumAttempts limit is reached, Temporal stops retrying and delivers an ActivityError to the Workflow. The Workflow can catch that error and decide whether to fail, alert, or escalate.
MaximumAttempts includes initial attempt in count
When setting maximum_attempts (Python), MaximumAttempts (Go/Java), or maximumAttempts (TypeScript) on the retry policy, the count includes the initial attempt. Setting maximum_attempts=3 means one attempt plus two retries, for a total of 3 attempts.
Default retry policy retries Activities indefinitely
Temporal's default retry policy retries Activities indefinitely with exponential backoff. This is appropriate for most infrastructure failures but creates problems when the Activity calls a paid third-party API, leading to potentially unbounded costs.
Fixed Count of Retries pattern overview
The Fixed Count of Retries pattern caps the total number of Activity execution attempts by setting MaximumAttempts on the RetryPolicy. Use this pattern when each attempt consumes a paid API call, a rate-limited token, or any scarce resource where unbounded retries translate directly to unbounded cost.
Fixed Wall-Time Retries pitfall: not accounting for backoff delays in budget
The total time includes both attempt durations and the backoff delays between them. A 1-hour budget with a 30-minute initial interval and coefficient 2.0 leaves room for only one or two attempts.
Fixed Wall-Time Retries pitfall: ScheduleToCloseTimeout shorter than StartToCloseTimeout
Setting ScheduleToCloseTimeout shorter than StartToCloseTimeout means the first attempt cannot finish within the budget — the Temporal Service times out the Activity Execution and returns an error before any attempt can succeed.
Fixed Wall-Time Retries pitfall: using StartToCloseTimeout alone for SLA enforcement
Using StartToCloseTimeout alone for SLA enforcement is insufficient. A downstream system that responds slowly but never fully times out can keep resetting the per-attempt clock indefinitely.
Fixed Wall-Time Retries best practice: handle ActivityError explicitly
When the SLA expires, Temporal delivers an error to the Workflow. Catch it to send an alert, trigger a compensation, or record a breach in an audit log.
Fixed Wall-Time Retries best practice: cap MaximumInterval below SLA
Cap MaximumInterval well below the SLA. If MaximumInterval is 2 hours and the SLA is 24 hours, only 12 retries are possible. Tune the interval so the backoff plateaus at a value that allows meaningful retries within the budget.
Fixed Wall-Time Retries best practice: set both timeouts for clarity
Use ScheduleToCloseTimeout as the total SLA and StartToCloseTimeout as a per-attempt safety valve. Omitting StartToCloseTimeout means a single slow response can consume the entire budget.
StartToCloseTimeout limitation without ScheduleToCloseTimeout
StartToCloseTimeout limits how long a single Activity attempt may run before Temporal cancels it and schedules a retry, but it does not limit how long retries collectively may run. A process with StartToCloseTimeout=5m and the default unlimited retry policy can run for days — each attempt times out at 5 minutes, then Temporal waits for the backoff delay and tries again, indefinitely.
ScheduleToCloseTimeout purpose and behavior
ScheduleToCloseTimeout starts when the Activity is first scheduled and expires when the clock runs out, regardless of how many attempts have occurred. When the timeout expires, the Temporal Service marks the Activity Execution as timed out and delivers an ActivityError to the Workflow. No further retries are scheduled. A timeout does not forcibly stop Activity code that is already running, so an Activity that runs past the budget must heartbeat and handle cancellation to stop cooperatively.
Fixed Wall-Time Retries pattern overview
The Fixed Wall-Time Retries pattern enforces a maximum total elapsed time across all Activity retry attempts using ScheduleToCloseTimeout. Use it when a business process must succeed or fail within a defined time budget, regardless of how many individual attempts occur.
Fixed Wall-Time Retries best practice: distinguish SLA breaches from transient errors
Inspect the error cause — check that the ActivityError's cause is a TimeoutError with TimeoutType.SCHEDULE_TO_CLOSE (Python) or a TimeoutFailure with TimeoutType.SCHEDULE_TO_CLOSE (TypeScript) or TIMEOUT_TYPE_SCHEDULE_TO_CLOSE (Go/Java) to separate an SLA breach from an application failure. This lets you log or alert specifically on SLA violations rather than treating all activity errors the same way.
Fixed Wall-Time Retries pitfall: ScheduleToStart delay consumes budget
ScheduleToCloseTimeout begins when the Activity is first scheduled, which includes the time the task waits in the queue before a Worker picks it up. Under high load or insufficient Worker capacity, tasks can sit in the queue for seconds or minutes before the first attempt starts — consuming SLA budget before any work is done. Provision Workers with enough capacity for peak traffic, or use autoscaling, to keep ScheduleToStart latency negligible relative to the SLA window.
Fixed Wall-Time Retries pitfall: ignoring SLA breach in Workflow
Letting the ActivityError propagate without handling it means SLA breaches go unlogged and uncompensated.
Sliding Window pattern
Sliding Window maintains a fixed number of concurrently active Child Workflows, starting a new one each time an existing one completes.
Batch Iterator pattern
Batch Iterator pages through unbounded datasets using Continue-As-New to prevent history overflow while maintaining exactly-once processing guarantees.
Request-Response via Updates pattern
Request-Response via Updates provides synchronous request-response with validation. Updates modify state and return results directly.
Fan-Out with Child Workflows pattern
Fan-Out with Child Workflows distributes a large record set across parallel Child Workflows for concurrent processing with automatic scaling.
Signal with Start pattern
Signal with Start starts a Workflow when Signaling it if it does not already exist. If already running, it receives the Signal directly.
Pick First (Race) pattern
Pick First starts multiple Activities in parallel and uses the first result, cancelling the rest.
Design Patterns Catalog
Temporal provides a comprehensive catalog of reusable design patterns organized into categories: Task orchestration patterns (Child Workflows, Parallel Execution, Pick First), Workflow messaging patterns (Signal with Start, Request-Response via Updates), Entity & lifecycle patterns (Entity Workflow, Continue-As-New, Updatable Timer), External interaction patterns (Polling, Long-Running Activity, Delayed Start, Delayed Callback, Approval), Distributed transaction patterns (Saga Pattern, Early Return), Error handling & retry patterns (Fixed Count Retries, Fixed Wall-Time Retries, Non-Retryable Errors, Delayed Retry, Fast/Slow Retries, Retry Alerting, Resumable Activity), Batch processing patterns (Fan-Out with Child Workflows, Batch Iterator, Sliding Window, MapReduce Tree), QoS & throughput patterns (Downstream Rate Limiting, Priority Task Queues, Fairness), Performance & latency patterns (Local Activities, Early Return + Local Activities, Eager Workflow Start), and Worker configuration patterns (Worker-Specific Task Queues, Activity Dependency Injection).
Resumable Activity pattern
Resumable Activity parks the Workflow after retries are exhausted and waits for a human to signal a correction, then resumes execution from where it left off.
Fixed Wall-Time Retries pattern
Fixed Wall-Time Retries bounds the total elapsed time across all retry attempts to enforce a business SLA, regardless of how many individual attempts occur.
Non-Retryable Errors pattern
Non-Retryable Errors marks error types that will never succeed — such as validation failures or missing records — so Temporal fails fast instead of retrying indefinitely.
Polling External Services pattern
Polling External Services defines strategies for polling external resources with varying frequencies: frequent, infrequent, and periodic patterns.
Entity Workflow pattern
Entity Workflow models long-lived business entities as individual Workflows that persist for the entity's entire lifetime, handling all state transitions through Signals and Updates.
Updatable Timer pattern
Updatable Timer provides dynamically adjustable timers that respond to Signals or Updates. Timers can be extended, shortened, or cancelled based on external events.
Long-Running Activity pattern
Long-Running Activity enables long-running Activities to report progress via heartbeats and resume after failures with cancellation support.
Delayed Retry pattern
Delayed Retry overrides the next retry interval for a specific failure using nextRetryDelay on ApplicationFailure. It is used when an error carries information about how long to wait before retrying.
Saga Pattern
Saga Pattern manages distributed transactions with compensating actions. Each step has a compensation that undoes its effects if subsequent steps fail.
Early Return pattern
Early Return provides synchronous initialization with asynchronous completion. It returns results immediately while processing continues in the background.
Delayed Callback (Webhooks) pattern
Delayed Callback integrates webhooks durably by receiving inbound webhooks via Signals, firing delayed outbound callbacks with durable timers, and completing Activities asynchronously via task tokens.
Approval pattern
Approval implements human-in-the-loop Workflows that block until external approval decisions are made. It uses Signals to capture approval data with metadata.
Fast/Slow Retries pattern
Fast/Slow Retries tries aggressively with a short interval first, then shifts to a long interval when fast retries are exhausted, keeping the Workflow alive until the downstream system recovers.
MapReduce Tree pattern
MapReduce Tree recursively splits a dataset into a binary tree of Child Workflows, processes leaves in parallel, then aggregates results back up the tree.
Continue-As-New pattern
Continue-As-New prevents unbounded history growth by completing the current execution and starting a new one with fresh history.
Delayed Start pattern
Delayed Start creates Workflows immediately but defers execution until a specified delay expires. It fits one-time scheduled operations and grace periods.
Fixed Count of Retries pattern
Fixed Count of Retries caps the number of Activity retry attempts to control cost when each attempt consumes a paid or limited resource.
Local Activity definition and behavior
A Local Activity executes the Activity function directly inside the Worker process that is currently running the Workflow Task. The result is recorded as part of the same Workflow Task completion event, so no additional server calls occur between Activity invocations.
Local Activity timeout configuration
Local Activities are subject to the Workflow Task timeout (default 10 seconds) rather than an independent start-to-close timeout. Always configure the `scheduleToCloseTimeout` option (not `startToCloseTimeout`) to set an upper bound.
Regular Activity five-step exchange
A regular Activity follows a five-step exchange with the Temporal server: schedule, dispatch, execute, complete, then resume the Workflow Task. On Temporal Cloud, each round-trip adds approximately 50 ms.
Local Activity server round-trips savings
A Local Activity bypasses all server scheduling overhead. The Workflow Task scheduler on the Worker invokes the Activity function directly. The result is folded into the WorkflowTask completion event sent at the end of the task. Each Activity converted saves approximately 50 ms of scheduling overhead on Temporal Cloud.