design-patterns

827 notes in this subject, read out of this brain and free to use. This is page 2 of 14.

Caching Run IDs with Continue-As-New

Continue-As-New creates a new Run ID. If external callers cache the old Run ID for Signals or Queries, they will get a 'workflow execution already completed' error. Always use Workflow ID without a Run ID (or an empty Run ID) so the request routes to the currently running execution.

Use built-in continue-as-new suggestion instead of fixed iteration count

Instead of tracking iteration counts manually, use the SDK's built-in suggestion to let Temporal tell you when the history is getting large. Different Workflow paths generate different numbers of events per iteration, so a fixed count may continue too early or too late. Use the SDK's built-in continue-as-new suggestion for accurate detection: `isContinueAsNewSuggested()` in Java, `continueAsNewSuggested` in TypeScript, `is_continue_as_new_suggested()` in Python, and `GetContinueAsNewSuggested()` in Go.

Continue-As-New benefits

Continue-As-New allows you to run Workflows indefinitely without history limits. Fresh history keeps Workflow execution fast. It reduces active storage costs by archiving old event history — more aggressive iteration limits mean more frequent archiving, keeping active storage minimal. The transition is atomic with no gap between old and new execution. You pass state as arguments to the new execution, and the Workflow ID remains the same, maintaining logical continuity for Queries and Signals.

Child Workflows and Continue-As-New interaction

Continue-As-New closes the current Workflow Execution, which triggers the Parent Close Policy on all Child Workflows. By default, children are terminated. If children must survive, set `ParentClosePolicy` to `ABANDON` and pass their Workflow IDs to the new execution so you can interact with them via external handles.

Continue-As-New vs alternatives comparison

Continue-As-New provides history reset with manual state continuity and is best for long-running periodic workflows. Child Workflows provide per-child history reset with automatic state continuity and are best for parallel processing. Cron Schedule provides history reset with no state continuity and is best for fixed schedule tasks. Manual Restart provides history reset with no state continuity and is best for one-time workflows.

Calling Continue-As-New from Signal handler

Triggering Continue-As-New inside a Signal handler can cause Signal loss because the handler may preempt other pending Signals. Always set a flag in the Signal handler and call Continue-As-New from the main Workflow thread, where all Signal handlers are guaranteed to have run first.

Continue-As-New use cases

The Continue-As-New pattern is a good fit for periodic Workflows running indefinitely (cron-like behavior), processing unbounded data streams, long-running Workflows with repetitive patterns, Workflows that accumulate state over many iterations, and preventing event history from growing too large. It is not a good fit for short-lived Workflows (under 1000 events), Workflows that naturally complete, one-time batch processing, or Workflows that require full history for audit purposes.

Continue-As-New pattern overview

The Continue-As-New pattern allows long-running Workflows to reset their event history by completing the current execution and immediately starting a new one with fresh state. This prevents Workflows from hitting Temporal's event history limits while maintaining logical continuity, making it essential for periodic tasks, infinite loops, and Workflows that process unbounded data streams. By archiving old event history and starting fresh, Continue-As-New also reduces active storage costs — only the current execution's history remains in active storage while previous runs are moved to cheaper archived storage.

Not versioning state arguments in Continue-As-New

When you change the Workflow method signature or state shape, in-flight executions may continue as new into code that cannot deserialize the old arguments. Use versioning or backward-compatible argument types to handle state evolution across Continue-As-New transitions.

nextRetryDelay longer than ScheduleToCloseTimeout causes expiration

If the nextRetryDelay override delay exceeds the remaining ScheduleToCloseTimeout budget, the retry will never execute because Temporal will expire the Activity before the delay elapses. The ScheduleToCloseTimeout must be long enough to accommodate the maximum possible nextRetryDelay value.

RetryPolicy constraints still apply with nextRetryDelay

When using nextRetryDelay to override the retry interval, the RetryPolicy's MaximumAttempts and ScheduleToCloseTimeout still apply. Only the interval for the next retry is overridden; the policy still governs maximum attempts and intervals for attempts where nextRetryDelay is not set.

nextRetryDelay only applies to the immediate next retry

The nextRetryDelay field only applies to the immediate next retry following the failure where it is set. If the following attempt also fails without setting nextRetryDelay again, the RetryPolicy interval resumes for subsequent retries.

Delayed Retry best practice: use error's own delay information

When implementing the Delayed Retry pattern, use the error's own delay information when available. HTTP 429 Retry-After headers, database lock timeouts, and API-provided backoff hints are more accurate than any value you could configure statically.

Delayed Retry best practice: fall back to RetryPolicy for unknown errors

When implementing the Delayed Retry pattern, only set nextRetryDelay for error types where you have reliable delay information. Let the RetryPolicy handle all other failures normally by not setting nextRetryDelay.

Delayed Retry pitfall: nextRetryDelay does not persist across retries

A common pitfall is assuming that nextRetryDelay persists across all retries. It only applies to the immediate next retry. If the following attempt also fails without setting nextRetryDelay, the RetryPolicy interval resumes.

Delayed Retry best practice: surface the delay in the failure message

When implementing the Delayed Retry pattern, include the delay value and its source in the ApplicationFailure message (for example, 'Rate limited — retrying after 60s (Retry-After header)') so it appears directly in the Workflow history and Activity failure details. This makes it clear why the Activity waited an unusual amount of time without requiring separate log correlation.

Delayed Retry best practice: still set meaningful RetryPolicy

Still set a meaningful RetryPolicy when using Delayed Retry. The nextRetryDelay overrides the interval for a single retry, but the RetryPolicy still governs maximum attempts and intervals for attempts where nextRetryDelay is not set. Ensure scheduleToCloseTimeout is long enough to accommodate the maximum possible nextRetryDelay value.

Delayed Start best practice: enable cancellation

Add Signal handlers to allow cancellation of a Workflow before it executes.

Delayed Start best practice: add Query methods

Expose status via Queries to allow clients to check Workflow state during the delay.

Delayed Start pitfall: Workflow ID reservation

A delayed Workflow reserves its Workflow ID immediately. Starting another Workflow with the same ID will fail depending on the ID reuse policy.

Delayed Start pitfall: sub-second delays

Temporal does not guarantee sub-second timer accuracy and the delay is rounded up due to scheduling latency. Treat the configured duration as a minimum, not an exact value.

Delayed Start pitfall: querying during delay

Querying a delayed Workflow before it starts may return errors or empty results because no Workflow code has executed yet and Query handlers have no state to return during the delay.

Signal handlers unavailable during delay

Signal handlers and Query handlers only run after the delay expires and the first Workflow Task is dispatched. To interact with a delayed Workflow before execution, use Signal-With-Start to bypass the delay, or cancel the Workflow Execution directly. Regular Signals sent to a delayed Workflow are buffered but have no handler to process them until the delay expires.

Delayed Start trade-offs

Trade-offs of the Delayed Start pattern include: you cannot dynamically adjust the delay after creation (use the Updatable Timer pattern for that), the pattern is for one-time delays only (for recurring Schedules use Temporal Schedules), very short delays (sub-second) provide minimal benefit since Temporal does not guarantee sub-second timer accuracy and the delay is rounded up to account for scheduling latency, the delay is time-based only not condition-based, and regular Signals sent during the delay are not delivered until the first Workflow Task fires, so Query and Signal handlers are not available until execution begins.

Delayed Start benefits

Benefits of the Delayed Start pattern include: the Workflow is queryable before execution starts (immediate visibility), no Worker resources are consumed during the delay, you can cancel the Workflow Execution before it runs, a Signal-With-Start or Update-With-Start bypasses the remaining delay, regular Signals sent during the delay do not interrupt it, the API is a single configuration option with no external schedulers needed, and the delay is managed by Temporal ensuring deterministic behavior.

Delayed Start best practice: set Workflow ID

Always set an explicit Workflow ID when using Delayed Start for tracking and cancellation purposes.

Delayed Start not suitable for recurring schedules

The Delayed Start pattern is not a good fit for recurring Schedules (use Temporal Schedules instead), immediate execution with internal delays (use Workflow sleep), complex scheduling logic (use Schedules with cron), or sub-second delays (minimal benefit).

Delayed Start use cases

The Delayed Start pattern is suitable for: scheduled one-time operations (send a reminder in 24 hours), grace periods before processing (cancel a subscription in 7 days), delayed notifications and alerts, deferred batch processing, and trial expiration Workflows.

firstWorkflowTaskBackoff delays first task execution

The Delayed Start pattern uses a start delay option in WorkflowOptions to defer the first Workflow Task by setting firstWorkflowTaskBackoff to the delay duration. The Workflow execution is created immediately with this backoff set, but no Workflow code runs until the delay expires.

Delayed Start pattern overview

The Delayed Start pattern enables Workflows to be created immediately but begin execution after a specified delay. The Workflow execution is registered in Temporal right away, but the first Workflow Task is scheduled to run only after the delay period expires. This is suitable for scheduled operations, grace periods, and deferred processing.

Saga Pattern requires compensation for every step with external effect

When using the Saga Pattern for distributed transactions, define a compensation for every step that has an external effect.

Early Return pattern for responding before transaction finishes

Use Early Return when you need to respond to the caller before the transaction finishes. Early Return returns a result to the caller as soon as initialization succeeds, while the remaining work continues asynchronously in the background.

Saga Pattern for undoing completed steps in distributed transactions

Use the Saga Pattern when you need to undo completed steps when a later step fails. The Saga Pattern manages a distributed transaction as a sequence of local steps, where each step defines a compensating action that undoes its effect if a later step fails.

Distributed transactions span multiple services with separate data

Distributed transactions span multiple services that each own their own data, with no shared database transaction to roll back. These patterns coordinate the steps, undo completed work when a later step fails, and keep external side effects correct under retries.

Pitfall: expecting perfectly even per-second rate

The limit is enforced across the queue's partitions, default four. The server maintains the configured rate as an average over time but can dispatch a short burst above it, up to roughly the rate divided across partitions. If the downstream service rejects any momentary overshoot, set the cap below the hard limit to leave headroom, or reduce the partition count for the queue.

When to use Downstream Rate Limiting

This pattern is a good fit when your Workflow calls a downstream service with explicit requests-per-second limits, when you need throughput enforcement that holds across many concurrent Workflow instances without per-Activity logic, and when only a subset of Activity types require throttling and others should run without restriction. It is not a good fit when you need concurrency limits rather than throughput limits, when the downstream system has no rate limit and throughput is bounded only by Workflow logic, or when all Activities require the same limit and a single shared queue suffices.

Downstream Rate Limiting pattern overview

The Downstream Rate Limiting pattern, also known as Task Queue rate limiting, caps how many Activities execute per second against a downstream service. Throttled Activities are placed on a dedicated Task Queue backed by Workers configured with MaxTaskQueueActivitiesPerSecond. The Temporal matching service enforces this limit before dispatching tasks, so the downstream service receives a controlled request rate regardless of how many Worker instances or Workflow executions are running concurrently.

Rate limiting enforcement point comparison

Pitfall: conflicting MaxTaskQueueActivitiesPerSecond limits

Setting conflicting MaxTaskQueueActivitiesPerSecond limits in workers is problematic. This setting is set in Workers and sent to the Task Queue when a Worker polls. If you have multiple Workers with conflicting settings, the Workers will overwrite each other as they poll.

MaxWorkerActivitiesPerSecond vs MaxTaskQueueActivitiesPerSecond

MaxWorkerActivitiesPerSecond instructs the SDK to self-throttle its polling — the Worker will not request a new Activity task if doing so would push it over this rate. Because the limit is per-process, multiple Workers on the same queue each apply it independently, so the effective queue throughput is the per-worker cap multiplied by Worker count. By contrast, MaxTaskQueueActivitiesPerSecond is a server-side instruction: the Temporal matching service slows dispatch for the entire queue regardless of how many Workers are polling, making it the correct tool for protecting a shared downstream service.

Pitfall: confusing throughput limits with concurrency limits

MaxTaskQueueActivitiesPerSecond controls starts per second; MaxConcurrentActivityExecutionSize controls simultaneous executions. Long-running Activities that hold slots for minutes may exhaust concurrency before the RPS cap applies. These are different kinds of limits and should not be confused.

Concurrency slots are not throughput limits

The concurrency slots (MaxConcurrentActivityExecutionSize, MaxConcurrentWorkflowTaskExecutionSize, MaxConcurrentLocalActivityExecutionSize) are not throughput limits but define the number of execution slots available on a Worker. A Worker will not accept more tasks than it has open slots, so a low slot count acts as an indirect throughput ceiling. This is distinct from throughput limits that control starts per second.

Pitfall: forgetting task_queue override in Activity options

If the Workflow does not explicitly specify task_queue in the Activity options, the Activity runs on the Workflow's default queue and bypasses the rate-limited Worker entirely.

Pitfall: setting rate cap far below actual demand

A cap much lower than actual submission rate causes the queue to grow unboundedly. Monitor queue depth and raise the cap or add more Workers when throughput requirements grow.

Downstream rate limiting best practice: worker redundancy and configuration

Run at least two Worker processes per queue for availability. A single Worker process is a single point of failure. Because MaxTaskQueueActivitiesPerSecond is a server-side per-queue limit rather than a per-worker one, set the same value on every Worker that polls the queue. Set each Worker to the target RPS — for example, 5 on each of two Workers yields a combined queue limit of 5, not 10. If Workers report different values, the server applies the value from the last Worker that polled.

Downstream rate limiting best practice: separate queues

Use a separate Task Queue for each rate limit. MaxTaskQueueActivitiesPerSecond applies to every Activity on the queue. Mixing rate-limited and unrestricted Activities on the same queue will throttle the unrestricted ones too.

Downstream rate limiting best practice: monitoring

Monitor queue depth and schedule latency. Track the temporal_activity_schedule_to_start_latency metric on the rate-limited queue; sustained growth signals that demand consistently exceeds the configured cap. You can also query the Task Queue's ApproximateBacklogCount via the DescribeTaskQueue API — a steadily growing backlog count is a direct indicator that the configured RPS cap is too low for the current submission rate.

Downstream rate limiting benefits and trade-offs

Centralizing rate limiting at the Task Queue ensures enforcement even when any number of Workflow instances run in parallel. Because the Temporal server controls dispatch, the limit holds regardless of how many Worker replicas are running — provided you account for Worker count when setting the per-worker cap. Dedicated Task Queues require operating additional Workers. If the throughput cap is set too low relative to demand, the queue depth grows and scheduling latency increases. You must size the Worker pool so that slot availability does not become the bottleneck before the rate limit is reached.

Entity Lifecycle Patterns overview

Entity Lifecycle Patterns model long-lived business entities as Workflows and keep those Workflows healthy as they run for days, months, or indefinitely. They cover how an entity holds and mutates state, how to bound Workflow history growth, and how to manage timers that change over time.

Continue-As-New pattern prevents unbounded history growth

The Continue-As-New pattern prevents unbounded history growth by completing the current Workflow execution and starting a fresh one that carries forward the current state. Apply this pattern when your Workflow runs long enough to grow a large history.

Updatable Timer pattern for dynamic wait periods

The Updatable Timer pattern provides a timer that you can extend, shorten, or cancel in response to Signals or Updates while the Workflow waits. Use this pattern when you need a wait that responds to new information instead of using a fixed sleep.

Early Return best practice: set WorkflowIdConflictPolicy to FAIL

For early return, use FAIL to assert a new Workflow is created per request. Use USE_EXISTING only for lazy initialization patterns.

Early Return pitfall: blocking too long in Update handler

The Update handler should return quickly. Perform long-running work in the main Workflow method and use Workflow.await in the Update handler to wait for a result.

Early Return pattern overview

The Early Return pattern returns initialization results to the caller immediately while continuing asynchronous processing in the background. It uses Update-with-Start to split operations into a fast synchronous initialization phase that validates and returns results immediately, and a slower asynchronous completion phase that runs in the background.

Early Return best practice: use local Activities for initialization

Use local Activities for initialization to avoid extra server roundtrips, keeping the synchronous phase fast (under 5 seconds).

Early Return pattern solution approach

The Workflow uses local Activities for quick initialization, signals completion via Update handlers, then either completes or cancels the operation based on initialization success. The client receives the initialization result in a single round trip while the Workflow continues processing.

Early Return pattern implementation across SDKs

The Workflow registers an Update handler that blocks until initialization completes using a condition flag (workflow.Await in Go, Workflow.await in Java, condition in TypeScript, workflow.wait_condition in Python), then returns the result to the caller. The client receives the initialization result via Update-with-Start in a single API call that also starts the Workflow.

Early Return pitfall: missing WorkflowIdConflictPolicy

Update-with-Start requires a WorkflowIdConflictPolicy. Omitting it causes an error. Use FAIL for early return (one Workflow per request) or USE_EXISTING for lazy initialization.

Early Return pitfall: assuming Update-with-Start is atomic

Unlike Signal-with-Start, Update-with-Start is not atomic. The Workflow may start even if the Update fails (for example, if no Worker is available). Handle this by checking Workflow state after the call.

Early Return best practice: avoid Workflow timeouts

Do not set Workflow Execution timeouts when using early return, as the background phase may take longer than expected.

Give your agent this brain