Early Return + Local Activities latency and complexity comparison table
Synchronous workflow: First Response ~850 ms, Total Latency ~850 ms, Complexity Low. Early Return (regular activities): First Response ~265 ms, Total Latency ~850 ms, Complexity Medium. Local Activities only: First Response Same as total ~275 ms, Total Latency ~275 ms, Complexity Medium. Early Return + Local Activities: First Response ~160 ms, Total Latency ~275 ms, Complexity Medium. Eager Workflow Start + Local Activities: First Response ~160 ms, Total Latency ~265 ms, Complexity High.
Early Return pattern use case description
The Early Return pattern is the baseline Update-with-Start pattern without Local Activity optimization. It uses Update-with-Start to send the client an early response after a fast Phase 1 completes, while slow Phase 2 work continues in the background. The Early Return + Local Activities pattern extends this by running Phase 1 Activities as Local Activities, eliminating all server round-trips on the synchronous hot path.
Delayed Callback pattern overview
The Delayed Callback pattern manages delayed completion notifications between systems using durable timers. It solves problems with webhook-based integrations by providing durable infrastructure without requiring ad hoc queues, cron jobs, or fragile state machines. Three sub-patterns are available: inbound webhooks via Signal-with-Start, delayed outbound callbacks with durable timers, and async activity completion via task tokens.
Pattern 3: Async activity completion via task tokens
The Workflow executes an activity that submits a job to an external system. The activity records its task token (an opaque handle provided by Temporal) alongside the submitted job ID in a database. The activity returns without waiting, and the Workflow pauses waiting for external completion. When the external system finishes, it calls the callback endpoint with the result. The callback handler retrieves the task token from the database and completes the activity through the Temporal client using that token. The Workflow resumes immediately with the result.
Timer duration limits for Pattern 2
Timer durations range from one second to several years. Workflows should not rely on sub-second accuracy for timers. Pattern 2 (delayed outbound callbacks) should not be used for delays shorter than one second.
Deterministic Workflow IDs for Signal-with-Start
Workflow IDs must be deterministic and stable across webhook deliveries (for example, order ID, user ID) so that Signal-with-Start routes to the correct instance. Non-deterministic Workflow IDs generated from timestamps or random values cause Signal-with-Start to create a new Workflow on every webhook delivery instead of routing to the existing one.
Signal-with-Start pitfall: plain Signal to non-existent Workflow
Sending a plain Signal to a Workflow that does not exist causes an error. Use Signal-with-Start when the Workflow may not be running.
Pattern 3 pitfall: losing the task token
If the service storing task tokens is unavailable when the callback arrives, the activity can never complete. Task tokens for async completion must be persisted durably outside Temporal (for example, in a database) in a transactional write alongside the job submission. Storing tokens only in process memory or an unavailable store prevents callback completion.
Pattern 3 pitfall: forgetting to signal async completion
Forgetting to signal async completion (raise_complete_async() in Python, ErrResultPending in Go, doNotCompleteOnReturn() in Java) causes Temporal to mark the activity as completed immediately when the function returns, before the external callback arrives. This prevents the external callback from completing the activity.
Pattern 2 pitfall: using time.sleep instead of workflow.sleep
Using time.sleep() (non-durable) in Pattern 2 instead of workflow.sleep() is a common mistake. A process sleep disappears on restart; only Temporal's durable timer survives worker restarts.
Pattern 1: Inbound webhooks via Signal-with-Start
Signal-with-Start atomically creates a Workflow (if not running) and delivers a Signal in one operation. An external service POSTs a webhook to your API handler, which calls signal_with_start with the Workflow ID and payload. The handler returns HTTP 200 immediately after the call, and Temporal takes responsibility for delivery. Temporal atomically starts the Workflow if needed, then delivers the Signal with no race condition. The Workflow wakes up where it was blocked and processes the payload.
Pattern 2: Delayed outbound callbacks with durable timers
The client starts a Workflow with a target URL, payload, and delay duration. The Workflow calls workflow.sleep() which stores a durable timer in the Temporal cluster, not in process memory. If any worker restarts during the delay, the timer continues. When the timer fires, Temporal schedules the next Workflow Task on a healthy worker. The Workflow then executes an activity that performs the outbound HTTP POST. If the POST fails, Temporal retries it with the configured retry policy.
Best practice: HTTP 200 response timing for inbound webhooks
Return HTTP 200 from your inbound webhook handler as soon as you have called signal_with_start; do not wait for the Workflow to process the payload. This ensures the caller knows their webhook was accepted without waiting for business logic processing.
Best practice: timeout for inbound webhook Workflows
Add a timeout to the workflow.wait_condition / Workflow.await call in inbound webhook Workflows so they do not wait indefinitely if the webhook is never delivered.
Best practice: start_to_close_timeout for outbound callbacks
Set a realistic start_to_close_timeout on outbound callback activities — long enough for the destination to respond, short enough to surface failures quickly.
Best practice: heartbeating for long-running external jobs in Pattern 3
For Pattern 3 (async activity completion), use heartbeat if the external job takes longer than the activity heartbeat timeout to report back. Heartbeating keeps the activity lease alive during extended external job execution.
Delayed Callback pattern use cases
Use Signal-with-Start (Pattern 1) when an external service POSTs a webhook and the Workflow may or may not be running. Use workflow.sleep() + activity (Pattern 2) to fire an outbound HTTP callback after a delay ranging from seconds to years. Use async activity completion (Pattern 3) to submit a job to an external system and wait for its completion webhook. Use the Polling External Services pattern instead when the external system does not support webhooks.
Benefits of Delayed Callback pattern
Retries and backoff on outbound HTTP calls come for free via Temporal's retry policy, eliminating the need for custom retry queues. Workflow state survives worker restarts, deploys, and infrastructure failures; durable timers continue without a running process. Every in-flight delayed callback is visible in the Temporal UI with its scheduled time, payload, and retry count. Signal-with-Start eliminates the race condition between checking if a Workflow exists and delivering the event. Async activity completion decouples job submission from job completion without polling.
Trade-offs of Delayed Callback pattern
Your inbound webhook handler requires a Temporal client; you need the client library in the service receiving webhooks. Task tokens for async completion must be persisted outside Temporal (for example, in a database); if that store is unavailable the callback cannot complete. Workflow IDs must be deterministic and stable across webhook deliveries (order ID, user ID, etc.) so that Signal-with-Start routes to the correct instance.
Delayed Callback pattern comparison with alternatives
Temporal Signals + Workflows provide durable execution that survives restarts, built-in configurable retries, full Temporal UI observability, and low complexity with composable primitives. Message queues (SQS, Kafka) offer durable queue-level execution but limited manual retries and require external tooling for observability, with medium complexity for handling ordering and DLQ. Redis SET + cron job has volatile in-memory storage, manual retries, no observability, and high complexity requiring cron + polling + error handling. Direct HTTP retry loops have process-lifetime-only durability, manual retries using time.sleep, no observability, and high fragility without a process supervisor.
Fairness weight example calculation
Assigning weights of 5.0, 3.0, and 2.0 to premium, basic, and free tiers causes 50% of dispatched tasks to come from premium, 30% from basic, and 20% from free—regardless of backlog depth. Within a single fairness key, tasks are dispatched in FIFO order.
Enable Fairness in Temporal Cloud
Navigate to the Namespace's Overview page in the UI and activate the Fairness toggle. Fairness is a paid feature in Temporal Cloud.
Fairness pitfall: not a hard rate limiter
Fairness controls proportional dispatch but does not cap the absolute throughput of any one key. For hard throughput caps, combine Fairness with per-fairness-key RPS limits via the CLI.
Fairness pitfall: unkeyed tasks participate in round-robin
Tasks without a FairnessKey are grouped under an implicit empty-string key and participate in round-robin dispatch alongside named keys with a weight of 1.0. They do not bypass Fairness and compete as one group.
Fairness pitfall: task queue partitioning reduces accuracy
Task Queues are internally partitioned and tasks are distributed to partitions randomly, which can interfere with fair dispatch proportions. If your workload requires higher accuracy, contact Temporal Support to configure a single-partition Task Queue.
Enable Fairness in self-hosted Temporal
Set matching.enableFairness to true in the dynamic config for the relevant Task Queues or Namespaces.
Fairness pattern overview and purpose
The Fairness pattern distributes Worker capacity proportionally across tenants or user groups within a single Task Queue so that a burst from one caller cannot starve others. Each group is assigned a fairness key and an optional weight; the Temporal matching service dispatches tasks in weighted round-robin order across all keys.
Fairness pitfall: inconsistent immediately after server restart
Fairness ordering is preserved across restarts for the most active keys. Less active keys may briefly dispatch new tasks ahead of their existing backlog until ordering normalizes.
Fairness pitfall: running task mix does not immediately reflect fair dispatch
Fairness governs which task is dispatched next; it does not account for tasks already running on Workers. The mix of in-flight tasks at any moment may not match the configured weight ratios.
Fairness key and weight assignment
Assign a FairnessKey (a string identifier such as a tenant name or tier) and an optional FairnessWeight (a positive float, default 1.0) to Workflows, Activities, and Child Workflows. The Temporal matching service creates a virtual queue for each key and dispatches tasks in proportion to their weights.
Activity fairness key inheritance
Activities inherit the parent Workflow's fairness key and weight. Override them in ActivityOptions when an Activity should belong to a different fairness group than its Workflow. Each field (priority_key, fairness_key, fairness_weight) is resolved independently in this order: Task Queue weight overrides (highest precedence), value set explicitly in the options, value inherited from the calling Workflow, then the default.
Set queue-level and per-key rate limits via CLI
Use temporal task-queue config set with --queue-rps-limit to set an overall Task Queue rate limit and --fairness-key-rps-limit-default to set a default per-fairness-key limit. The per-key limit is scaled by the fairness weight for that key, so a key with weight 2.5 and a default per-key limit of 10 gets an effective limit of 25 tasks/second.
Example:
```sh
temporal task-queue config set \
--task-queue my-task-queue \
--task-queue-type activity \
--namespace my-namespace \
--queue-rps-limit 500 \
--queue-rps-limit-reason "overall limit" \
--fairness-key-rps-limit-default 33.3 \
--fairness-key-rps-limit-reason "per-key limit"
```
Override fairness weights via CLI
When it is more convenient to manage weights through configuration than to embed them in client code, use temporal task-queue config set with --fairness-key-weight to override weights for up to 1000 keys per Task Queue. Overrides take precedence over the weight attached to a task's options and can be updated without a code deploy.
Example:
```sh
temporal task-queue config set \
--task-queue my-task-queue \
--task-queue-type workflow \
--namespace my-namespace \
--fairness-key-weight premium=5.0 \
--fairness-key-weight basic=3.0 \
--fairness-key-weight free=2.0
```
Combine Priority and Fairness together
Priority and Fairness can be combined. Priority determines which sub-queue (1–5) a task enters; Fairness determines the dispatch order within each priority level. Set both PriorityKey and FairnessKey on the same options object.
When to use Fairness pattern
This pattern is a good fit for multi-tenant applications where large tenants should not block small tenants, for workloads that need proportional capacity allocation across groups without hard rate limits, and when the set of tenants or groups is dynamic (new keys can be introduced without deploying new Workers).
When not to use Fairness pattern
It is not a good fit when absolute throughput isolation is required; dedicated queues per tenant or task queue priorities are the appropriate choice.
Fairness pitfall: does not apply across Worker Versioning boundaries
When using Worker Versioning and moving Workflows between versions, Priority still applies across versions but Fairness is only guaranteed within tasks originally queued on the same Worker version. Tasks moved from one version to another may not dispatch in fairness order relative to tasks on the destination version.
Fairness benefits
A single Worker pool serves all tenants; idle capacity from a low-traffic tenant automatically benefits high-traffic tenants rather than going to waste. New tenants require no Worker deployment—add a fairness key and Temporal starts dispatching their tasks immediately. Weights can be updated via CLI without redeploying application code.
Fairness trade-offs
Fairness requires explicit enablement on Temporal Cloud and self-hosted deployments. Accuracy can degrade with a very large number of fairness keys. Fairness weight applies at schedule time, not dispatch time: changing a weight does not retroactively reorder tasks already in the backlog.
Comparison of Fairness with alternatives
Comparison table of multi-tenant approaches:
- Temporal FairnessKey (native): Soft tenant isolation, dynamic tenants supported, shares idle capacity, low complexity
- Dedicated queue per tenant: Hard tenant isolation, dynamic tenants not supported, does not share idle capacity, medium complexity
- Single shared queue (no control): No tenant isolation, dynamic tenants supported, shares idle capacity, lowest complexity
- External queue with per-tenant consumer groups: Hard tenant isolation, dynamic tenants supported, does not share idle capacity, high complexity
Use stable, consistent naming for fairness keys
Use account IDs or tenant slugs rather than display names for fairness keys. Key names cannot be changed retroactively on tasks already in the backlog.
Combine Priority and Fairness for multi-class, multi-tenant workloads
Priority separates urgent from batch work; Fairness prevents any single tenant from dominating within each priority level.
Monitor queue depth by fairness key
Sustained backlog growth for a particular key means its weight fraction of Worker capacity cannot drain its submission rate.
Fairness pitfall: does not reorder existing backlog
Fairness weight is evaluated at schedule time. Enabling Fairness on a Namespace with an existing backlog drains that backlog in its original order first; the fairness-aware dispatch mode takes effect only for newly submitted tasks.
Event Accumulator pitfall: Continue-As-New blocking
Signal rate too high to allow Continue-As-New to complete is a pitfall. Continue-As-New requires a brief window (approximately 100 ms) with no unhandled signals. If producers send signals continuously without pause, the workflow can never enter that window, history grows without bound, and Temporal will eventually terminate the workflow. Partition by a finer-grained key, throttle producers, or batch multiple events into a single signal payload to keep the per-instance signal rate low enough for Continue-As-New to succeed.
Event Accumulator pattern purpose
The Event Accumulator pattern is used to durably collect and process events from multiple senders over unlimited time. The workflow accumulates signals, deduplicates by a stable item key, and processes the batch after a sliding inactivity timeout, with no external coordination and no lost events on retry.
Event Accumulator group key concept
A group key is a stable, domain-specific identifier such as an order ID, customer ID, or session token that logically binds related events belonging to the same accumulation window. A single workflow instance per group key receives signals as events arrive, deduplicates them, and waits with a sliding inactivity timer.
Event Accumulator problems without the pattern
Without the Accumulator pattern, you must handle race conditions when multiple producers try to start the same collection workflow simultaneously, implement deduplication externally since at-least-once delivery is common, manage a reset timer that extends the collection window each time a new event arrives without a reliable durable timer primitive, persist collection state externally across restarts and failures, and handle gracefully cases where a long accumulation period grows the workflow history beyond safe limits.
Event Accumulator solution approach
Assign each group a deterministic workflow ID derived from the group key (for example, accumulator-order-123). Producers call Signal-With-Start so the workflow is created on first use and receives additional signals on subsequent calls without any client-side coordination. Inside the workflow, Workflow.await() with a timeout implements a sliding inactivity window where each arriving signal resets the countdown. When the countdown expires or when an explicit flush signal is sent, the workflow passes all accumulated, deduplicated events to a batch processing activity and completes. If the accumulation period is long enough to grow the workflow history near its limit, the workflow uses Continue-As-New to carry its state forward into a fresh run.
Event Accumulator sliding window mechanism
A sliding inactivity window is implemented using Workflow.await() with a timeout. Each arriving signal resets the countdown. When the countdown expires with no new signals, or when an explicit flush signal is sent, the workflow processes the batch.
Event Accumulator deduplication
The accumulator maintains a deduplication set of seen keys. When a signal arrives, the workflow checks if the item key is already in the set. If the item is already recorded, it is discarded as a duplicate. Alternatively, you can replace the existing record with the new payload, which is useful when a producer may resend an updated version of the same event under the same key.
Event Accumulator Signal-With-Start requirement
Producers must always use Signal-With-Start to send signals to the accumulator workflow. This atomically starts the workflow if not running and delivers the first signal without any client-side coordination. Calling start and signal separately is not atomic; a signal sent between the two calls can be lost if the workflow has not yet started.
Event Accumulator Continue-As-New state passing
Accumulated state must be passed as workflow arguments into each Continue-As-New run. Workflow state does not survive a Continue-As-New transition automatically. Always pass items and the seen-keys list as arguments to the new run. If you omit the seen-keys list, any signal delivered to both the old and new run during the Continue-As-New handoff will be processed again in the new run, producing duplicate entries in the batch.
Event Accumulator signal handler requirements
Signal handlers must be fast and side-effect free. They must not call activities or yield to the scheduler. Buffer incoming signals and process them in the main workflow loop.
Event Accumulator deterministic workflow ID
Use a deterministic workflow ID per bucket key. Encode the group key and, optionally, an accumulation period (for example, accumulator-order-123-2026-05-14) to control when a new window starts. If the workflow ID is not derived deterministically from the group key, multiple accumulator instances are created for the same group, splitting the batch.
Event Accumulator deduplication key requirement
Include a deduplication key in every signal payload. At-least-once delivery is common in event streams; without a dedup key, retried events add duplicate entries to the batch. Every signal payload must carry a stable, unique key.
Event Accumulator inactivity timeout sizing
Size the inactivity timeout to your domain's quiet period. The timeout should reflect how long you are confident no more events will arrive for this batch. Tune it based on observed producer behavior, not an arbitrary constant. A timeout set too short causes the workflow to process a partial batch while more events are still in flight, forcing producers to re-send unprocessed events.
Event Accumulator flush signal
Add a flush signal for testing and operational runbooks. An explicit flush signal lets you trigger early batch processing without waiting for the timeout, which is useful for end-to-end tests and manual intervention.
Event Accumulator signal rate limit
Keep producer signal rate below 5/sec per workflow instance. Each signal briefly locks the workflow execution. A sustained rate above roughly 5 signals/second causes workflow task backlog, limits throughput, and can eventually prevent Continue-As-New from completing. If your producer rate is higher, partition by a finer-grained key so each accumulator workflow receives a manageable share of the total signal volume.
Event Accumulator Temporal Cloud signal limit
Account for the 10,000-signal-per-run limit on Temporal Cloud. A single workflow run in Temporal Cloud can receive at most 10,000 signals. If your accumulation window is long and producers are active, ensure your Continue-As-New trigger fires well before the per-run signal count reaches this limit.