Saga pattern: register compensations BEFORE forward Activity execution for partial failure handling
For Activities that produce external side effects (appraisal bookings, credit inquiries, title holds, lending capacity reservations, loan disbursements), register compensating Activities BEFORE executing the forward Activity. This handles the case where an Activity partially succeeds: the Worker POSTs a booking to an external vendor, the vendor records it and reserves a fee, then the response is lost to a network blip. The Activity throws because it never received the response, yet the booking exists on the vendor side. If compensation were only registered after successful execution, that booking would be orphaned. Pre-registering guarantees the compensation runs during rollback either way; idempotency makes it a safe no-op when the side effect never actually landed.
Saga compensation unwinds in LIFO order: last forward step first to undo
When the forward pipeline aborts (either via explicit cancelApplication Signal or RollbackRequired ApplicationFailure), unwind the compensation stack in LIFO (Last-In-First-Out) order. In TypeScript, achieve this by pushing compensations onto the front of an array with unshift() and iterating forward, or by iterating a push()-built array in reverse. The last step to touch external state is the first to undo. For example, if the forward pipeline executed in order [verifyIncome, runCreditCheck, orderAppraisal, performTitleSearch, underwrite, closeLoan] and then fails at closeLoan, compensations run as [releaseLoan, releaseUnderwriting, releaseTitleHold, cancelAppraisal, withdrawCreditInquiry, verifyIncome-nocompensation].
Compensations must be idempotent to safely handle partial side effects
Write every compensation Activity to be idempotent: safe to call multiple times on already-completed or already-undone states. For example, withdrawCreditInquiry simply resubmits the withdrawal request; the credit bureau accepts duplicates without error. cancelAppraisal checks for an existing booking before issuing the cancellation, allowing multiple calls without confusion. releaseTitleHold is safe to call on an already-released hold. This idempotency is critical because the 'register before execution' discipline means compensations may run for forward Activities that never actually produced their side effects, and it ensures that saga rollbacks are safe even in the presence of retries or duplicate Signal delivery.
Compensation failures use same recoverable wrapper to pause with ROLLBACK_PENDING_FIX
Run each compensation Activity through the same recoverableStep wrapper as forward Activities. If a compensation fails—vendor API goes down, external system rejects the request—the Workflow pauses with status ROLLBACK_PENDING_FIX instead of crashing or leaving the pipeline half-unwound. An operator can then patch the corrected state and send a Signal to retry the stuck compensation, ensuring the rollback eventually completes. This pattern prevents partial unwinding: either the entire saga completes forward or fully reverses, never in an intermediate state.
Skip compensation for read-only Activities like income verification and credit checks
Not every forward Activity requires a compensation. verifyIncome is a read-only lookup against the employer verification database—there is no external state to undo, so no compensation is needed. However, runCreditCheck records a hard inquiry that lowers the applicant's score, so its compensation submits a withdrawal request to the bureau. Use judgment: compensate for Activities that produce external side effects or state changes, skip compensation for Activities that only read or validate data.
Escalation workflow triggered by SLA expiration
When the SLA timer expires with no response from the primary approver, the Workflow automatically escalates if an escalation email is configured. The escalation process: records an SLA_EXPIRED audit entry; sends an escalation notification to the backup contact; waits for a response from the escalation contact with a fresh SLA timeout. If the escalation contact also times out and no escalation email was configured, the Workflow auto-rejects with a TIMED_OUT status. If both primary and escalation approvers time out, the decision is auto-rejected with detailed reasoning.
Resubmission handling after changes requested
When an approver requests changes, the Workflow: sets status to REJECTED; sends a notification to the submitter with the requested changes; waits for resubmission using the resubmit_document Update handler with a configurable timeout (default 7 days). If the submitter resubmits within the timeout, the resubmission_count increments and the Workflow loops back to send a fresh approval request. If the resubmission window expires or max resubmissions (default 3) are exceeded, the document is permanently rejected. The Update handler prevents duplicate resubmissions by tracking processed update IDs and returning a duplicate flag.
Reminder implementation using concurrent tasks
Reminders are sent periodically while waiting for an approval decision. The _send_reminders coroutine runs concurrently with the decision wait using asyncio.create_task(). Each reminder after a configurable interval (default 24 hours) is sent via the send_notification Activity with a unique idempotency key including the reminder count. The reminder loop continues up to max_reminders (default 3) and is cancelled when a decision arrives or the SLA expires. Each reminder is recorded in the audit trail.
Storing document before approval request
In the approval Workflow, the document is persisted via the store_document Activity before sending the approval request. This ensures the document is available in external storage (not just the Workflow's Event History, which uses the Claim Check pattern). The Activity is only executed if self._state.document_stored is False, preventing re-execution on replay. The storage reference is recorded in the audit trail. This pattern separates the Workflow's durable state (in Event History) from the actual document content (in external storage).
Temporary rate limit increases pattern overview
This pattern demonstrates how to dynamically provision and automatically deprovision Temporal Cloud capacity. A parent Workflow executes an Activity to raise the capacity limit, then starts an asynchronous Child Workflow with an abandon policy. The parent Workflow completes to unblock the client, while the Child Workflow waits for a designated duration before executing an Activity to revert the capacity to its original limit. This approach automates capacity management, guarantees cleanup operations using durable Timers, and unblocksclient requests while long-running deprovisioning tasks continue in the background.