Workflow Id
A Workflow Id is a customizable, application-level identifier for a Workflow Execution that is unique to an Open Workflow Execution within a Namespace.
workflows
70 notes in this subject, read out of this brain and free to use. This is page 1 of 2.
Run Id
A Run Id is a globally unique, platform-level identifier for a Workflow Execution.
Workflow Definition
A Workflow Definition is the code that defines the constraints of a Workflow Execution.
Workflow Execution
A Temporal Workflow Execution is a durable, scalable, reliable, and reactive function execution. It is the main unit of execution of a Temporal Application.
Workflow Type
A Workflow Type is a name that maps to a Workflow Definition.
Event History
An Event History is an append-only log of Events that represents the full state a Workflow Execution.
Event
Events are created by a Temporal Service in response to external occurrences and Commands generated by a Workflow Execution.
Workflow Task
A Workflow Task is a Task that contains the context needed to make progress with a Workflow Execution.
Workflow Task Execution
A Workflow Task Execution occurs when a Worker picks up a Workflow Task and uses it to make progress on the execution of a Workflow Definition.
Workflow Id Reuse Policy
A Workflow Id Reuse Policy determines whether a Workflow Execution is allowed to spawn with a particular Workflow Id, if that Workflow Id has been used with a previous, and now Closed, Workflow Execution.
Workflow Id Conflict Policy
A Workflow Id Conflict Policy determines how to resolve the conflict when spawning a new Workflow Execution with a particular Workflow Id that is used by an Open Workflow Execution already.
Continue-As-New
Continue-As-New is the mechanism by which all relevant state is passed to a new Workflow Execution with a fresh Event History.
Memo
A Memo is a non-indexed user-supplied set of Workflow Execution metadata that is returned when you describe or list Workflow Executions.
State Transition
A State Transition is a unit of progress by a Workflow Execution.
Side Effect
A Side Effect is a way to execute a short, non-deterministic code snippet, such as generating a UUID, that executes the provided function once and records its result into the Workflow Execution Event History.
Durable Execution
Durable Execution in the context of Temporal refers to the ability of a Workflow Execution to maintain its state and progress even in the face of failures, crashes, or server outages.
Workflow Execution Timeout
A Workflow Execution Timeout is the maximum time that a Workflow Execution can be executing (have an Open status) including retries and any usage of Continue As New.
Workflow Run Timeout
A Workflow Run Timeout is the maximum amount of time that a single Workflow Run is restricted to.
Workflow Task Timeout
A Workflow Task Timeout is the maximum amount of time that the Temporal Server will wait for a Worker to start processing a Workflow Task after the Task has been pulled from the Task Queue.
Timer
Temporal SDKs offer Timer APIs so that Workflow Executions are deterministic in their handling of time values.
Delay Workflow Execution
Start Delay determines the amount of time to wait before initiating a Workflow Execution. If the Workflow receives a Signal-With-Start or Update-With-Start during the delay, it dispatches a Workflow Task and the remaining delay is bypassed.
Child Workflow
A Child Workflow Execution is a Workflow Execution that is spawned from within another Workflow.
Reset
A Reset terminates a Workflow Execution, removes the progress in the Event History up to the reset point, and then creates a new Workflow Execution with the same Workflow Type and Id to continue.
Retry Policy
A Retry Policy is a collection of attributes that instructs the Temporal Server how to retry a failure of a Workflow Execution or an Activity Task Execution.
Temporal Cron Job
A Temporal Cron Job is the series of Workflow Executions that occur when a Cron Schedule is provided in the call to spawn a Workflow Execution.
Schedule
A Schedule enables the scheduling of Workflow Executions.
Signal
A Signal is an asynchronous request to a Workflow Execution.
Signal-With-Start
Signal-With-Start starts and Signals a Workflow Execution, or just Signals it if it already exists.
Query
A Query is a synchronous operation that is used to report the state of a Workflow Execution.
Update
An Update is a request to and a response from Workflow Execution.
Dynamic Handler
Dynamic Handlers are Workflows, Activities, Signals, or Queries that are unnamed and invoked when no other named handler matches the call from the Server at runtime.
Distributed lock pattern: design overview
Temporal Workflow IDs can be used to build a durable distributed lock for shared resources without requiring external databases, cache layers, or central limiter services. Each held permit or lock is represented as its own short-lived child Workflow. The child Workflow's ID encapsulates the resource and slot, such as 'permit:gpu-pool:gpu-2'. Temporal does not allow two running Workflows with the same ID, making permit acquisition an atomic operation. The Workflow releases on a Signal pinned to the specific run, and if no release Signal is received, a per-permit lease timeout handles orphan recovery.
Distributed lock use cases and limitations
The distributed lock pattern is best suited for small, fixed pools of scarce resources where each permit is held for minutes, not milliseconds, such as GPU devices, lab hardware, and tenant migrations. The latency may be too high for general-purpose high-throughput mutex or rate limiter use cases. Lock acquisition is not first-in-first-out (FIFO); when a slot frees, whomever claims it first wins. Capacity changes apply only to acquires after the change; already-held permits run to release or lease expiry on their original capacity.
ParentClosePolicy.TERMINATE for orphan recovery
The default ParentClosePolicy.TERMINATE is the fastest orphan recovery mechanism. When a parent Workflow closes for any reason without releasing, Temporal terminates the permit child immediately and the slot frees right away. This prevents leaking slots even if the permit Workflow never runs or the parent crashes.
Three-layer orphan recovery in distributed lock
Three mechanisms work together to bound slot occupancy and handle orphan permits. First, the default ParentClosePolicy.TERMINATE terminates the permit child immediately when the parent Workflow closes without releasing. Second, an in-workflow timer using wait_condition(timeout=lease) covers cases where the parent is still alive but the holder is silent, allowing the permit Workflow to exit cleanly. Third, the execution_timeout=lease server-enforced backstop ticks from Workflow creation regardless of whether a Worker picks it up, preventing slots from being leaked if the permit Workflow never runs.
Required conceptual knowledge for distributed lock implementation
Understanding the following concepts is required: Temporal Workflows and Activities, child Workflows and parent close policies, Signals and Queries, and Workflow determinism constraints.
Fencing tokens for resource corruption prevention
Lease expiry is a recovery mechanism but not proof that the old permit holder stopped using the external resource. For resources that can be corrupted by concurrent access overlap, pass the permit run ID as a fencing token to the Activity or downstream system and reject stale holders there. This prevents two permit holders from using the same resource during the brief window between lease expiry and the holder's awareness of the expiry.
PermitSlotWorkflow implementation
PermitSlotWorkflow is a child Workflow that accepts PermitSlotInput containing resource name, slot name, and lease duration. It accepts a 'release' Signal that sets an internal flag. The Workflow's run method calls workflow.wait_condition(lambda: self._released, timeout=timedelta(seconds=input.lease_seconds)). If the Signal is received, it returns 'released'. If the timeout expires, it returns 'lease_expired' and logs a warning. The Workflow ID is not set by the Workflow itself but by the caller via start_child_workflow(id=...).
Semaphore helper class usage pattern
The Semaphore is a thin async context manager (not a Workflow itself) that wraps workflow.start_child_workflow. Construct it with a fixed capacity; Semaphore auto-generates slot names '0' through 'N-1'. Use it as: sem = Semaphore('resource-name', capacity=N); async with sem.acquire(lease=timedelta(...)) as slot: ... yield slot ... The context manager yields the name of the acquired slot and sends a release Signal on exit. The yielded slot is a string that Activities can use directly or convert with int(slot) for numeric indexing.
Semaphore slot shuffling with workflow.random()
The Semaphore shuffles its slot list using workflow.random() so Workflows always probe in a random order. The shuffle is deterministic across replay (satisfying Workflow determinism requirements) and uniform across callers. This spreads contention across the pool rather than concentrating on slot 0. Each acquire attempt reshuffles the order before trying to start child Workflows.
ChildWorkflowHandle pinning in Semaphore
The Semaphore keeps the ChildWorkflowHandle returned by start_child_workflow and signals release on that handle, not by ID. This pins the release Signal to the specific run started at acquire time. Without this pinning, a late release Signal from a holder whose lease already expired could revoke a subsequent acquirer's permit.
Semaphore backoff strategy for slot contention
When all slots are busy and every attempt to start a child Workflow fails with WorkflowAlreadyStartedError, the Semaphore calls workflow.sleep(backoff) before retrying the full slot list. The default backoff is 5 seconds and the default max backoff is 1 minute. These are conservative starting points that should be adjusted based on the resource and typical lease duration.
permit_workflow_id function
The permit_workflow_id(resource: str, slot: str) -> str function produces Workflow IDs in the format 'permit:{resource}:{slot}', for example 'permit:gpu-pool:2'. This string becomes the lock primitive; Temporal rejects any second attempt to start a Workflow with the same ID while the first is running.
Configuration module structure for distributed lock
The config module defines: TEMPORAL_TASK_QUEUE (default 'distributed-lock-tq'), DEFAULT_LEASE (10 minutes), DEFAULT_BACKOFF (5 seconds), DEFAULT_MAX_BACKOFF (1 minute), and PERMIT_WORKFLOW_ID_PREFIX (default 'permit'). It provides permit_workflow_id(resource, slot) function and connect_temporal_client() that uses temporalio.envconfig to read from ~/.config/temporalio/temporal.toml with environment variable overrides (TEMPORAL_ADDRESS, TEMPORAL_NAMESPACE, TEMPORAL_API_KEY, TEMPORAL_TLS_*).
Two application Workflow examples using Semaphore
Example 1 (ThrottledGateWorkflow): Cap concurrent holders without caring about slot identity. sem = Semaphore('app-gate', capacity=4); async with sem.acquire(lease=timedelta(minutes=10)) as slot: return await workflow.execute_activity('do_gated_work', args=[job_id, slot], ...). Example 2 (GpuTrainingWorkflow): Map yielded slot to a specific resource. sem = Semaphore('gpu-pool', capacity=4); async with sem.acquire(lease=timedelta(minutes=45)) as slot: return await workflow.execute_activity('run_training', args=[model_id, slot], ...). In both cases, convert slot string to int(slot) if the resource is indexed numerically.
Pause workflow on permanent failure using condition() to suspend without consuming resources
When an Activity throws a non-retryable error, catch it and call await condition(() => retryRequested) to suspend the Workflow. This pauses execution without consuming Worker capacity or in-memory cache. The Workflow's execution state (local variables, call stacks, pending timers) is persisted to the Temporal Cluster, and the in-memory execution can be evicted to make room for other Workflows. The Workflow resumes from where it left off when the condition becomes true, triggered by a Signal handler setting retryRequested = true. This pattern allows thousands of loan applications to sit in a PENDING_FIX state simultaneously without consuming Worker resources.
recoverableStep pattern: wrap activities in while(true) loop with pause-and-resume
Implement a recoverableStep helper function that wraps Activity calls in a while(true) loop. On Activity success, return the result and exit the loop. On ActivityFailure, check if the error is actually retryable (filter out non-ActivityFailure exceptions and isCancellation() errors by re-throwing them), update Search Attributes to PENDING_FIX with the failed Activity name and error message, then await condition(() => retryRequested) to suspend. When Signal handler wakes the condition by setting retryRequested = true, the loop continues and retries the same Activity with potentially patched data. This pattern centralizes the pause-and-resume logic for reuse across multiple Activity calls.
Example: recoverableStep helper implementing pause-and-resume on permanent failure
```typescript const recoverableStep = async <T>( activityName: string, fn: () => Promise<T> ): Promise<T> => { while (true) { try { const result = await fn(); return result; } catch (e) { // Re-throw non-ActivityFailure exceptions (Workflow-side bugs) if (!(e instanceof ActivityFailure)) throw e; // Re-throw cancellation to unwind cleanly if (isCancellation(e)) throw e; // Extract error message from wrapped ActivityFailure const message = e.cause?.message || e.message || String(e); log.warn(`Activity ${activityName} failed: ${message}`); // Update Search Attributes to advertise PENDING_FIX state updateStatus('PENDING_FIX', activityName, message); retryRequested = false; // Suspend without consuming resources until Signal wakes condition await condition(() => retryRequested); // Resume after fix updateStatus('STARTED', '', ''); log.info(`Retrying activity ${activityName} after fix`); } } }; ``` This pattern centralizes pause-and-resume logic for reuse across multiple Activity calls in the pipeline.
Web service endpoints: start Workflows, query state, list blocked, send fixes
Implement web service endpoints for: (1) POST /api/workflows — start new Workflow with LoanApplication, set initial Search Attributes to STARTED. (2) GET /api/workflows — list all Workflows with TaskQueue matching 'recoverable-activity' and ExecutionStatus != 'Terminated', return Search Attributes from Visibility store and Query full state for RUNNING Workflows. (3) GET /api/workflows/search?failedActivity=X&status=Y — query by Search Attributes to filter for specific failure types, returns matching Workflow IDs for routing. (4) POST /api/workflows/{workflowId}/fix — send corrective Signal with field name and new value, Workflow wakes and retries failed Activity. These endpoints power operations dashboards and automated resolution agents.
Data models for loan processing: LoanApplication, LoanStatus, FixEntry, LoanState, RetryUpdate
LoanApplication: applicationId (string), applicantName (string), ssn (string), employerName (string), annualIncome (number), propertyAddress (string), propertyId (string), loanAmount (number), downPayment (number). LoanStatus: type union of 'STARTED' | 'INCOME_VERIFIED' | 'CREDIT_CHECKED' | 'APPRAISAL_ORDERED' | 'TITLE_SEARCHED' | 'UNDERWRITTEN' | 'CLOSED' | 'PENDING_FIX' | 'FAILED'. FixEntry: {activity, field, oldValue, newValue, error, id?} records each correction. LoanState: {status, failedActivity, failureMessage, completedActivities[], fixHistory[], application} is the complete queryable state returned by the Query handler. RetryUpdate: {key?, value?, id?} is the Signal payload containing the field to correct and its new value.
Prerequisites for this pattern: TypeScript SDK v1.13.0+, Node.js v18+, Temporal CLI v1.6.1+
To implement this recovery-without-restart pattern, you must have: (1) Temporal TypeScript SDK version 1.13.0 or later. (2) Node.js version 18 or later. (3) Temporal CLI version 1.6.1 or later. (4) A running Temporal Cluster (local dev server or Temporal Cloud) with permissions to create custom Search Attributes and start Workflows. Create Search Attributes by running: temporal operator search-attribute create --name LoanStatus --type Keyword and temporal operator search-attribute create --name FailedActivity --type Keyword.
Start Workflow with initial Search Attributes to enable visibility from startup
When starting a Workflow, immediately set initial Search Attributes so the Workflow is queryable from the start. Use the client.workflow.start() typedSearchAttributes option to pre-populate LoanStatus = 'STARTED' and FailedActivity = ''. Example: await client.workflow.start(homeLoanWorkflow, { taskQueue: 'recoverable-activity', workflowId: application.applicationId, args: [application], typedSearchAttributes: [ { key: LoanStatusKey, value: 'STARTED' }, { key: FailedActivityKey, value: '' }, ], });
Completed Activities list tracks progress and prevents re-execution after resumption
Maintain a completedActivities array in the Workflow state. After each recoverableStep() successfully returns, immediately push the activity name to completedActivities. When unwinding via saga compensation, skip compensations for Activities not in completedActivities (the Activity never fully executed, so there is no external state to undo). This list also enables the Query response to show which steps have already completed, providing operators with visibility into progress and context for which activities remain to be executed.
Workflow waits durably without consuming compute
When a Workflow calls workflow.wait_condition(), the Worker returns the current task to the Temporal Server and becomes idle, consuming no compute. The Workflow's state is persisted and suspended on the Server. When a Signal arrives or a timeout fires, the Server schedules a new Workflow Task, the Worker replays the Event History to reconstruct the Workflow's state, and execution resumes from the wait_condition call. This mechanism works identically whether the wait is five seconds or five months—durable timers are persisted to the Server's database and survive Worker restarts, deployments, and infrastructure migrations.
Event History limits and Continue-As-New
Temporal's Event History has a hard limit of 51,200 events or 50 MB. The Server emits warnings at approximately 10,240 events or 10 MB; hitting either hard limit terminates the Workflow. For approval workflows: Workflow start (baseline) generates 3 events. Single Activity execution generates ~8 events. Timer (asyncio.sleep / workflow.sleep) generates ~4 events. Signal received generates ~4 events. Query received generates 0 events (not recorded). Continue-As-New generates 1 event on the old run; fresh history on the new run. A single approval with reminders and escalation generates roughly 40–80 events—well within limits. If the process supports resubmission loops or runs for months, use workflow.info().is_continue_as_new_suggested() to detect when the Server recommends a Continue-As-New, and act on it at a safe boundary.
Timer-based enforcement of SLAs
Durable timers fire whether or not Workers restart. In the approval pattern, timers are implemented using asyncio.sleep() or workflow.sleep() within workflow.wait_condition() calls with a timeout parameter. When the timeout expires, a TimeoutError is raised, allowing the Workflow to trigger automatic escalation to a backup approver or auto-reject the document. The SLA timeout duration is specified in seconds and is tracked by the Server's database, surviving infrastructure failures.
Continue-As-New for Event History management
When workflow.info().is_continue_as_new_suggested() returns True, indicating the Server recommends resetting the Event History, the Workflow should call workflow.continue_as_new() at a safe boundary. In the approval pattern, Continue-As-New is checked: after storing the document initially; before looping back to await the next decision in a resubmission cycle. When continue_as_new is called with the current ApprovalState, a new Workflow Run is started with the same Workflow Id and input, carrying forward all mutable state while discarding the Event History and receiving a fresh history limit.
Workflow state persistence across restarts
The Workflow's mutable state is maintained in the ApprovalState dataclass, which includes: the original DocumentSubmission, the current DocumentStatus, any ApprovalDecision, the growing audit_trail, resubmission_count, and document_stored flag. When the Workflow calls continue_as_new(), the entire ApprovalState is serialized and passed as input to the new Workflow Run, ensuring that all context (audit entries, resubmission history, decision records) is preserved across Event History resets and infrastructure restarts.
Timeout parameters for approval workflow waiting
The approval Workflow uses asyncio-style timeout management: workflow.wait_condition(lambda: condition, timeout=timedelta(seconds=sla_seconds)) to wait for either a Signal or a timer. When the timeout expires, asyncio.TimeoutError is raised. The Workflow catches this exception and proceeds to escalation or auto-rejection logic. Separate timeouts are used: sla_seconds for the primary approver (default 48 hours in the example), another sla_seconds window for escalation contact, and resubmission_timeout_seconds (default 7 days) for waiting on resubmitted documents.
Async task cancellation pattern in Workflows
When a decision arrives before the SLA timer fires, the reminder task must be cancelled. The pattern: reminder_task = asyncio.create_task(...); await workflow.wait_condition(..., timeout=sla_timeout); reminder_task.cancel(); try: await reminder_task except asyncio.CancelledError: pass. This ensures reminders stop sending as soon as a decision arrives, preventing unnecessary notifications and Activity executions after the decision is made.