workflow-execution/lifecycle

107 notes in this subject, read out of this brain and free to use. This is page 1 of 2.

Step 15: Client reads result or status from Frontend after Workflow closes

While the Workflow is open, Clients can query it (read-only inspection of state) and send Signals to it. After it is closed, a Client can request the final result or failure by calling the SDK, which talks to Frontend; the Frontend Service reads the necessary data from the History Service/Persistence Layer and returns it.

Step 2: History Service creates Workflow Execution and initializes Event History

The History Service creates a new Workflow Execution in the database and initializes its Event History with WorkflowExecutionStarted and WorkflowTaskScheduled events (to tell a Worker to run Workflow code). It also creates an internal Transfer Task telling the system to put a Workflow Task on the specified Task Queue.

Step 3: Matching Service adds Workflow Task to Task Queue

A background queue processor in the History Service reads the Transfer Task and calls the Matching Service to AddWorkflowTask for that Task Queue. The Task Queue now has one pending Workflow Task for this Workflow Execution.

Step 4: Worker polls for Workflow Task

A Worker process is already polling the Task Queue using PollWorkflowTask via Frontend Service. The Frontend Service asks Matching for a Task. The Matching Service picks the Workflow Task and tells the History Service that it was started. The History Service appends WorkflowTaskStarted to the Event History and returns the Workflow Task (plus history) to the Worker through Frontend.

Step 5: Worker runs Workflow code via replay and replay

The SDK replays the Event History to reconstruct logical Workflow state, then starts the Workflow function/method. The Workflow runs until either it returns a result or until it needs to wait (for an Activity, Timer, Signal, etc.).

Step 6: Workflow code generates Commands sent to Server

When Workflow code calls Temporal APIs (for example, 'execute Activity'), the SDK records Commands like ScheduleActivityTask or StartTimer instead of executing them directly. When the Workflow cannot make more progress, the Worker sends RespondWorkflowTaskCompleted to Frontend, carrying the list of Commands. Then the Frontend forwards this list to History.

Step 7: History Service converts Commands to Events and Tasks

The History appends Events based on the Commands and updates state. For example, for ScheduleActivityTask: WorkflowTaskCompleted, then ActivityTaskScheduled. It then creates new internal Tasks (Transfer/Timer Tasks) and, through the queue processors, calls Matching to add Activity Tasks to Activity Task Queues or later add more Workflow Tasks when needed. At this point, the Workflow is waiting on whatever it asked for (Activities, Timers, etc.).

Step 8: Matching Service adds Activity Task to Activity Task Queue

For each ActivityTaskScheduled Event, a queue processor in History calls Matching to AddActivityTask on the corresponding Activity Task Queue.

Step 1: Client starts Workflow via Frontend Service

The Temporal Client calls StartWorkflowExecution on the Temporal API (gRPC) exposed by the Frontend Service. The request includes the Workflow type (which Workflow function/class to run), input arguments, and the Task Queue name. Frontend forwards the request to the History Service, which owns this Workflow Execution.

Step 9: Worker polls Activity Task and History records start

A Worker is polling the Activity Task Queue using PollActivityTask via Frontend. Frontend asks Matching; Matching selects the Activity Task and notifies History that it has started. History appends ActivityTaskStarted and sets up any Activity timeout timers (for example, schedule-to-close). The Activity Task is then returned to the Worker via Frontend.

Step 10: Worker executes Activity code

The Worker calls the Activity function/method with the inputs from the Task. This code can do I/O, call external services, etc., because it is not replayed the same way as Workflow code.

Step 11a: Activity success path records completion and reschedules Workflow Task

On success, the Worker sends RespondActivityTaskCompleted (with the result) to Frontend; Frontend forwards it to History. The History appends ActivityTaskCompleted (including the result) and WorkflowTaskScheduled (to wake up the Workflow), and adds a Transfer Task to create the next Workflow Task.

Step 11b: Activity failure path records failure and handles retries

On failure, the Worker sends RespondActivityTaskFailed; History appends ActivityTaskFailed and either will append a new ActivityTaskScheduled (for a retry), or leaves the failure to propagate to the Workflow, depending on retry settings.

Step 12: New Workflow Task scheduled after Activity completion

Because of the Activity completion (or other Events like Timers), History has appended Events and scheduled a new Workflow Task. A queue processor in History calls Matching to add that Workflow Task to the Workflow Task Queue.

Step 13: Worker continues Workflow by replaying history and receiving new state

A Worker polls the Workflow Task Queue again (PollWorkflowTask). The Matching Service selects the Task; the History Service appends WorkflowTaskStarted. The Worker receives the updated Event History via Frontend. The SDK replays the Event History, unblocks the waiting Activity or Timer, and the Workflow code continues from this new state.

Step 14: Workflow closes with completion or continues in other ways

The cycle of Workflow Task → Commands → Events → new Tasks repeats as many times as needed. When the Workflow function/method returns a result, the Worker sends RespondWorkflowTaskCompleted with a CompleteWorkflowExecution command. The History Service appends WorkflowTaskCompleted and WorkflowExecutionCompleted and marks the Workflow Execution as closed. Workflows can also close via failure, cancellation, termination, or 'continue-as-new,' but in all cases they move from open to closed and do not reopen.

Workflow Task Timeout maximum value and recommendation

Although a Workflow Task Timeout can be extended up to a maximum value of 120 seconds, it is not recommended to move beyond the default value of 10 seconds.

Workflow Execution Timeout definition and behavior

A Workflow Execution Timeout is the maximum time that a Workflow Execution can be executing with an Open status, including retries and any usage of Continue As New. The default value is infinite. If this timeout is reached, the Workflow Execution changes to a Timed Out status. Temporal generally does not recommend setting Workflow Timeouts because Workflows are designed to be long-running and resilient, and setting a timeout can limit the ability to handle unexpected delays or long-running processes.

Workflow Run Timeout definition and behavior

A Workflow Run is an instance of a specific Workflow Execution. A Workflow Run Timeout restricts the maximum duration of a single Workflow Run. Because this timeout only applies to an individual Workflow Run, it does not include retries or Continue-As-New. If the Workflow Run Timeout is reached, the Workflow Execution will be Timed Out. The default is set to the same value as the Workflow Execution Timeout. This timeout is most commonly used to limit the execution time of a single Temporal Cron Job Execution.

Workflow Task Timeout definition and behavior

A Workflow Task Timeout is the maximum amount of time allowed for a Worker to execute a Workflow Task after the Worker has pulled that Workflow Task from the Task Queue. The default value is 10 seconds. This timeout is primarily available to recognize whether a Worker has gone down so that the Workflow Execution can be recovered on a different Worker. The main reason for increasing the default value is to accommodate a Workflow Execution that has an extensive Workflow Execution History, requiring more than 10 seconds for the Worker to load.

Difference between Workflow Execution Timeout and Workflow Run Timeout

Workflow Execution Timeout covers the entire Workflow Execution including retries and Continue-As-New, while Workflow Run Timeout applies only to a single Workflow Run and does not include retries or Continue-As-New. Due to retries or Continue-As-New, a Workflow Execution may have multiple Workflow Runs that share the same Workflow ID but have unique Run IDs.

Workflows do not execute actions directly

When a Workflow Definition makes an API call to execute an Activity or start a Timer, it does not perform the action directly. Instead, it sends a Command to the Temporal Service.

Default Workflow Task Timeout

The default Workflow Task Timeout is 10 seconds. If a Worker crashes partway through a Workflow Task, the Temporal Service waits for this timeout to elapse before recording a WorkflowTaskTimedOut Event and scheduling a new Workflow Task.

Workflow Execution guarantees

A Temporal Workflow Execution executes a Temporal Workflow Definition (also called a Temporal Workflow Function) exactly once and to completion, whether the code executes for seconds or years, in the presence of arbitrary load and arbitrary failures.

When Workflow Tasks are scheduled

The Temporal Service creates and schedules a new Workflow Task whenever: the Workflow Execution is started, a Signal is sent to the Workflow, an Update is sent to the Workflow, an Activity completes (successfully or with a failure), a Timer fires, a Child Workflow completes, or a Workflow Task fails and needs to be retried. Any event that might affect the Workflow's state triggers a new Workflow Task. The Workflow Task bundles together all new events that have occurred since the last Workflow Task completed.

Workflow Execution versus Workflow Function Execution

A Workflow Execution effectively executes once to completion, while a Workflow Function Execution occurs many times during the life of a Workflow Execution.

Workflow Type definition and scope

A Workflow Type is a name that maps to a Workflow Definition. A single Workflow Type can be instantiated as multiple Workflow Executions. A Workflow Type is scoped by a Task Queue. It is acceptable to have the same Workflow Type name map to different Workflow Definitions if they are using completely different Workers.

Workflow Definition is code that defines workflow constraints and execution flow

A Workflow Definition is the code that defines a Workflow. It is written with a programming language and corresponding Temporal SDK and typically implemented as a function or object method. It encompasses the end-to-end series of steps of a Temporal application. A Workflow Definition is also referred to as a Workflow Function.

Continue-As-New passes state as arguments to new Workflow

The latest relevant state is passed to Continue-As-New as arguments to the new Workflow Execution. These parameters are typically optional and left unset by the original caller of the Workflow.

Temporal signals when to Continue-As-New

Temporal provides a signal to indicate when a Workflow is approaching performance or scalability problems. Developers can check for this signal at spots in their Workflow implementation where they are ready to checkpoint state.

Continue-As-New creates a fresh Event History

Continue-As-New allows a Workflow to checkpoint its state and start a fresh Workflow with a new Event History. The new Workflow Execution has the same Workflow ID but a different Run ID.

Continue-As-New solves Workflow Versioning problems

When a Workflow Execution starts on an older version of code and then begins executing on a newer version, it can hit Workflow Versioning problems. Continue-As-New provides a way to reset and avoid these compatibility issues.

Workflows can repeat Continue-As-New indefinitely

Continue-As-New can be repeated as often as needed, which means a Workflow can run forever. Workflows that do this are often called Entity Workflows because they represent durable objects, not just processes.

Workflow Execution Callback limits

There is a limit to the total number of Workflow Callbacks that may be attached to a single Workflow Execution. This limit is set with the MaxCallbacksPerWorkflow dynamic configuration key.

Dynamic configuration key for Nexus Operations limit

The MaxConcurrentOperations dynamic configuration key controls the maximum number of Nexus Operations in a Workflow Execution.

No limit on concurrent Workflow Executions

There is no limit to the number of concurrent Workflow Executions, but each must abide by the Workflow Execution's Event History limit.

Incomplete operation type limits

Temporal enforces a maximum number of incomplete Activities, Child Workflows, Signals, or Cancellation requests per Workflow Execution. The default limit is 2,000 for each type of operation. Each in-progress operation generates a metadata entry in the Workflow Execution's mutable state, and too many entries cause unstable persistence. When a limit is reached for a type of operation, attempting to start another operation of that type produces a Workflow Task Execution failure that gets retried.

Dynamic configuration keys for incomplete operation limits

The following dynamic configuration keys control the limits for incomplete operations: NumPendingActivitiesLimit, NumPendingChildExecutionsLimit, NumPendingSignalsLimit, and NumPendingCancelRequestsLimit.

Nexus Operation concurrency limit per Workflow

Temporal enforces a maximum number of incomplete Nexus Operation requests per Workflow Execution, with a default of 30 Nexus Operations. Each in-progress Nexus Operation generates a metadata entry in the Workflow Execution's mutable state. When the limit is reached, attempting to start another Nexus operation causes the Workflow Task Execution to fail and get retried.

Start Delay can be triggered with Update

A delayed-start Workflow can be triggered with an Update. This is because a Workflow Task needs to be scheduled to deliver an Update to a Worker. Once a Workflow Task is scheduled, the Workflow is unblocked. Delay start works by not scheduling the first Workflow Task on Workflow creation.

Start Delay interrupted by Signal-With-Start or Update-With-Start

If the Workflow receives a Signal-With-Start or Update-With-Start during the delay, it dispatches a Workflow Task and the remaining delay is bypassed.

Start Delay delays initial Workflow Execution

Start Delay determines the amount of time to wait before initiating a Workflow Execution. This is useful if you have a Workflow you want to schedule out in the future, but only want it to execute once, in comparison to reoccurring Workflows using Schedules.

Start Delay incompatible with Schedules and Cron Jobs

Start Delay Workflow Execution is incompatible with both Schedules and Cron Jobs.

Start Delay not interrupted by regular Signals

If the Workflow receives a Signal during the delay that is not a Signal-With-Start, the Signal does not interrupt the delay, and the Workflow continues to be delayed until the delay expires or a Signal-With-Start is received.

Start Delay only applies to initial Workflow Execution

Start Delay only applies to the initial Workflow Execution and does not affect subsequent executions, such as when the Workflow Continues-as-New.

Workflow Execution status: Open vs Closed

A Workflow Execution can be either Open or Closed. Open means the Workflow Execution is able to make progress. Closed means the Workflow Execution cannot make further progress.

Closed Workflow Execution statuses

A Closed Workflow Execution can have one of six statuses: (1) Cancelled - the Workflow Execution successfully handled a cancellation request, (2) Completed - the Workflow Execution has completed successfully, (3) Continued-As-New - the Workflow Execution Continued-As-New, (4) Failed - the Workflow Execution returned an error and failed, (5) Terminated - the Workflow Execution was terminated, and (6) Timed Out - the Workflow Execution reached a timeout limit.

Workflow Execution definition and main unit of execution

A Temporal Workflow Execution is a durable, reliable, and scalable function execution. It is created by executing a Workflow Definition (the code). The Workflow Execution is the main unit of execution of a Temporal Application.

Durability: absence of imposed time limit

A Workflow Execution is durable because it executes a Temporal Workflow Definition effectively once and to completion whether the code executes for seconds or years. Durability is defined as the absence of an imposed time limit.

Reliability: responsiveness in presence of failure

A Workflow Execution is reliable because it is fully recoverable after a failure. The Temporal Platform ensures the state of the Workflow Execution persists in the face of failures and outages and resumes execution from the latest state. Reliability is defined as responsiveness in the presence of failure.

Scalability: responsiveness in presence of load

A single Workflow Execution is limited in size and throughput but is scalable because it can Continue-As-New in response to load. A Temporal Application is scalable because the Temporal Platform is capable of supporting millions to billions of Workflow Executions executing concurrently. Scalability is defined as responsiveness in the presence of load.

Exclusive access to local state

Each Temporal Workflow Execution has exclusive access to its local state. It executes concurrently to all other Workflow Executions, and communicates with other Workflow Executions through Signals and the environment through Activities.

Two main operations of Workflow Execution

A Workflow Execution does two things: (1) Issue Commands, and (2) Wait on Awaitables (often called Futures).

Awaitables provided by Temporal SDK APIs

A Workflow Execution may only ever block progress on an Awaitable that is provided through a Temporal SDK API. Awaitables are provided when using APIs for: Awaiting (explicit Await APIs), Requesting cancellation of another Workflow Execution, Sending a Signal, Spawning a Child Workflow Execution, Spawning an Activity Execution, and Starting a Timer.

Open Workflow Execution statuses

An Open Workflow Execution has two possible statuses: (1) Running - the Workflow Execution is actively progressing or is waiting on something, and (2) Paused - the Workflow Execution has been Paused and remains Open, but the Temporal Service doesn't dispatch new Workflow Tasks until the Workflow Execution is Unpaused.

Workflow Execution Chain definition

A Workflow Execution Chain is a sequence of Workflow Executions that share the same Workflow Id. Each link in the Chain is often called a Workflow Run. Each Workflow Run in the sequence is connected by one of the following: Continue-As-New, Retries, or Temporal Cron Job.

Workflow Execution Timeout vs Workflow Run Timeout

The Workflow Execution Timeout applies to a Workflow Execution Chain. The Workflow Run Timeout applies to a single Workflow Execution (Workflow Run).

Memo definition and purpose

A Memo is a non-indexed set of Workflow Execution metadata that developers supply at start time or in Workflow code and that is returned when you describe or list Workflow Executions. The primary purpose of using a Memo is to enhance the organization and management of Workflow Executions by allowing developers to add their own metadata, such as notes or descriptions, to a Workflow Execution, which lets you annotate and categorize Workflow Executions based on developer-defined criteria.

Memo limitations and warnings

Memos shouldn't store data that's critical to the execution of a Workflow because: (1) Unlike Workflow inputs, Memos lack type safety, (2) Memos are subject to eventual consistency and may not be immediately available, and (3) Excessive reliance on Memos hides mutable state from the Workflow Execution History. On Temporal Cloud, a Memo is limited to 40 KB.

Starting a Workflow Execution creates WorkflowExecutionStarted event

A request to spawn a Workflow Execution causes the Temporal Service to create the first Event (WorkflowExecutionStarted) in the Workflow Execution Event History. The Temporal Service then creates the first Workflow Task, resulting in the first WorkflowTaskScheduled Event.

Give your agent this brain