event-history/replay

44 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Event History replay enables Workflow state reconstruction

When a Worker receives a Workflow Task, the SDK replays the Event History to reconstruct logical Workflow state before running the Workflow function/method. This replay mechanism allows Workflows to safely resume and continue even if Worker processes or machines fail.

Event History enables recovery from Worker crashes

When a Worker crashes, the Worker uses the Event History to replay the code and recreate the state of the Workflow Execution to what it was immediately before the crash. The Workflow then resumes progress from the point of failure as if the failure never occurred.

Workflow Execution recovery through History Replay in four phases

When a Worker crashes during a Workflow Execution, recovery occurs in four phases: Original execution where the Client starts the Workflow and Commands become Events. Worker crash where the Worker dies partway through a Workflow Task, causing a WorkflowTaskTimedOut Event after the default 10-second Workflow Task Timeout. History Replay where a new Worker requests the Event History and re-executes the Workflow code with the original input, matching Commands against the Event History so Activities don't run again but results are used from ActivityTaskCompleted Events. Execution resumes where past the crash point, the Worker issues Commands for real again until completion, resulting in identical execution to one that never crashed.

Workflow code to Commands mapping in Go

Four statements in a Go Workflow produce Commands sent to the Temporal Service: workflow.ExecuteActivity() produces ScheduleActivityTask, workflow.Sleep() produces StartTimer, and return statements produce CompleteWorkflowExecution. All other operations like setting timeouts, calculations, and struct population run locally in the Worker without contacting the Temporal Service.

Worker crash recovery phases in Temporal

When a Worker crashes during a Workflow Task, recovery proceeds in four phases: (1) Original execution where Commands become Events; (2) Worker crash occurs, Workflow Task Timeout elapses (default 10 seconds), Temporal Service records WorkflowTaskTimedOut and schedules a new Workflow Task; (3) History Replay where the Worker re-executes Workflow code against the Event History, matching generated Commands instead of issuing them, using results from previous Activity completions; (4) Execution resumes past the crash point where no matching Events exist, so new Commands are issued for real until completion, producing identical results to a non-crashed execution.

Event History enables durable execution through replay

The Event History is a durable log of Events persisted in the Temporal Service's database, surviving even Service crashes. In case of Worker failure, Temporal uses the Event History to replay the Workflow code with the original input stored in the WorkflowExecutionStarted Event. During replay, Commands generated are matched against the Event History instead of being issued again, so Activities do not re-execute and use cached results from ActivityTaskCompleted Events.

Non-determinism example: random number in Workflow

A Workflow that uses a random number generator violates determinism. If during original execution a random number is 84 causing a Timer to be scheduled (producing TimerStarted Event), but during replay the random number is 14 causing the Timer to be skipped and ScheduleActivityTask to be produced instead, this mismatch causes replay to fail because the generated Command sequence no longer matches the Event History.

Workflow code maps to Commands in Python SDK

In the Python SDK, four types of statements in Workflow Definitions produce Commands: workflow.execute_activity_method() produces ScheduleActivityTask, asyncio.sleep() produces StartTimer, and return statements produce CompleteWorkflowExecution. All other internal steps like totaling prices, evaluating distances, and populating variables run in the Worker without contacting the Temporal Service.

Non-determinism causes Workflow Task Failures

Non-deterministic failures in Workflows do not fail the Workflow Execution by default. A non-determinism mismatch is treated as a Workflow Task Failure, which is a transient failure that retries automatically. Users can fix the source of non-determinism and restart Workers for recovery. Versioning can also be used to address non-determinism errors.

History Replay provides durable execution recovery

When a Worker crashes partway through a Workflow Task, the Temporal Service waits for the Workflow Task Timeout (10 seconds by default), records a WorkflowTaskTimedOut Event, and schedules a new Workflow Task. A new Worker then requests the Event History and re-executes the Workflow code with the original input stored in the WorkflowExecutionStarted Event. During Replay, Commands generated by the Worker are matched against the Event History instead of being issued to the Temporal Service, preventing Activities from running again. The Worker uses results stored in ActivityTaskCompleted Events. Once Replay reaches the point of the original crash, past events no longer match, so the Worker issues new Commands for real until the Workflow completes, resulting in an execution identical to one that never crashed.

Determinism requirement for Workflows

A Workflow is deterministic if every execution of its Workflow Definition produces the same Commands in the same sequence given the same input. Workflows must be deterministic for Replay to work, because during Replay the Worker re-executes the Workflow code and the resulting Commands are compared against the Event History. When Commands mismatch due to non-determinism, Replay cannot continue.

How Workers process Workflow Tasks using replay

When a Worker picks up a Workflow Task, it replays the entire Workflow Execution from the beginning using the Event History. The Worker receives the Workflow Task containing the complete Event History for the Workflow Execution, replays the Workflow code from the start using the Event History to recreate the Workflow's state, and during replay, previously executed operations like Activity calls or Timers return their results immediately from the Event History instead of executing again. The replay continues until the Worker reaches a point where it needs to make new progress. The Workflow code executes any new decisions and generates Commands, which the Worker sends back to the Temporal Service, completing the Workflow Task. The Temporal Service persists the Commands as new Events in the Event History.

Workflow Task replay enables fault tolerance

The replay mechanism makes Temporal Workflows durable and fault-tolerant. If a Worker crashes mid-execution, another Worker can pick up the Workflow Task and replay the entire history to reconstruct the exact state before continuing.

patched() function behavior when not replaying - first call

When a Workflow execution is not replaying and encounters a call to patched(), if the patch ID is not in the event history, the execution adds a marker to the event history, upserts a search attribute, and returns true.

patched() function behavior when not replaying - subsequent calls

When a Workflow execution is not replaying and encounters a call to patched(), if the patch ID is already in the event history, the execution doesn't modify the history and returns true.

patched() function behavior during replay with marker at or before current location

If the execution is replaying and has a call to patched(), and the event history has a marker from the same patch ID in the same place, it writes a marker to the replay event history and returns true. If the event history has a marker with that patch ID earlier in the history, it returns true and does not modify the replay event history.

patched() function behavior during replay with marker after current location

If the Event History's Marker Event is after the current execution point during replay, the execution will encounter the new patch before the original. The execution will attempt to write the marker to the replay event history but will throw a non-deterministic exception because the replay and original event histories don't match.

patched() function behavior during replay with no marker for that patch ID

During a Replay, if there is no marker for a given patch ID, the execution will return false and will not add a marker to the event history. All future calls to patched() with that ID will also return false, even after replay completes and is running new code.

patched() unexpected behavior caveat - marker in future history

If the execution hits a call to patched() but that patch ID isn't at or before that point in the event history, the event history after the current execution location matters. If that patch ID exists later in the history, you get a non-determinism error. If the patch doesn't exist later, you don't get a non-determinism error and the call returns false.

patched() unexpected behavior caveat - missing patch ID in replay

If the execution hits a call to patched() with an ID that doesn't exist in the history, then not only will it return false in that occurrence, but it will also return false if the execution surpasses the Replay threshold and is running new code.

Workflow execution behavior after redeployment during replay

If you deploy new code while Workflows are executing, any Workflows that were in the middle of executing will Replay up to the point they were at when the Worker was shut down. When they replay, they will not follow the patched() branches in the code. After replaying to the point before deployment, they will either use new code if there was no call to patched() in the replay code, or run the non-patched code during and after replay if there was a call to patched() in the replay code.

Workflow does not always run newest code

The Workflow only runs the newest code if not replaying or if replay is surpassed and there hasn't been a call to patched() (with that ID) throughout the replay. This means that if the future patched code depends on earlier patched code, it won't use the new code but the old code instead.

Example of non-determinism from code changes

If a Workflow Definition initially defines sequence: Start Timer, Spawn Activity, Complete; and then is changed to: Spawn Activity, Start Timer, Complete; when the Workflow Function re-executes after the Timer fires, the first Command would be ScheduleActivityTask instead of the expected TimerStarted Event. The Workflow Execution would fail with a nondeterminism error.

Workflow Definition must be deterministic to support replay

Workflow code must be deterministic, meaning that any time the Workflow code is executed it must make the same Workflow API calls in the same sequence, given the same input. This is required to support replay. To handle non-deterministic operations like API calls, LLM/AI invocations, database queries, and other external interactions, put them in Activities.

Safe Workflow Definition changes that do not cause non-determinism

The following changes to Workflow Definition are safe and do not cause non-determinism: changing the duration of Timers (except in Java, Python, Go where changing to or from 0 is unsafe, and in .NET where changing to or from -1 is unsafe); changing arguments to Activity Options, Child Workflow Options, or Signal calls; adding a Signal Handler for a Signal Type that has not been sent to the Workflow Execution.

Unsafe Workflow API changes that produce Commands

The following Workflow API calls produce Commands and must not be reordered, added, or removed without proper Versioning techniques: starting or cancelling a Timer; scheduling or cancelling Activity Executions; starting or cancelling Child Workflow executions; signalling or cancelling signals to external Workflow Executions; scheduling or cancelling Nexus operations; ending the Workflow Execution in any way; Patched or GetVersion calls; upserting Workflow Search Attributes or Memos; running a SideEffect or MutableSideEffect.

Commands and Event History matching during replay

When Workflow code replays, the Commands that are emitted are compared with the existing Event History. If a corresponding Event already exists within the Event History that matches that Command, then the Execution progresses. If a generated Command does not match what it needs to in the existing Event History, then the Workflow Execution returns a non-deterministic error.

Causes of non-deterministic Commands in Workflow Definition

A Command might be generated out of sequence or the wrong Command might be generated altogether for two reasons: (1) Code changes are made to a Workflow Definition that is in use by a running Workflow Execution, or (2) there is intrinsic non-deterministic logic such as inline random branching.

Intrinsic non-determinism definition and prevention

Intrinsic non-determinism is when a Workflow Function Execution might emit a different sequence of Commands on re-execution, regardless of whether all input parameters are the same. For example, a Workflow Definition cannot have inline logic that branches based off a local time setting or a random number. All operations that do not purely mutate the Workflow Execution's state should occur through a Temporal SDK API so that results are stored in Event History.

What is a Side Effect

A Side Effect is a way to execute a short, non-deterministic code snippet, such as generating a UUID, that executes the provided function once and records its result into the Workflow Execution Event History. A Side Effect does not re-execute upon replay, but instead returns the recorded result. Side Effects are included in the Go, Java, and PHP SDKs, but not in other SDKs. Local Activities fit the same use case and are slightly less resource intensive.

Side Effect failure risk

Do not ever have a Side Effect that could fail, because failure could result in the Side Effect function executing more than once. If there is any chance that the code provided to the Side Effect could fail, use an Activity instead.

Replay: method to resume progress

A Replay is the method by which a Workflow Execution resumes making progress. During a Replay, the Commands that are generated are checked against an existing Event History. Replays are necessary and often happen to give the effect that Workflow Executions are resumable, reliable, and durable. If a failure occurs, the Workflow Execution picks up where the last recorded event occurred in the Event History.

workflow reset from last ContinueAsNew

Example command to reset a workflow from where it last continued as new: temporal workflow reset --workflow-id YourWorkflowId --type LastContinuedAsNew

workflow reset batch restrictions

For batch resets, limit resets to FirstWorkflowTask, LastWorkflowTask, or BuildId. Do not use Workflow IDs, run IDs, or event IDs with batch reset commands.

workflow reset with-workflow-update-options flags

workflow reset with-workflow-update-options subcommand flags: --versioning-override-behavior (string-enum, required): Override the versioning behavior of a Workflow, accepted values: pinned, auto_upgrade; --versioning-override-build-id (string, optional): When overriding to pinned behavior, specifies the Build ID of the version to target; --versioning-override-deployment-name (string, optional): When overriding to pinned behavior, specifies the Deployment Name of the version to target.

workflow show - display event history

The workflow show command shows a Workflow Execution's Event History. When using JSON output (--output json), results can be passed to an SDK to perform a replay.

workflow show flags

workflow show command flags: --detailed (bool, optional): Display events as detailed sections instead of table, does not apply to JSON output; --follow/-f (bool, optional): Follow the Workflow Execution progress in real time, does not apply to JSON output; --reverse (bool, optional): Fetch Event History newest-event-first, cannot be combined with --follow; --run-id/-r (string, optional): Run ID; --workflow-id/-w (string, required): Workflow ID.

workflow reset with-workflow-update-options subcommand

The workflow reset with-workflow-update-options subcommand runs Workflow Update Options atomically after the Workflow is reset. Workflows selected by the reset command are forwarded onto this subcommand.

workflow reset - resume from event history point

The workflow reset command resets a Workflow Execution so it can resume from a point in its Event History without losing its progress up to that point.

workflow reset by event ID example

Example command to reset a workflow to a specific event: temporal workflow reset --workflow-id YourWorkflowId --event-id YourLastEvent

Activity parameters and Workflow Execution Event History impact

All Payload data is recorded in the Workflow Execution Event History. Large Event Histories can affect Worker performance because the entire Event History could be transferred to a Worker Process with a Workflow Task.

SDK Replay capability

SDKs have the ability to replay Workflow Executions, which contributes significantly to the Platform's promised reliability. The SDKs can automatically continue a process from the point of interruption should a failure occur. This capability stems from the SDK's ability to persist each step the program takes.

Replay and non-determinism detection

When the SDK Worker Replays the Workflow code, it uses the Event History as guidance on what to expect. If the Replay encounters an Event that does not match up with what is expected from the code, a non-determinism error gets thrown. If there is alignment, the Worker continues evaluating code.

Workflow replay mechanism restores state from Event History

When a Workflow yields or encounters an error, Temporal brings the Workflow back to the exact same state it was in before the pause occurred by keeping the Event History. When it is time to continue the Workflow, Temporal does not restore memory from a snapshot. Instead, it starts the Workflow code from the beginning, replays the Event History step by step, and uses that history to guide the code back to the exact state as before. The Workflow code is re-run but uses the recorded events instead of redoing work. Temporal does not always have to start from the beginning if the state is cached.

Give your agent this brain