Workflow Execution failure event recording
An Activity failure does not directly cause a Workflow Execution failure. If an Activity fails and the error propagates out of the Workflow function without being caught (or is caught and intentionally re-raised as an Application Failure), the Workflow Execution fails. When a Workflow Execution fails, the Temporal Service adds a WorkflowExecutionFailed Event. If the failure was caused by an unhandled Activity error, the activityFailureInfo is attached to that Event.
Activity Task events in Event History
An Activity Execution that completes results in three Events: ActivityTaskScheduled, ActivityTaskStarted, and ActivityTaskCompleted. If an Activity fails and the Retry Policy does not cause it to retry, the Temporal Service adds an ActivityTaskFailed Event that contains the error details. If an Activity times out, an ActivityTaskTimedOut Event is added instead. While an Activity is running, ActivityTaskScheduled is the most recent Event visible for that Activity. The ActivityTaskStarted Event is not written until the Activity Task closes, because the final retry attempt number is not known until then.
Pending Activity Executions visibility
You can view pending Activity Executions in the Web UI's Pending Activities section, which shows the Activity Type, current retry attempt, remaining attempts, and heartbeat information.
Event History enables durability
When an error or failure happens, Temporal recreates state by parsing the Event History and replaying each step. This Event History is the key to making applications reliable and crash-proof, which is why determinism and idempotency are important when creating Workflows.
History Service function
The History Service maintains an ordered Event History in the database, recording everything that happens in each Workflow including when it started, Activities ran, Signals were received, timers fired, and when it completed or failed. The Event History is persisted along with mutable state and internal task queues like timers, transfers, replication, and visibility/indexing.
Event History is a durable log of Workflow Execution
The Event History is a complete and durable log of everything that has happened in the lifecycle of a Workflow Execution. The Temporal Service durably persists Events during Replay to enable workflow recovery.
Commands are mapped to Events
Commands issued by a Worker are mapped to Events which are persisted in the Event History. This mapping allows the system to recover from failures.
Direct and indirect Events from Commands
Commands produce Events in the Event History: ScheduleActivityTask produces the direct Event ActivityTaskScheduled and indirect Events ActivityTaskStarted and ActivityTaskCompleted. StartTimer produces the direct Event TimerStarted and the indirect Event TimerFired.
Event History is persisted and survives crashes
The Event History is a detailed log of Events that occur during a Workflow Execution lifecycle, such as execution of Workflow Tasks or Activity Tasks. Event Histories are persisted to the database used by the Temporal Service and are durable—they survive crashes of the Temporal Service itself.
Commands map to Events with direct and indirect relationships
Commands issued by Workers are converted into Events that build the Event History. Some events are direct results of Commands (like ActivityTaskScheduled from ScheduleActivityTask), while others are indirect results recorded by the Temporal Service (like ActivityTaskStarted and ActivityTaskCompleted). ScheduleActivityTask produces ActivityTaskScheduled directly, then ActivityTaskStarted and ActivityTaskCompleted indirectly. StartTimer produces TimerStarted directly, then TimerFired indirectly.
Command types and their resulting Events
Commands produce Events as follows: ScheduleActivityTask produces ActivityTaskScheduled (direct) and ActivityTaskStarted and ActivityTaskCompleted (indirect); StartTimer produces TimerStarted (direct) and TimerFired (indirect); CompleteWorkflowExecution completes the Workflow.
Workflow Commands map to Events in Event History
Commands issued by Workers are converted into Events that build up the Event History. The Event History is a detailed, durable log persisted to the Temporal Service's database and survives Temporal Service crashes. Direct Events result immediately from Commands: ScheduleActivityTask creates ActivityTaskScheduled, StartTimer creates TimerStarted. Indirect Events occur as side effects of Commands: ScheduleActivityTask also generates ActivityTaskStarted and ActivityTaskCompleted; StartTimer also generates TimerFired.
Workflow code maps to Commands sent to Temporal Service
Workflow statements produce Commands that are sent to the Temporal Service. In a typical workflow, four types of statements generate Commands: awaiting activity execution (ScheduleActivityTask), using sleep/timers (StartTimer), awaiting more activities (ScheduleActivityTask), and returning completion (CompleteWorkflowExecution). Other operations like setting timeouts, declaring variables, evaluating conditions, and populating objects run locally in the Worker without contacting the Temporal Service.
Event History enables state recreation after failures
The Event History is used to recreate a Workflow Execution's state in the case of failure, such as a Worker crash or Temporal Service outage.
Non-retryable Nexus errors add NexusOperationFailed event
When the caller's Nexus Machinery receives a non-retryable error, a NexusOperationFailed event is added to the caller's Event History.
Archival triggers after Workflow Execution closes with randomized delay
When a Workflow Execution closes, Temporal schedules close-processing tasks for both Visibility records and Event History archival. Archival then runs asynchronously after a randomized delay. By default, that delay is up to 5 minutes, controlled by the `history.archivalProcessorArchiveDelay` setting and capped by the Namespace Retention Period.
Archival copies closed Event Histories and Visibility records to blob storage
Archival is a feature that automatically backs up Event Histories and Visibility records from Temporal Service persistence to a custom blob store. This enables Workflow Execution data to persist beyond retention without overwhelming the Temporal Service persistence store.
Closed executions remain in persistence during archival delay
The closed execution stays in Temporal persistence until retention cleanup runs. For some time, the same closed execution can exist in both persistence and archival storage simultaneously.
Archival feature status and limitations
Temporal's Archival feature is considered experimental and not subject to normal versioning and support policy. Archival is not supported when running Temporal through Docker. It is disabled by default when installing the system manually and when deploying through helm charts, but can be enabled in the configuration.
Archival use cases: compliance and debugging
Archival is helpful for compliance and debugging purposes, allowing Workflow Execution data to be retained long-term outside of the main Temporal Service persistence store.
Activity Task event history and retries
For an Activity Execution, the ActivityTaskStarted Event will not show up in the Workflow Execution Event History until the Activity Execution has completed or failed (having exhausted all retries). This is to avoid filling the Event History with noise. Use the Describe API to get a pending Activity Execution's attempt count.
Workflow execution failed with retry state in event history
When a Workflow fails and has a Retry Policy, the failed run ends with WorkflowExecutionFailed, with retryState=IN_PROGRESS and newExecutionRunId set, and the Temporal Service starts a new Workflow Execution. The new Workflow Execution is created immediately, but the first Workflow Task won't be scheduled until the backoff duration is exhausted. That duration is recorded as the first_workflow_task_backoff field on the new run's WorkflowExecutionStartedEventAttributes.
Durable Execution definition and mechanism
Durable Execution in the context of Temporal refers to the ability of a Workflow Execution to maintain its state and progress even in the face of failures, crashes, or server outages. This is achieved through Temporal's use of an Event History, which records the state of a Workflow Execution at each step. If a failure occurs, the Workflow Execution can resume from the last recorded event, ensuring that progress isn't lost.
Updates write to Workflow Event History
Updates, if accepted, will write an entry to the Workflow Event History, whereas Queries never do.
Continue-As-New solves long Event History performance problems
When a Workflow Execution has a long or large Event History from calling many Activities, it may experience performance issues or even exceed Event History limits. Continue-As-New allows the Workflow to reset and continue with a fresh history.
Event History hard limit
The Workflow Execution's Event History is limited to 51,200 Events or 50 MB. Warnings are issued at 10,240 Events or 10 MB.
Time constraints on Workflow Execution duration
There is no time constraint on how long a Workflow Execution can run. However, if a Workflow performs many actions or receives many messages, it can run into Event History limits. It can also hit Workflow Versioning and other backwards incompatibility problems. For these reasons, it can be a good idea to Continue-As-New periodically.
What is an Event in Temporal
Events are created by the Temporal Service in response to external occurrences and Commands generated by a Workflow Execution. Each Event corresponds to an enum defined in the Server API. All Events are recorded in the Event History. A comprehensive list of all possible Events that could appear in a Workflow Execution Event History is provided in the Event reference documentation.
Purpose of Event History
Event History serves two critical purposes: it enables developers to know what took place during a Workflow Execution, and it is essential for providing Durable Execution by enabling the Workflow Execution to recover from a crash and continue making progress. Event History is an append-only log of Events for an application and serves as an audit log for debugging.
Event History durability and recovery
Event History is durably persisted by the Temporal service, enabling seamless recovery of an application state from crashes or failures.
Event History limits on number of events
The Temporal Service logs a warning after 10,240 Events and periodically logs additional warnings as new Events are added. The Workflow Execution is terminated when the Event History exceeds 51,200 Events.
Event History limits on Updates and Signals
The Workflow Execution is terminated when the Event History contains more than 2,000 Updates or more than 10,000 Signals.
Continue-As-New for Event History limits
To avoid hitting Event History limits, you can use the Continue-As-New feature to close the current Workflow Execution and create a new one.
Activity Events sequence
Seven Activity-related Events are added to Event History at various points in an Activity Execution. After a Workflow Task Execution reaches code that starts an Activity, the Worker sends the Activity Type and arguments to the Temporal Service, which adds an ActivityTaskScheduled Event to Event History. When ActivityTaskScheduled is added, the Temporal Service adds a corresponding Activity Task to the Task Queue. A Worker polling the Task Queue picks up the Activity Task and runs the Activity function. If the Activity function returns, the Worker reports completion and the Temporal Service adds ActivityTaskStarted and ActivityTaskCompleted to Event History. If the Activity function throws a non-retryable Failure, the Temporal Service adds ActivityTaskStarted and ActivityTaskFailed to Event History. If the Activity function throws a retryable Failure, the Temporal Service schedules an Activity Task retry to be added to the Task Queue (unless Maximum Attempts is reached, in which case ActivityTaskStarted and ActivityTaskFailed are added). If Start-to-Close Timeout passes before the Activity function returns or throws, the Temporal Service schedules a retry. If Schedule-to-Close Timeout passes before Activity Execution is complete, or if Schedule-to-Start Timeout passes before a Worker gets the Activity Task, the Temporal Service writes ActivityTaskTimedOut to Event History. If the Activity is canceled, the Temporal Service writes ActivityTaskCancelRequested to Event History, and if the Activity accepts cancellation, the Temporal Service writes ActivityTaskCanceled.
ActivityTaskStarted timing in Event History
While an Activity is running and retrying, ActivityTaskScheduled is the only Activity-related Event in History. ActivityTaskStarted is written along with a terminal Event like ActivityTaskCompleted or ActivityTaskFailed.
Event loop definition
A Workflow Execution is made up of a sequence of Events called an Event History. Events are created by the Temporal Service in response to either Commands or actions requested by a Temporal Client, such as a request to spawn a Workflow Execution.
What is a Reset
A Reset terminates a Workflow Execution and creates a new Workflow Execution with the same Workflow Type and Workflow ID. The Event History is copied from the original execution up to and including the reset point. The new execution continues from the reset point. Valid reset points are: WorkflowTaskStarted, WorkflowTaskCompleted, WorkflowTaskTimedOut, and WorkflowTaskFailed. Signals in the original history can be optionally copied to the new history, whether they appear after the reset point or not.
Principal Attribution definition
Principal Attribution lets you see who triggered each event in a Workflow's Event history. For example, you can determine which user or service started a Workflow, sent a Signal, or requested cancellation. These identity fields are set by the Temporal Service itself based on the authenticated caller, so they can't be forged by client code. Principal Attribution is especially valuable for compliance and audit use cases, incident investigation and root cause analysis, and access governance and internal accountability.
Principal Attribution implementation requirements
To populate Principal Attribution fields, you need a custom Authorizer that sets the Principal field on the authorization result.
Principal Attribution Temporal Cloud values
When enabled, Temporal Cloud populates the Principal value with Principal Type and Principal Name fields. Possible values are: Type 'users' with Name as user email address; Type 'service-accounts' with Name as service account name; Type 'mtls' with Name as Common Name (CN) or Subject Domain Name (DN) if CN is not present; Type 'temporal' with Name as Temporal internal services. Anyone who has permission to read Event history in the Namespace (ReadOnly access and above) can see the Principal and the metadata such as email address. To enable Principal Attribution for a Namespace, contact Temporal Cloud support.
Principal Attribution self-hosted Temporal configuration
In self-hosted Temporal, you can control Principal Attribution with a dynamic config flag scoped to the Namespace. When enabled, the Principal returned by the Authorizer is stamped on Event history events. To enable, set 'frontend.enablePrincipalPropagation' to 'true' for the appropriate Namespace.
Principal Attribution self-hosted Temporal default values
When using the default Authorizer with the default JWT ClaimMapper in self-hosted Temporal, the following values are populated: Type 'jwt' with Name as the value of the JWT 'sub' claim; Type 'temporal' with Name as 'internal' for internal frontend requests. A custom Authorizer must set the Principal field on authorization.Result for the request to be attributed. Custom ClaimMapper implementations control the AuthType and Subject values that the default Authorizer then copies into the Principal.
State Transition definition
A State Transition is a unit of progress made by a Workflow Execution. Each State Transition is recorded in a persistence store.
State Transitions for Activity Heartbeats
Some operations, such as Activity Heartbeats, require only one or two State Transitions each. With an Activity Heartbeat, there are two: the Activity Heartbeat and a Timer.
State Transitions example: two sequential Activity Tasks
A simple Workflow with two sequential Activity Tasks (and no retries) produces 11 State Transitions: two for Workflow start, four for each Activity, and one for Workflow completion.
Pause recorded in Event History
Pause is recorded in Event History. The Event includes the identity, reason, and request ID.
Unpause recorded in Event History
Unpause is recorded in Event History with the identity, reason, and request ID.
Activity Execution generates three event history events
Calling an Activity Execution from a Workflow Definition generates the ScheduleActivityTask Command, which results in three Activity Task related Events in the Workflow Execution Event History: ActivityTaskScheduled, ActivityTaskStarted, and ActivityTaskClosed.
Activity invocation parameters and return values are recorded in history
The values passed to Activities through invocation parameters or returned through a result value are recorded in the Execution history. The entire Execution history is transferred from the Temporal service to Workflow Workers when a Workflow state needs to recover. Large Execution histories can adversely impact the performance of your Workflow.
Calling Activity method on interface schedules invocation
Calling a method on the Activity interface schedules the Activity invocation with the Temporal service and generates an ActivityTaskScheduled Event.
Child Workflow events logged in Event History
When using a Child Workflow API, Child Workflow related Events are logged in the Workflow Execution Event History: StartChildWorkflowExecutionInitiated, ChildWorkflowExecutionStarted, and ChildWorkflowExecutionCompleted.
ChildWorkflowExecutionStarted Event timing requirement
The ChildWorkflowExecutionStarted Event must be logged to the Event History before the Parent Workflow completes to ensure the Child Workflow has started.
Command execution workflow
A Workflow might request to execute an Activity, start a Timer, or start a Child Workflow, each of which translates into a Command, dispatched to the Temporal Service. The Temporal Service acts on these Commands and documents that interaction by appending their corresponding Events into the Workflow Execution's Event History.
Workflow Execution start process
Starting a Workflow Execution creates a new Event, WorkflowExecutionStarted, and adds it to the Workflow Execution's Event History. The Temporal Service then schedules a Workflow Task by adding it to the Task Queue. When the Worker has capacity, it picks up this Task and begins executing code.
Task lifecycle events in Event History
Each step of the Task gets recorded into the Event History. Scheduled means that the Temporal Service has added a Task to the Task Queue. Started means that the Worker has dequeued the Task. Completed means that the Worker finished executing the Task by responding to the Temporal Service.
Activity completion and result persistence
When Activity code completes successfully, the Worker tells the Temporal Service the Activity Task completed, including any data that was returned from the Activity (results of the API call). This data is then persisted in the Workflow Execution Event History and is now accessible to the Workflow code.
Event History contains ordered log of Workflow events
Each Workflow Execution emits a series of Commands and processes a sequence of Events, which are recorded in an Event History. The Event History is a complete, ordered log of everything that has already happened in a Workflow and serves as the source of truth for everything that happens in the Workflow.