Activity result persistence
When the Activity Function returns, the Worker sends the results back to the Temporal Service as part of the ActivityTaskCompleted Event. The Event is added to the Workflow Execution's Event History.
Temporal · Concepts · all subjects
activities/execution
180 notes in this subject, read out of this brain and free to use. This is page 1 of 3.
Activity execution and failure recovery
Activity Functions are executed by Worker Processes. Workflow code orchestrates the execution of Activities, persisting the results. If an Activity Execution fails, any future attempt will start from the initial state, unless the code uses Heartbeat details payloads for checkpointing to store state on the server and use it when resuming subsequent attempts.
Activity retry policy behavior
For an Activity with a Retry Policy that allows retries, Temporal guarantees that the Activity will be observed as completed exactly once. However, the Activity may be executed multiple times and may even partially complete more than once during this process, which could lead to certain parts being executed multiple times before a successful execution is completed.
Pitfall: internal retries in Activities
Be cautious when doing retries within your Activity because it lengthens the needed Activity timeout. Such internal retries also prevent users from counting failure metrics and make it harder for users to debug in Temporal UI when something is wrong.
Activity Cancellation handling
An Activity may accept or ignore Cancellation. To allow Cancellation to happen, let the Cancellation Failure propagate. To ignore Cancellation, catch it and continue executing. Some SDKs have ways to shield tasks from being stopped while still letting the Cancellation propagate. The Workflow can also decide if it wants to wait for the Activity Cancellation to be accepted or to proceed without waiting.
Activity Execution failure when retry policy is 1
In scenarios where the Activity Execution Retry Policy is set to 1 and a Timeout occurs, the Activity Execution will not be tried again.
Activity ID definition
The Activity ID is the identifier for an Activity Execution. The identifier can be generated by the system or provided by the Workflow code that spawns the Activity Execution. The identifier is unique among the open Activity Executions of a Workflow Run. A single Workflow Run may reuse an Activity ID if an earlier Activity Execution with the same ID has closed.
Getting activity results from external systems
When an external system has the final result of a computation started by an Activity, there are three main ways of getting the result to the Workflow: (1) the external system uses Async Completion to complete the Activity with the result, (2) the Activity completes normally without the result and the external system sends a Signal to the Workflow with the result, or (3) a subsequent Activity polls the external system for the result.
Activity Execution guarantees
Temporal guarantees that an Activity Task either runs or timeouts. Temporal does not detect task loss directly; instead it relies on the Start-To-Close timeout. If the Activity Task times out, the Activity Execution will be retried according to the Activity Execution Retry Policy.
Cancelled Failure message types
When an Activity receives a Cancelled Failure, the message can be 'TIMED_OUT' if the Activity was considered failed by the Server because an Activity timeout triggered, or 'NOT_FOUND' if the Workflow Run reached a Closed state.
Standalone Activity ID space
Standalone Activities have a separate ID space from Workflows and other Temporal primitives. Use of conflict policy and reuse policy will only observe the Standalone Activity ID space.
Asynchronous Activity Completion definition
Asynchronous Activity Completion is a feature that enables an Activity Function to return without causing the Activity Execution to complete. The Temporal Client can then be used from anywhere to both Heartbeat Activity Execution progress and eventually complete the Activity Execution and provide a result.
Activity Cancellation single request limit
Cancellation can only be requested a single time. If you try to cancel your Activity Execution more than once, it will not receive more than one Cancellation request.
Task Token definition
A Task Token is a unique identifier for an Activity Task Execution.
Activity Execution error propagation
If an Activity Execution fails because it exhausted all retries, threw a non-retryable error, or was canceled, the error is returned to the Workflow code when it attempts to fetch the Activity result. For Standalone Activities, the error is returned to the Client when you attempt to fetch the Activity result.
Activity Cancellation requirements
Activities must heartbeat to receive cancellations from a Temporal Service.
Activity Execution definition
An Activity Execution is the full chain of Activity Task Executions.
Asynchronous Activity Completion arguments
Asynchronous Activity Completion calls take either a Task Token as an argument, or an Activity ID, a Workflow ID, and optionally a Run ID.
Task Token invalidation risk with retries
Since a Task Token is unique for an Activity execution, retries can cause a remote service using the Task Token to end up with an invalid one. For example, an Activity might fail after passing its current Task Token to a remote service but before returning the complete async error, leaving that service with a Task Token that is no longer valid. To avoid this risk, you can provide the Activity ID and Workflow ID to the remote service instead of the Task Token.
When to use Async Completion vs Signals
Use Async Completion if the external system is unreliable and might fail to Signal, or if you want the external process to Heartbeat or receive Cancellation. If the external system can reliably be trusted to do the task and Signal back with the result, and it does not need to Heartbeat or receive Cancellation, you may want to use Signals instead. The benefit to using Signals has to do with the timing of failure retries: with Signals, a shorter Start-To-Close Timeout can be set on the Activity, allowing failures to be retried sooner.
Activity Cancellation scenarios
An Activity may receive Cancellation if: (1) the Activity was requested to be cancelled, which can cascade from Workflow Cancellation; (2) the Activity was considered failed by the Server because any of the Activity timeouts have triggered and the Server did not receive a heartbeat within the Activity's Heartbeat timeout; (3) the Workflow Run reached a Closed state; (4) in some SDKs, the Worker is shutting down, the Activity sends a Heartbeat but the Heartbeat details cannot be converted by the Worker's configured Data Converter, or the Activity timed out on the Worker side and is not Heartbeating.
Monitoring Local Activities
You can monitor Local Activities by using the local_activity_total metric to determine how many Local Activity Executions have been made. You can also check your Event History for RecordMarker entries.
Mixing Local Activities and regular Activities
A Workflow can freely combine Local Activities and regular Activities. When the Workflow Task completes after scheduling both types, the Worker sends MarkerRecorded events for Local Activities and ScheduleActivityTask commands for regular Activities. If a regular Activity later fails or retries, the Workflow replays using the recorded markers for Local Activities. Those completed Local Activities are not executed again because their results have already been persisted in the Event history.
Workflow Task heartbeating mechanism
Workflow Task heartbeating means that if a Local Activity approaches approximately 80% of the Workflow Task Timeout (10 seconds by default), the Worker completes the current Workflow Task and requests a new one from the Temporal Service. This renews the Worker's authorization to continue executing the Workflow and allows the Local Activity to keep running without exceeding the Workflow Task Timeout.
Local Activity execution semantics
Local Activities provide at-least-once execution semantics and should always be idempotent. Once a MarkerRecorded event has been written to the Event history, replay uses the recorded result rather than executing the Local Activity again.
Local Activity durability and MarkerRecorded events
A Local Activity result becomes durable only when the enclosing Workflow Task successfully completes and records a MarkerRecorded event. Until then, execution exists only in Worker memory. If the Worker crashes before the Workflow Task completes, the Workflow Task is retried, causing Local Activities executed during that Workflow Task to run again.
Workflow Task heartbeating tradeoffs
Workflow Task heartbeating comes with tradeoffs: (1) Each Workflow Task heartbeat adds additional Events to the Event history. (2) Signals and other external Workflow events are not processed until the Local Activities finish. (3) Commands generated by the Workflow are not sent to the Temporal Service until the Local Activity completes or the next Workflow Task heartbeat occurs.
Local Activities do not support Activity heartbeats
Local Activities do not support Activity heartbeats. Instead, the SDK supports Workflow Task heartbeating.
Local Activity execution flow
Local Activities follow a shorter execution path: (1) A Workflow schedules a Local Activity. (2) The Local Activity is placed into an in-process queue within the Workflow Worker. (3) The Workflow Worker executes the Local Activity immediately. (4) The result is returned directly to the Workflow. (5) When the Workflow Task completes, the Worker records a MarkerRecorded event containing the Local Activity result. Unlike regular Activities, scheduling and execution occur entirely within the Worker process. Only the final marker is persisted to Event history.
Regular Activity execution flow
The regular Activity execution flow is: (1) A Workflow schedules an Activity. (2) The Workflow completes its Workflow Task by sending a ScheduleActivityTask command to the Temporal Service. (3) The Temporal Service creates an Activity Task. (4) An Activity Worker polls the Activity Task Queue and executes the Activity. (5) The Activity result is recorded in Event history. (6) The Workflow resumes in a new Workflow Task.
Detect Pause in Activity code by SDK
Activities with Heartbeat can detect Pause to handle it differently from timeouts or cancellations. Go v1.34.0+: catch activity.ErrActivityPaused. Java v1.29.0+: catch ActivityPausedException. TypeScript v1.12.3+: check cancellationDetails.paused === true. Python v1.12.0+: check cancellation_details().paused on asyncio.CancelledError. .NET v1.7.0+: check CancellationDetails.IsPaused on OperationCanceledException.
Pause operation stops retries but continues in-flight execution
Pause stops the Temporal Service from scheduling new retries of an Activity Execution. For Activities with Heartbeat, the in-flight execution is interrupted on the next Heartbeat and the SDK raises a Pause-specific error, allowing the Activity to clean up resources. For Activities without Heartbeat, the in-flight execution continues to completion; if it succeeds, the result is delivered normally; if it fails, no retry is scheduled. Pause takes effect after the in-flight execution ends. Pausing an Activity doesn't affect the parent Workflow, which continues running. Pause is idempotent and doesn't produce an Event History event.
Detect Reset in Activity code by SDK
Activities with Heartbeat can detect Reset to handle it differently from timeouts or cancellations. Go: activity.GetCancellationDetails(ctx).Cause() returns activity.ErrActivityReset. Java: catch ActivityResetException. TypeScript: catch ApplicationFailure with error.type === "ActivityReset". Python: check cancellation_details().reset on asyncio.CancelledError. .NET: check CancellationDetails.IsReset on OperationCanceledException.
Update Options important considerations and limitations
Changes to a running Activity take effect on the next execution, not the current one. If you need the change to apply immediately, the Activity must finish or fail its current execution first. --restore-original-options is batch-only and only works with --query; it's silently ignored in single-workflow mode and can't be combined with other option changes in the same command. Update Options is CLI and gRPC only; it's not available in the UI.
Unpause important considerations and pitfalls
Unpausing many Activities at once can overwhelm downstream services. If multiple Activities were Paused because a service was down, Unpausing them all simultaneously sends all retries at once. Consider Unpausing in batches to avoid overwhelming a recovering service. Unpausing doesn't override Workflow Pause; if the parent Workflow is also Paused, Unpausing the Activity alone isn't enough. Both must be Unpaused before the Activity resumes. Unpausing doesn't reset the attempt count; the Activity retries from its current attempt number. Use Reset to restart from attempt 1. A Paused Activity can time out before you Unpause it; the Schedule-To-Close Timeout isn't stopped or extended while Paused. Use update-options to extend the timeout before Unpausing if needed. Unpause doesn't interrupt or duplicate an in-flight execution; if an Activity without Heartbeat is still running when you Unpause, it continues to completion without concurrent execution.
Update Options operation changes runtime configuration without restarting
Update Options changes an Activity's runtime configuration without restarting it. You can change timeouts (Schedule-To-Close, Start-To-Close, Schedule-To-Start, Heartbeat), Retry Policy (initial interval, maximum interval, backoff coefficient, maximum attempts), and Task Queue. Only the fields you specify are changed; all other options remain unchanged. If the Activity is waiting for retry (scheduled), the new options take effect immediately and any pending retry timer is regenerated with the updated configuration. If the Activity is currently running, the new options are stored but take effect on the next execution; the in-flight execution isn't interrupted. If the Activity is Paused, the new options are stored immediately and take effect when the Activity is Unpaused. Update Options doesn't produce an Event History event. Update Options is idempotent.
Reset important considerations and limitations
A Reset Activity can still time out; Reset doesn't restart the Schedule-To-Close Timeout. The deadline is calculated from when the Activity was originally scheduled. Use update-options to extend the timeout before or after Reset. Heartbeat details are preserved by default; if your Activity uses Heartbeat details for progress tracking and you want a clean restart, pass --reset-heartbeats. Reset won't interrupt an Activity that doesn't Heartbeat; the current execution runs to completion, which could take up to the full Start-To-Close Timeout. If the Activity had already retried (attempt > 1), the Temporal Service rejects the current execution's result because Reset changed the expected attempt number. The Activity waits for its Start-To-Close Timeout to expire before a new execution is scheduled. --restore-original-options restores the Activity's original configuration, reverting timeouts, Retry Policy, and Task Queue to values from when the Activity was first scheduled. Bulk Reset can overwhelm downstream services; when using --query to Reset Activities across many Workflows, use --jitter to stagger the restart times.
Reset operation clears retry state and schedules fresh execution
Reset clears an Activity's retry state and schedules a fresh execution. The attempt count resets to 1, giving the Activity a full set of retry attempts regardless of how many it had used. Retry backoff is discarded; if the Activity was in a backoff wait, it's rescheduled to run immediately. If the Activity is Paused, Reset also Unpauses it unless --keep-paused is specified. With --keep-paused, the attempt count and Heartbeat data (if --reset-heartbeats) are reset, but the Activity stays Paused until Unpaused separately. Resetting an Activity doesn't affect the parent Workflow. Reset doesn't produce an Event History event. For Activities with Heartbeat, they are interrupted on the next Heartbeat and the SDK may raise a Reset-specific error. For Activities without Heartbeat, Reset doesn't cancel or interrupt; if the Activity was already retrying, the Temporal Service rejects the current execution's result because Reset changed the expected attempt number, and a fresh execution is scheduled after Start-To-Close Timeout expires. Reset is idempotent.
Pause CLI command syntax
Pause can be invoked with: temporal activity pause --workflow-id my-workflow --activity-id my-activity --reason "Downstream API is down, pausing until recovery"
Unpause CLI command syntax
Unpause can be invoked with: temporal activity unpause --workflow-id my-workflow --activity-id my-activity
Unpause operation resumes a Paused Activity immediately
Unpause resumes a Paused Activity Execution. The Activity is rescheduled immediately, and any remaining retry backoff is discarded. The next execution starts right away. Attempt count and Heartbeat data are preserved by default, allowing the Activity to resume from where it left off. Use --reset-attempts or --reset-heartbeats on the CLI to clear these, or use Reset to restart from attempt 1. Unpause is idempotent; unpausing an Activity that isn't Paused has no effect. Unpausing an Activity that has already completed returns an error.
Update Options CLI command syntax
Update Options can be invoked with: temporal activity update-options --workflow-id my-workflow --activity-id my-activity --schedule-to-close-timeout 24h
Pause limitations and important considerations
A Paused Activity can still time out; Pause doesn't stop or extend the Schedule-To-Close Timeout. Use update-options to adjust the timeout if needed. Pause won't interrupt an Activity that doesn't Heartbeat; the current execution runs to completion, which could take up to the full Start-To-Close Timeout. Pause operates on individual Activities by ID within a single Workflow; there's no --query flag. To pause multiple Activities, issue separate commands for each Activity ID. There's no Namespace-wide query for Paused Activities; you must know the Workflow Id. Workflow Pause and Activity Pause are independent; both stop Activity retries but must be Unpaused separately.
Activity Operations observability and audit trail
Activity Operations have a limited audit trail because they are not recorded in a Workflow's Event History. temporal workflow describe shows the current state of each pending Activity, including whether it's Paused, its current attempt count, and last failure. The UI shows who performed an operation, when, and why (if a --reason was provided). The TemporalPauseInfo Search Attribute is filterable within a Workflow. There's no Namespace-wide query to find all Paused Activities across Workflows; you must know the Workflow Id. Activity Operations don't produce Event History events; there is no record of a Pause, Reset, or option change in the Workflow's Event History. Nothing that reads the Event History—Workflow code, Replays, or external tooling—will see that an Operation occurred. Evidence of an Operation is gone when the Activity completes or the Workflow closes. There's no persistent record that an Activity was Paused, Reset, or had its options changed. The only way to confirm the current state of an Activity is temporal workflow describe or the UI.
Reset CLI command syntax
Reset can be invoked with: temporal activity reset --workflow-id my-workflow --activity-id my-activity or temporal activity reset --workflow-id my-workflow --activity-id my-activity --keep-paused
Activity Operations summary table
Pause: stops retries; in-flight execution continues unless the Activity uses Heartbeat. CLI: temporal activity pause. Unpause: resumes a Paused Activity; next execution starts immediately. CLI: temporal activity unpause. Reset: clears retry state (attempts, backoff) and schedules a new execution. CLI: temporal activity reset. Update Options: changes timeouts, Retry Policy, or Task Queue without restarting the Activity. CLI: temporal activity update-options.
Activity code is not replayed, can perform I/O
Activity code is not replayed the same way as Workflow code. Activity functions can do I/O and call external services because they execute directly rather than being reconstructed from Event History.
Global Namespace semantics: at-least-once for activities
Unlike regular Namespaces, which provide at-most-once semantics for an Activity Execution, Global Namespaces support only at-least-once semantics. This difference is due to the possibility of conflicts that can occur during multi-cluster replication.
Use Activity Heartbeats for very long-running Activities
If a long-running Activity runs longer than half the maximum invocation deadline, the constraints for configuration may be difficult or impossible to meet. In this case, use Activity Heartbeats to record the state of the Activity execution so that the next retry can pick up where it left off.
Use Activity Heartbeats with Cloud Run to handle instance interruption
If an Activity is interrupted by instance termination on Cloud Run, use Activity Heartbeats so that a retry can resume from the last recorded progress instead of restarting from the beginning.
Activity Task definition
An Activity Task runs one attempt of an Activity.
Activity Task completion events
Once an Activity Task finishes execution, the Worker responds to the Temporal Service with one of these specific Events: ActivityTaskCanceled, ActivityTaskCompleted, ActivityTaskFailed, ActivityTaskTerminated, or ActivityTaskTimedOut.
Activity Task Heartbeat and retry behavior
Across retries, an Activity Task can carry forward Heartbeat payload from the previous attempt, allowing long-running Activities to resume from their last checkpoint.
Activity Task Execution definition and lifecycle events
An Activity Task Execution occurs when a Worker uses the context provided from the Activity Task and executes the Activity Definition. The ActivityTaskScheduled Event corresponds to when the Temporal Service puts the Activity Task into the Task Queue. The ActivityTaskStarted Event corresponds to when the Worker picks up the Activity Task from the Task Queue. Either ActivityTaskCompleted or one of the other Closed Activity Task Events corresponds to when the Worker has yielded back to the Temporal Service. The API to schedule an Activity Execution provides an 'effectively once' experience, even though there may be several Activity Task Executions that take place to successfully complete an Activity.
ExecuteActivityAsync operation spawns Activity Execution in .NET SDK
To spawn an Activity Execution in the .NET SDK, use the ExecuteActivityAsync operation from within a Workflow Definition. It takes an Activity method invocation and Activity Options as parameters.
Activity result returned from ExecuteActivityAsync Task
The Activity result is returned in the Task from the ExecuteActivityAsync call.
.NET ExecuteActivityAsync example with timeout
using Temporalio.Workflows; [Workflow] public class MyWorkflow { public async Task<string> RunAsync(string name) { var param = MyActivityParams("Hello", name); return await Workflow.ExecuteActivityAsync( (MyActivities a) => a.MyActivity(param), new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) }); } } This example shows how to execute an Activity from a Workflow using ExecuteActivityAsync with a StartToCloseTimeout of 5 minutes.
What is asynchronous Activity completion and how is it triggered
Asynchronous Activity completion is triggered by throwing a CompleteAsyncException inside an Activity after capturing the TaskToken from ActivityExecutionContext.Current.Info.TaskToken. This signals that the Activity will be completed by an external system rather than completing when the Activity Function returns. The external system can then use the Temporal Client's GetAsyncActivityHandle() method with the captured token to complete, fail, heartbeat, or report cancellation of the Activity.
GetAsyncActivityHandle method and available operations
The GetAsyncActivityHandle() method retrieves a handle for an asynchronous Activity using the captured task token. On this handle, you can call HeartbeatAsync, CompleteAsync, FailAsync, or ReportCancellationAsync to update the Activity.
How to mark an Activity as completing asynchronously in .NET
To mark an Activity as completing asynchronously in .NET, capture the task token using ActivityExecutionContext.Current.Info.TaskToken and then throw a CompleteAsyncException() to signal that the activity will be completed elsewhere.