activity execution

26 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

ScheduleToCloseTimeout enforces SLA across all retry attempts

ScheduleToCloseTimeout sets a hard time budget that starts when an Activity is first scheduled and expires regardless of how many retry attempts occur. When the timeout expires, the Temporal Service marks the Activity Execution as timed out, delivers an ActivityError to the Workflow, and stops scheduling further retries. This enforces a business SLA by ensuring the Activity either succeeds or fails within the defined time window.

StartToCloseTimeout does not limit total retry time

StartToCloseTimeout limits how long a single Activity attempt may run before cancellation and retry, but it does not limit how long retries collectively may run. A process with StartToCloseTimeout=5m and unlimited retry policy can run for days as each attempt times out at 5 minutes, then Temporal waits for backoff delay and tries again indefinitely.

ScheduleToCloseTimeout does not forcibly stop running Activity code

When ScheduleToCloseTimeout expires, the Activity code that is already running is not forcibly stopped. An Activity running past the budget must heartbeat and handle cancellation to stop cooperatively.

Fixed Wall-Time Retries pattern example: 2-minute SLA with per-attempt cap

Set both schedule_to_close_timeout (the total budget) and start_to_close_timeout (the per-attempt cap). The retry policy controls the interval between attempts. Temporal stops retrying automatically when the budget runs out. Python example: await workflow.execute_activity(activities.authorize_transaction, transaction_id, schedule_to_close_timeout=timedelta(minutes=2), start_to_close_timeout=timedelta(seconds=30), retry_policy=RetryPolicy(initial_interval=timedelta(seconds=5), backoff_coefficient=1.5, maximum_interval=timedelta(seconds=30))). Go example: ActivityOptions{ScheduleToCloseTimeout: 2*time.Minute, StartToCloseTimeout: 30*time.Second, RetryPolicy: &temporal.RetryPolicy{InitialInterval: 5*time.Second, BackoffCoefficient: 1.5, MaximumInterval: 30*time.Second}}. Java example: ActivityOptions.newBuilder().setScheduleToCloseTimeout(Duration.ofMinutes(2)).setStartToCloseTimeout(Duration.ofSeconds(30)).setRetryOptions(RetryOptions.newBuilder().setInitialInterval(Duration.ofSeconds(5)).setBackoffCoefficient(1.5).setMaximumInterval(Duration.ofSeconds(30)).build()).build(). TypeScript example: proxyActivities({scheduleToCloseTimeout: '2m', startToCloseTimeout: '30s', retry: {initialInterval: '5s', backoffCoefficient: 1.5, maximumInterval: '30s'}}).

Short SLA without per-attempt timeout uses ScheduleToCloseTimeout alone

For tighter budgets such as a 30 second authorization window, you may omit StartToCloseTimeout and let ScheduleToCloseTimeout act as the only bound. Temporal requires at least one timeout to be set; ScheduleToCloseTimeout alone satisfies that requirement.

ScheduleToCloseTimeout includes ScheduleToStart delay

ScheduleToCloseTimeout begins when the Activity is first scheduled, which includes the time the task waits in the queue before a Worker picks it up. Under high load or insufficient Worker capacity, tasks can sit in the queue for seconds or minutes before the first attempt starts, consuming SLA budget before any work is done. Provision Workers with enough capacity for peak traffic or use autoscaling to keep ScheduleToStart latency negligible relative to the SLA window.

Do not use StartToCloseTimeout alone for SLA enforcement

A downstream system that responds slowly but never fully times out can keep resetting the per-attempt clock indefinitely, preventing the per-attempt timeout from enforcing the overall SLA. Use ScheduleToCloseTimeout as the outer boundary for SLA enforcement.

Do not set ScheduleToCloseTimeout shorter than StartToCloseTimeout

If the total budget (ScheduleToCloseTimeout) is shorter than a single attempt's maximum (StartToCloseTimeout), the first attempt cannot finish within the budget. The Temporal Service times out the Activity Execution and returns an error before any attempt can succeed.

Account for backoff delays when budgeting ScheduleToCloseTimeout

The total time in ScheduleToCloseTimeout includes both attempt durations and the backoff delays between them. For example, a 1-hour budget with a 30-minute initial interval and backoff coefficient 2.0 leaves room for only one or two attempts. Plan the budget to account for both work time and retry delays.

Three steps for asynchronous Activity completion

The process involves: (1) The Activity provides the external system with identifying information needed to complete the Activity Execution, which can be a Task Token or a combination of Namespace, Workflow Id, and Activity Id. (2) The Activity Function completes in a way that identifies it as waiting to be completed by an external system. (3) The Temporal Client is used to Heartbeat and complete the Activity.

Mark Activity as completing asynchronously in .NET

To mark an Activity as completing asynchronously in .NET, capture the task token from ActivityExecutionContext.Current.Info.TaskToken and throw a CompleteAsyncException().

Get async activity handle in .NET

Use the GetAsyncActivityHandle() method on the Temporal Client, passing the captured task token as a byte array parameter, to get the handle of the Activity for external completion.

Async activity handle methods in .NET

On the async activity handle obtained via GetAsyncActivityHandle(), you can call HeartbeatAsync, CompleteAsync, FailAsync, or ReportCancellationAsync methods to update the Activity.

CompleteAsync example for .NET

To complete an Activity asynchronously in .NET: var handle = myClient.GetAsyncActivityHandle(capturedToken); await handle.CompleteAsync("Completion value.");

Capture task token for async completion in .NET

Inside an Activity Function, capture the task token using: capturedToken = ActivityExecutionContext.Current.Info.TaskToken;

CompleteAsyncException in .NET Activities

The CompleteAsyncException is a special exception thrown inside an Activity to indicate that the Activity will be completed asynchronously by an external system rather than completing within the Activity Function itself.

Reset stuck activities to retry immediately after API recovery

When activities are stuck at their maximum retry intervals due to downstream API outages, use temporal activity reset --workflow-id <workflow-id> --activity-id <activity-id> to reset activities to retry immediately. This is useful when downstream APIs recover from outages and you want to immediately retry activities that are waiting at long backoff intervals (for example, 10 minutes).

Heartbeat implementation in long-running activities

Send periodic heartbeats during long-running activities to signal the Worker is alive. In Python, use activity.heartbeat() with a progress message. Example: For a file processing loop, call activity.heartbeat(f"Processing file - {elapsed}s elapsed") inside the loop to prove the Worker is responding. Heartbeat failures trigger the heartbeat_timeout, enabling fast detection of Worker crashes.

Activity execution failures include benign failures

ApplicationFailure instances marked with category BENIGN do not increment the activity execution failed counter. How well this metric tracks only unexpected failures depends on how consistently the application marks expected failures as benign.

Activity schedule-to-start latency grows with retry backlog

When Activity failures trigger a burst of retry Tasks that Workers cannot keep up with, the Activity Task backlog grows, causing elevated Activity schedule-to-start latency.

NOT_FOUND on Activity heartbeat causes

A Worker heartbeated a running Activity and the Temporal Service replied that the task no longer exists. The Service has already cancelled the in-flight Activity Task: either the `heartbeatTimeout` fired before the next heartbeat call arrived, the `startToClose` timeout expired while the Activity was still executing, or the Workflow Execution is no longer running. Normal Workflow-side cancellation is not a cause—cancellation returns `CancelRequested=true` in the heartbeat response body rather than a gRPC error.

Heartbeat timeout requires frequent calls

The Worker must call heartbeat more frequently than the `heartbeatTimeout`. If the Activity slows down between heartbeat calls because of CPU pressure, blocking I/O, or downstream throttling, the effective interval grows past the timeout even though the code is calling heartbeat.

Check temporal_activity_execution_latency for Activity timeout diagnosis

For Activity NOT_FOUND on respond operations, check `temporal_activity_execution_latency` for the affected `activity_type`. If p99 is at or above the corresponding timeout, that is the direct cause of the timeout.

Check temporal_activity_execution_latency for Activity heartbeat NOT_FOUND

For NOT_FOUND on Activity heartbeat, compare `temporal_activity_execution_latency` for the affected `activity_type` against the configured `startToClose` timeout. If the Activity has run longer than `startToClose`, the Service times it out while the Activity is still executing, and the next heartbeat returns NOT_FOUND.

Check temporal_request_failure for throttling on heartbeat calls

Query `temporal_request_failure` with `status_code=RESOURCE_EXHAUSTED` and `operation=RecordActivityTaskHeartbeat`. If the Temporal Service is throttling these calls, the effective heartbeat interval grows past `heartbeatTimeout` even when the Worker calls on time.

Activity execution latency metric

The metric temporal_activity_execution_latency is tagged by activity_type and is used to detect if Activities are holding slots longer than expected.

Give your agent this brain