activity retries & timeouts

9 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Activity retry policy configuration

Retry policies specify initial_interval, backoff_coefficient, maximum_interval, and maximum_attempts. For database writes, use conservative retry policies with gentle backoff. For non-critical operations like notifications, use more aggressive retry limits since these operations can fail without blocking the entity.

Recommended retry policy for rate-limited APIs

For rate-limited APIs, use a RetryPolicy with: initial_interval=timedelta(seconds=1), maximum_interval=timedelta(minutes=10), backoff_coefficient=2.0, maximum_attempts=5. This creates retry intervals of 1s, 2s, 4s, 8s, 16s across 5 attempts, totaling ~31 seconds before failure. The maximum_attempts of 5 prevents infinite retries for persistent rate limiting issues and ensures timely failure detection. The maximum_interval of 10 minutes caps retry delays at a reasonable duration.

Activity timeout configuration for worker affinity

When executing Activities on a Worker-specific unique queue, configure three timeouts: (1) start_to_close_timeout: Set to slightly longer than the Activity should take (e.g., 10 minutes for download_file). (2) schedule_to_start_timeout: Set to a short duration (e.g., 5 minutes) to detect Worker crashes before they pick up tasks. (3) heartbeat_timeout: Set to a short duration (e.g., 30 seconds) to detect Worker crashes during execution.

ActivityTaskTimedOut event and timeout types

ActivityTaskTimedOut event indicates that the Activity has timed out according to the Temporal Server, due to Activity timeouts: Schedule-to-Close Timeout or Schedule-to-Start Timeout. It has fields: failure (Serialized result of Workflow failure), scheduled_event_id (Id of ActivityTaskScheduled Event this timeout Event corresponds to), started_event_id (Id of ActivityTaskStarted Event this timeout corresponds to), retry_state (Reason provided for whether Task should or shouldn't be retried), timeout_type (Type of timeout that led to Event, for example Start-to-Close, Schedule-to-Close, Schedule-to-Start). A Workflow can contain an Activity Execution that takes longer than the Start-to-Close Timeout with a RetryPolicy setting MaxAttempts to 1 to prevent indefinite retry, and when the Activity times out, the ActivityTaskTimedOut Event contains the type of timeout that led to the Event.

ApplicationFailureException non-retryable parameter

ApplicationFailureException in .NET SDK supports a nonRetryable parameter that allows you to decide whether an error should not be retried automatically by Temporal. This is useful for deliberately failing a Workflow due to bad input data rather than waiting for a timeout to elapse.

Handling TimeoutError in Go Workflows

Use errors.As(err, &timeoutErr) to check for *TimeoutError. Call timeoutErr.TimeoutType() to determine the timeout type. Timeout types include commonpb.ScheduleToStart (scheduled but not started), commonpb.StartToClose (started but not completed), and commonpb.Heartbeat (no heartbeat received).

ScheduleToCloseTimeout for Activities

ScheduleToCloseTimeout is set on the Activity call options, not in RetryPolicy. It caps the total wall-clock time from when the Activity is first scheduled to when it must complete, across all retry attempts.

RetryPolicy fields and defaults

RetryPolicy has the following fields: MaximumAttempts (default 0, meaning unlimited; caps total attempts including the first), InitialInterval (default 1 second; delay before the first retry), BackoffCoefficient (default 2.0; multiplier applied after each retry), MaximumInterval (default 100× InitialInterval; upper bound on the backoff delay), and NonRetryableErrorTypes (default empty list; error types that skip retries entirely).

RetryPolicy default behavior

Temporal's default RetryPolicy retries Activities indefinitely with exponential backoff. Unless you configure a policy, a failing Activity will keep retrying until the ScheduleToCloseTimeout or the Workflow itself completes.

Give your agent this brain