error-handling

109 notes in this subject, read out of this brain and free to use. This is page 1 of 2.

Use idempotency keys with Workflow Run ID and Activity ID

To prevent duplicate operations when Activities are retried, use idempotency keys by combining the Workflow Run ID and Activity ID for a value that is consistent across retries but unique across Workflow Executions.

Saga pattern for compensation in multi-step failures

When a multi-step process fails partway through, the Saga pattern coordinates a sequence of operations where each step has a compensating action that reverses its effects. If any step fails, the compensating actions for previously completed steps execute in reverse order to undo previous steps.

Three categories of failures: transient, intermittent, permanent

Failures fall into three categories based on whether retrying can resolve them. Transient failures are one-off events that resolve without intervention and are resolved by retrying shortly after the failure. Intermittent failures recur but resolve over time, requiring retries spaced out over a longer period. Permanent failures recur indefinitely until the cause is fixed and cannot be resolved through retries.

Use cases for marking errors as non-retryable

Mark errors as non-retryable in situations like: invalid input data (malformed email address, negative payment amount, missing required field), business rule violations (customer outside service area, order exceeding credit limits, expired promotion code), authorization failures (caller does not have permission), and data validation errors (referenced record does not exist, data fails integrity checks).

Two ways to mark errors as non-retryable

Errors can be marked as non-retryable in two ways: (1) In the Activity by setting the non_retryable flag when throwing an Application Failure, which enforces the constraint for all callers when the Activity implementer knows the error can never be resolved through retries; (2) In the Retry Policy by adding the error type to the list of non-retryable error types, which lets different Workflows make different decisions about the same Activity when the decision depends on the caller's business logic.

Preserve non-retryable flag when wrapping errors

When an Activity returns an error, the SDK checks the outermost error type to determine retryability. If you catch a non-retryable Application Failure and re-throw it wrapped in a generic language error, the non_retryable flag is lost and the Activity will be retried. To add context to an error while preserving its retry behavior, wrap it in another Application Failure with the same non_retryable flag instead of wrapping Application Failures in generic language errors.

Use non-retryable errors sparingly

In most cases, let the Retry Policy handle retry limits through timeouts and maximum attempts. Reserve non_retryable for cases where retrying is guaranteed to be futile.

Design Activities for idempotence

Activities may execute more than once due to retries, so they should be designed to be idempotent, producing the same result whether executed once or multiple times. This is especially important because a Worker can execute an Activity, complete it, and then crash before reporting the result to the Temporal Service, causing the Activity to be retried even though it completed.

Transient failure definition and handling

A transient failure is a one-off event that resolves on its own without intervention, such as a Worker making a network request at the moment an administrator replaces a network cable. Transient failures are resolved by retrying the operation shortly after the failure, and Temporal's default Retry Policy handles transient failures automatically.

Intermittent failure definition and handling

An intermittent failure is one that recurs but resolves over time, such as a service that uses rate limiting and will reject requests once the threshold is reached but accept requests again after the rate limiter resets. Intermittent failures require retries spaced out over a longer period, and you should configure your Retry Policy with an appropriate backoffCoefficient and maximumInterval to avoid overwhelming the failing service.

Permanent failure definition and handling

A permanent failure is one that will recur indefinitely until the cause is fixed, such as a request that fails due to an invalid email address. Permanent failures cannot be resolved through retries and require different input data, a code fix, or some external intervention. These errors should be marked as non-retryable to fail fast instead of consuming resources on retries that will not succeed.

Failure

Temporal Failures are representations of various types of errors that occur in the system.

ApplicationFailure.nonRetryable() bypasses retry policy for permanent failures

Use ApplicationFailure.nonRetryable(message) to throw errors that should not be retried by Temporal's built-in retry policy. This instructs the SDK to skip retries and propagate the error directly to the Workflow code. Examples of non-retryable failures include invalid input data (malformed SSN, unrecognized employer), policy violations (DTI ratio exceeding lending limits), missing records (property ID not found in title search), and compliance blocks (OFAC sanctions matches). The Workflow then has full control to handle the failure: logging, updating Search Attributes for visibility, and suspending via condition() until corrected data arrives via Signal.

ApplicationFailure with 'RollbackRequired' error type triggers saga compensation instead of pause-and-fix

An Activity can throw ApplicationFailure.nonRetryable(message, 'RollbackRequired') to signal that no data correction will help and the application must be withdrawn. The Workflow catches this specific failure type and routes it to the compensation phase instead of the pause-and-resume loop. For example, underwrite() throws ApplicationFailure.nonRetryable('Compliance block: OFAC/sanctions match — application must be withdrawn', 'RollbackRequired'). This distinction allows the Workflow to handle two failure categories: permanent data errors that pause-and-fix can resolve (invalid SSN, bad address, unknown employer) versus regulatory or policy blocks that require unwinding the entire pipeline.

NonRetryableErrorTypes in provisioning activities

Both AddTRUs and RemoveTRUs Activities define a RetryPolicy with NonRetryableErrorTypes set to strings containing HTTP status codes: '401' (unauthorized), '403' (forbidden), and '400' (bad request). These errors indicate configuration or authentication issues that will not resolve with retries and should fail immediately.

Activity and Workflow default retry policy dynamic configuration

Default retry policies for Activities and Workflows can be defined at the Cluster level. history.defaultActivityRetryPolicy (Map) sets server configuration for an Activity Retry Policy when not explicitly set in code. history.defaultWorkflowRetryPolicy (Map) sets Retry Policy for unset fields where the user has set an explicit RetryPolicy but not specified all fields. Both use default values documented in the retry policies reference.

Bad Signal Workflow Execution Attributes error

This Workflow Task error indicates failure to validate attributes for SignalExternalWorkflowExecution. Reset any unset, missing, nil, or invalid attributes and adjust the input to fit within system size limits.

Bad Start Child Execution Attributes error

This Workflow Task error indicates failure to validate attributes for StartChildWorkflowExecution. Adjust the input size of the attributes to fall within system size limits and ensure Search Attribute validation is performed after unaliasing keys.

Bad Start Timer Attributes error

This Workflow Task error indicates that the scheduled Event is missing a Timer Id. Set a valid Timer Id and retry the Workflow Task.

Cause Bad Binary error

This Workflow Task error indicates that the Worker deployment returned a bad binary checksum.

Cause Bad Update error

This Workflow Task error indicates that a Workflow Execution tried to complete before receiving an Update. It occurs when a Worker generates a Workflow Task Completed message with missing fields or an invalid Update response format. This may indicate usage of an unsupported SDK. Make sure you are using a supported SDK.

Cause Reset Workflow error

This Workflow Task error indicates failure due to a request to reset the Workflow. If the system hasn't started a new Workflow, manually reset the Workflow.

Cause Unhandled Update error

UnhandledUpdate occurs when a Workflow Update is received by the Temporal Server while a Workflow Task being processed on a Worker produces a Command that would cause the Workflow to transition to a closed state. Temporal rejects the Workflow Task completion to guarantee that the Update is eventually handled and rewinds the Workflow so it can handle the pending Update. This can happen when the Workflow receives frequent Updates.

Cause Unspecified error

This Workflow Task error indicates failure for an unknown reason. If encountered, examine the Workflow Definition.

Failover Close Command error

This Workflow Task error indicates that a Namespace failover forced the Workflow Task to close. The system automatically schedules a retry when this error occurs.

Force Close Command error

This Workflow Task error indicates that the Workflow Task was forced to close. A retry will be scheduled if the error is recoverable.

Pending Activities Limit Exceeded error

The Workflow has reached capacity for pending Activities. Therefore, the Workflow Task was failed to prevent the creation of another Activity. Let the Workflow complete any current Activities before redeploying the code.

Pending Child Workflows Limit Exceeded error

This Workflow Task error indicates that the Workflow has reached capacity for pending Child Workflows. Wait for the system to finish any currently running Child Workflows before redeploying the Task.

Pending Nexus Operations Limit Exceeded error

The Workflow has reached capacity for pending Nexus Operations. Therefore, the Workflow Task was failed to prevent the creation of another Nexus Operation. Let the Workflow complete any current Nexus Operation before retrying the Task.

Pending Request Cancel Limit Exceeded error

This Workflow Task error indicates failure after attempting to add more cancel requests. The Workflow has reached capacity for pending requests to cancel other Workflows. Give the system time to process pending requests before retrying the Task.

Pending Signals Limit Exceeded error

The Workflow has reached capacity for pending Signals. Therefore, the Workflow Task was failed after attempting to add more Signals to an external Workflow. Wait for Signals to be processed by the Workflow before retrying the Task.

Reset Sticky Task Queue error

This error indicates that the Sticky Task Queue needs to be reset. If encountered, reset the Sticky Task Queue and the system will retry automatically.

Resource Exhausted Cause Concurrent Limit error

This Workflow Task error indicates that the concurrent poller count has been exhausted. Adjust the poller count per Worker.

Resource Exhausted Cause System Overload error

This Workflow Task error indicates that the system is overloaded and cannot allocate further resources to Workflow Tasks.

Resource Exhausted Cause Unspecified error

This Workflow Task error indicates that an unknown cause is preventing resources from being allocated to further Workflow Tasks.

Schedule Activity Duplicate Id error

The Workflow Task failed because the Activity Id is already in use. Check the Workflow code to see if the same Activity Id has already been specified. Enter another Activity Id and try running the Workflow Task again.

Start Timer Duplicate Id error

This error indicates that a Timer with the given Timer Id has already started. Try entering a different Timer Id and retry the Workflow Task.

Unhandled Command error

This Workflow Task error indicates new available Events since the last Workflow Task started. The Workflow Task was failed because the Workflow attempted to close itself without handling the new Events. UnhandledCommand can happen when the Workflow is receiving a high number of Signals. If the Workflow doesn't have enough time to handle these Signals, a RetryWorkflow Task is scheduled. To prevent this error, drain the Signal Channel with the ReceiveAsync function. If the error continues, check logs for failing Workflow Tasks, as the Workflow may have been picked up by a different Worker.

Workflow Worker Unhandled Failure error

This Workflow Task error indicates that the Workflow Task encountered an unhandled failure from the Workflow Definition.

gRPC Message Too Large error 4 MB limit

This error occurs when the Workflow Task response exceeds the gRPC message size limit of 4 MB. The Workflow Execution is automatically terminated because this is a non-recoverable error. This typically happens when a Workflow schedules too many Activities, Child Workflows, or commands in a single Workflow Task, or when a Workflow returns a large result. To resolve this error, fix the Workflow code and start a new Workflow Execution. Break work into smaller batches, reduce the size of Workflow returns, use Continue-As-New for long-running Workflows, or compress large payloads with a custom Payload Codec.

Resource Exhausted Cause Persistence Limit error

This Workflow Task error indicates that the persistence rate limit has been reached.

Resource Exhausted Cause RPS Limit error

This Workflow Task error indicates that the Workflow has exhausted its RPS limit.

Bad Cancel Timer Attributes error

This Workflow Task error occurs when attempting to cancel a Timer with missing Timer Id. Check Timer attributes for missing Timer Id values and add a valid Timer Id, then redeploy the code.

Bad Cancel Workflow Execution Attributes error

This Workflow Task error occurs due to unset CancelWorkflowExecution attributes. Reset any missing attributes and redeploy the Workflow Task.

Bad Complete Workflow Execution Attributes error

This Workflow Task error occurs due to unset attributes on CompleteWorkflowExecution or when the payload exceeds size limits. Reset any missing attributes and adjust the size of the payload to stay within limits.

Bad Continue as New Attributes error

This Workflow Task error occurs when ContinueAsNew attributes fail validation, either because they are unset or invalid. Reset any missing attributes, adjust input size if payload or memo exceeds limits, and verify the Workflow validates search attributes after unaliasing keys.

Bad Fail Workflow Execution Attributes error

This Workflow Task error occurs due to unset FailWorkflowExecution attributes. Ensure that StartToCloseTimeout or ScheduleToCloseTimeout are set and restart the Worker that the Workflow and Activity are registered to.

Bad Modify Workflow Properties Attributes error

This Workflow Task error occurs when Upsert Memo or payload attributes fail validation because they are unset or exceed size limits. Reset any unset and empty attributes and adjust the Memo or payload size to fit within system limits.

Bad Record Marker Attributes error

This Workflow Task error occurs due to an unset or incorrect Marker name. Enter a valid Marker name and redeploy the Task.

Bad Request Cancel Activity Attributes error

This Workflow Task error indicates unset attributes for RequestCancelActivity or an invalid History Builder state. Update the Temporal SDK to the most recent release, reset any unset attributes, and review code for nondeterministic causes if the error persists.

Bad Request Cancel External Workflow Execution Attributes error

This Workflow Task error occurs while trying to cancel an external Workflow due to unset or invalid attributes. Reset missing attributes such as Workflow Id or Run Id, adjust fields that exceed length limits, and ensure Child Workflow is not set to both Start and RequestCancel in the same Workflow Task.

Bad Schedule Activity Attributes error

This Workflow Task error indicates unset or invalid attributes for ScheduleActivityTask or CompleteWorkflowExecution. Reset any unset or empty attributes and adjust the payload size to stay within the given size limit.

Bad Schedule Nexus Operation Attributes error

This Workflow Task error indicates unset or invalid attributes for ScheduleNexusOperation, such as when the Nexus Endpoint name used in the caller Workflow doesn't exist. Inspect the reason given in the error for mitigation when possible.

Bad Search Attributes error

This Workflow Task error indicates unset or invalid Search Attributes that can cause Workflow Tasks to retry without success. Make sure all attributes are defined before retrying the Task and adjust the payload size to fit within system limits.

Bad Signal Input Size error

This Workflow Task error indicates that the Payload has exceeded the Signal's available input size. Adjust the size of the Payload and redeploy the Workflow Task.

Application Failure definition

Application Failures are used by Workflow, Activity, and Nexus Operation code to communicate application-specific failures. This is the only type of Temporal Failure created and thrown by user code.

Workflow Task Failure versus Workflow Execution Failure

A Workflow Task Failure is an unexpected situation failing to process a Workflow Task, triggered by non-Temporal exceptions raised in Workflow code. These cause the Workflow Task to be retried until the Workflow Execution Timeout (unlimited by default). A Workflow Execution Failure puts the Workflow Execution into the 'Failed' state and no more attempts will be made. Only Temporal Failures cause Workflow Execution to fail; all other exceptions cause Workflow Task to fail and be retried (in Go, any error returned from Workflow fails Workflow Execution, and a panic fails Workflow Task).

Custom workflow exceptions must extend ApplicationError

When creating custom exceptions to fail a Workflow Execution, they must extend the ApplicationError class, which is a child class of FailureError.

Non-retryable ApplicationFailure flag

Activities and Workflows can avoid retrying by setting an Application Failure's non_retryable flag to true. When an Activity or Workflow throws an Application Failure, the Failure's type field is matched against a Retry Policy's list of non-retryable errors to determine whether to retry.

Next Retry Delay setting for ApplicationFailure

By setting the Next Retry Delay for a given Application Failure, you can tell the server to wait that amount of time before trying the Activity or Workflow again. This overrides whatever the Retry Policy would have computed for the specific exception.

Give your agent this brain