errors & failures

67 notes in this subject, read out of this brain and free to use. This is page 1 of 2.

Transient failures definition and handling

A transient failure is a one-off event that resolves on its own without intervention, such as a Worker making a network request at the moment an administrator replaces a network cable. Transient failures are resolved by retrying the operation shortly after the failure. Temporal's default Retry Policy handles transient failures automatically.

Intermittent failures definition and handling

An intermittent failure is one that recurs but resolves over time, such as a service using rate limiting that rejects requests once a threshold is reached but accepts them again after the rate limiter resets. Intermittent failures require retries spaced out over a longer period. Configure your Retry Policy with an appropriate backoffCoefficient and maximumInterval to avoid overwhelming the failing service.

Permanent failures definition and handling

A permanent failure is one that will recur indefinitely until the cause is fixed, such as a request failing due to an invalid email address. Permanent failures cannot be resolved through retries and require different input data, a code fix, or some external intervention. Mark these errors as non-retryable to fail fast instead of consuming resources on retries that will not succeed.

Non-retryable error use cases

Use non-retryable errors for: invalid input data (malformed email address, negative payment amount, missing required field), business rule violations (customer outside service area, order exceeding credit limits, expired promotion code), authorization failures (caller does not have permission), and data validation errors (referenced record does not exist, data fails integrity checks).

Two ways to mark errors as non-retryable

Errors can be marked as non-retryable in two ways: (1) In the Activity by setting the non_retryable flag when throwing an Application Failure, which enforces the constraint for all callers and is used when the Activity implementer knows the error can never be resolved through retries; (2) In the Retry Policy by adding the error type to the list of non-retryable error types, which lets different Workflows make different decisions about the same Activity and is used when the decision depends on the caller's business logic.

Preserving retryability when wrapping errors

When an Activity returns an error, the SDK checks the outermost error type to determine retryability. If you catch a non-retryable Application Failure and re-throw it wrapped in a generic language error, the non_retryable flag is lost and the Activity will be retried. To add context to an error while preserving its retry behavior, wrap it in another Application Failure with the same non_retryable flag. Do not wrap Application Failures in generic language errors.

When to use non-retryable errors sparingly

In most cases, let the Retry Policy handle retry limits through timeouts and maximum attempts. Reserve non_retryable for cases where retrying is guaranteed to be futile.

Error handling covers failure categorization and compensation patterns

Error handling documentation covers categorizing failures, when to mark errors non-retryable, and implementing compensation with the Saga pattern.

ActivityFailure must be distinguished from other Workflow errors

When catching Activity errors in a Workflow, only errors that are specifically ActivityFailure instances should be handled in the pause-and-resume loop. Anything that is not an ActivityFailure — a Workflow-side bug or a non-determinism error — must be re-thrown to fail the Workflow Task and let Temporal retry it, not be silently parked in PENDING_FIX. Additionally, if the error is a cancellation (checked via isCancellation(e)), it must be re-thrown so the framework can unwind the Workflow cleanly.

RollbackRequired failure type signals unrecoverable errors

An Activity can throw ApplicationFailure.nonRetryable(message, 'RollbackRequired') to signal that no data correction will help — the application must be withdrawn. The Workflow catches this specific failure type and routes it to the compensation phase instead of the pause-and-resume loop. This distinguishes between failures that can be fixed by correcting data versus failures that require rolling back the entire process.

FailWorkflowExecution command

The FailWorkflowExecution Command is triggered when the Workflow Execution returns an error or an exception is thrown. The corresponding Event is WorkflowExecutionFailed. This Command is not awaitable.

Temporal Failure base class types by SDK

The base Failure class that other Failures extend varies by SDK: TypeScript uses TemporalFailure, Java uses TemporalFailure, Python uses FailureError, and PHP uses TemporalFailure.

Base Failure proto message fields

The base Failure proto message has the following fields: string message, string stack_trace, string source (the SDK this Failure originated in, for example 'TypeScriptSDK'), Failure cause (the Failure message of the cause of this Failure if applicable), and Payload encoded_attributes (contains the encoded message and stack_trace fields when using a Failure Converter).

Application Failure is only user-created Temporal Failure type

Application Failure is the only type of Temporal Failure created and thrown by user code. Workflow, Activity, and Nexus Operation code use Application Failures to communicate application-specific failures.

Application Failure class names by SDK

Application Failure class names vary by SDK: TypeScript ApplicationFailure, Java ApplicationFailure, Go ApplicationError, Python ApplicationError, and PHP ApplicationFailure.

Workflow Task Failure vs Workflow Execution Failure

A Workflow Task Failure occurs when an unexpected situation fails to process a Workflow Task, triggered by a non-Temporal exception (or panic in Go). Only Temporal Failures cause the Workflow Execution to fail; all other exceptions cause the Workflow Task to fail and be retried. In Go, any error returned from the Workflow fails the Workflow Execution, while a panic fails the Workflow Task.

Workflow Task Failure retry behavior

Workflow Task Failures cause the Workflow Task to be retried until the Workflow Execution Timeout, which is unlimited by default.

Workflow Execution Failure state and retry behavior

Workflow Execution Failures put the Workflow Execution into the Failed state and no more attempts will be made in progressing this execution. An ApplicationError (extension of FailureError) can be raised in a Workflow to fail the Workflow Execution.

Cancelled Failure class names by SDK

Cancelled Failure class names vary by SDK: TypeScript CancelledFailure, Java CanceledFailure, Go CanceledError, Python CancelledError, and PHP CanceledFailure.

Cancelled Failure representation and cause

When a Workflow, Activity, or Nexus Operation is successfully cancelled, a Cancelled Failure is the cause field of the Activity Failure, Nexus Operation Failure, or Workflow failed error. In TypeScript, the isCancellation helper checks for cancellation in both direct throws and wrapped cases.

Activity Failure class names by SDK

Activity Failure class names vary by SDK: TypeScript ActivityFailure, Java ActivityFailure, Go ActivityError, Python ActivityError, and PHP ActivityFailure.

Activity Failure structure and cause

An Activity Failure is delivered to the Workflow Execution when an Activity fails. It contains information about the failure and the Activity Execution, such as the Activity Type and Activity Id. The reason for the failure is in the cause field. For example, if an Activity Execution times out, the cause is a Timeout Failure.

Terminated Failure class names by SDK

Terminated Failure class names vary by SDK: TypeScript TerminatedFailure, Java TerminatedFailure, Go TerminatedError, Python TerminatedError, and PHP TerminatedFailure.

Terminated Failure usage location

A Terminated Failure is used as the cause of an error when a Workflow is terminated and received in one of the following locations: inside a Workflow waiting for the result of a Child Workflow, or when waiting for the result of a Workflow on the Client.

Server Failure class names by SDK

Server Failure class names vary by SDK: TypeScript ServerFailure, Java ServerFailure, Go ServerError, Python ServerError, and PHP ServerFailure.

Server Failure origin

A Server Failure is used for errors that originate in the Temporal Service.

Bad Fail Workflow Execution Attributes requires StartToCloseTimeout or ScheduleToCloseTimeout

When Bad Fail Workflow Execution Attributes error occurs, it indicates that the Workflow Task failed due to unset attributes on FailWorkflowExecution command. To resolve this error, make sure that StartToCloseTimeout or ScheduleToCloseTimeout are set, and restart the Worker that the Workflow and Activity are registered to.

gRPC Message Too Large error and 4 MB limit

The gRPC Message Too Large error occurs when the Workflow Task response exceeds the gRPC message size limit of 4 MB. The Workflow Execution is automatically terminated because this is a non-recoverable error. This typically happens when a Workflow schedules too many Activities, Child Workflows, or commands in a single Workflow Task, or when a Workflow returns a large result. To resolve this error, break work into smaller batches, reduce the size of Workflow returns, use Continue-As-New for long-running Workflows, or compress large payloads with a custom Payload Codec.

Bad Cancel Timer Attributes error cause

The Bad Cancel Timer Attributes error indicates that the Workflow Task failed while attempting to cancel a Timer. This occurs due to a missing Timer Id value. To resolve this, check Timer attributes for missing Timer Id, add a valid Timer Id, and redeploy the code.

Bad Cancel Workflow Execution Attributes error cause

The Bad Cancel Workflow Execution Attributes error indicates that the Workflow Task failed due to unset CancelWorkflowExecution attributes. To resolve this, reset any missing attributes and redeploy the Workflow Task.

Bad Complete Workflow Execution Attributes error cause

The Bad Complete Workflow Execution Attributes error indicates that the Workflow Task failed due to unset attributes on CompleteWorkflowExecution. To resolve this, reset any missing attributes and adjust the size of the Payload if it exceeds size limits.

Bad Continue as New Attributes error cause

The Bad Continue as New Attributes error indicates that the Workflow Task failed to validate a ContinueAsNew attribute. The attribute could be unset or invalid. To resolve this, reset any missing attributes, adjust the input size if the payload or memo exceeded size limits, and check that the Workflow is validating search attributes after unaliasing keys.

Bad Modify Workflow Properties Attributes error cause

The Bad Modify Workflow Properties Attributes error indicates that the Workflow Task failed to validate attributes on a property in the Upsert Memo or in a payload. These attributes are either unset or exceeding size limits. To resolve this, reset any unset and empty attributes, and adjust the size of the Memo or payload to fit within the system's limits.

Bad Record Marker Attributes error cause

The Bad Record Marker Attributes error indicates that the Workflow Task failed due to an unset or incorrect Marker name. To resolve this, enter a valid Marker name and redeploy the Task.

Bad Request Cancel Activity Attributes error cause

The Bad Request Cancel Activity Attributes error either indicates the possibility of unset attributes for RequestCancelActivity, or an invalid History Builder state. To resolve this, update the Temporal SDK to the most recent release, reset any unset attributes before retrying the Workflow Task, and if the error continues, review code for nondeterministic causes.

Bad Request Cancel External Workflow Execution Attributes error cause

The Bad Request Cancel External Workflow Execution Attributes error indicates that the Workflow Task failed while trying to cancel an external Workflow due to unset or invalid attributes. To resolve this, reset any missing attributes such as Workflow Id or Run Id, adjust any fields that exceed length limits, and if Child Workflow is set to both Start and RequestCancel, remove one of these attributes as a Child Workflow cannot perform both actions in the same Workflow Task.

Bad Schedule Activity Attributes error cause

The Bad Schedule Activity Attributes error indicates unset or invalid attributes for ScheduleActivityTask. To resolve this, reset any unset or empty attributes and adjust the size of the received payload to stay within the given size limit.

Bad Schedule Nexus Operation Attributes error cause

The Bad Schedule Nexus Operation Attributes error indicates unset or invalid attributes for ScheduleNexusOperation, for example if the Nexus Endpoint name used in the caller Workflow doesn't exist. Inspect the reason given in the error for mitigation when possible.

Bad Search Attributes error cause

The Bad Search Attributes error indicates that the Workflow Task has unset or invalid Search Attributes. This can cause Workflow Tasks to continue to retry without success. To resolve this, make sure that all attributes are defined before retrying the Task and adjust the size of the Payload to fit within the system's size limits.

Bad Signal Input Size error cause

The Bad Signal Input Size error indicates that the Payload has exceeded the Signal's available input size. To resolve this, adjust the size of the Payload and redeploy the Workflow Task.

Bad Signal Workflow Execution Attributes error cause

The Bad Signal Workflow Execution Attributes error indicates that the Workflow Task failed to validate attributes for SignalExternalWorkflowExecution. To resolve this, reset any unset, missing, nil, or invalid attributes and adjust the input to fit within the system's size limits.

Bad Start Child Execution Attributes error cause

The Bad Start Child Execution Attributes error indicates that the Workflow Task failed to validate attributes for StartChildWorkflowExecution. To resolve this, adjust the input size of the attributes to fall within the system's size limits and make sure that Search Attribute validation is performed after unaliasing keys.

Bad Start Timer Attributes error cause

The Bad Start Timer Attributes error indicates that the scheduled Event is missing a Timer Id. To resolve this, set a valid Timer Id and retry the Workflow Task.

Cause Bad Binary error cause

The Cause Bad Binary error indicates that the Worker deployment returned a bad binary checksum.

Cause Bad Update error cause

The Cause Bad Update error indicates that a Workflow Update message (Acceptance, Rejection, or Response) has an invalid format or is missing required fields. WORKFLOW_TASK_FAILED_CAUSE_BAD_UPDATE_WORKFLOW_EXECUTION_MESSAGE can happen when a Worker generates a malformed Update message. This error might indicate usage of an unsupported SDK. Make sure you're using a supported SDK.

Cause Reset Workflow error cause

The Cause Reset Workflow error indicates that the Workflow Task failed due to a request to reset the Workflow. If the system hasn't started a new Workflow, manually reset the Workflow.

Cause Unhandled Update error cause

UnhandledUpdate occurs when a Workflow Update is received by the Temporal Server while a Workflow Task being processed on a Worker produces a Command that would cause the Workflow to transition to a closed state. Temporal rejects the Workflow Task completion to guarantee that the Update is eventually handled by Workflow code and rewinds the Workflow so it can handle the pending Update. This error can happen when the Workflow receives frequent Updates.

Cause Unspecified error cause

The Cause Unspecified error indicates that the Workflow Task has failed for an unknown reason. If you see this error, examine your Workflow Definition.

Failover Close Command error cause

The Failover Close Command error indicates that a Namespace failover forced the Workflow Task to close. The system automatically schedules a retry when this error occurs.

Force Close Command error cause

The Force Close Command error indicates that the Workflow Task was forced to close. A retry will be scheduled if the error is recoverable.

Nondeterminism Error cause

The Nondeterminism Error indicates that the Workflow Task failed due to a nondeterminism error in the Workflow Definition.

Pending Activities Limit Exceeded error cause

The Pending Activities Limit Exceeded error indicates that the Workflow has reached capacity for pending Activities. Therefore, the Workflow Task was failed to prevent the creation of another Activity. To resolve this, let the Workflow complete any current Activities before redeploying the code.

Pending Child Workflows Limit Exceeded error cause

The Pending Child Workflows Limit Exceeded error indicates that the Workflow has reached capacity for pending Child Workflows. Therefore, the Workflow Task was failed to prevent additional Child Workflows from being added. To resolve this, wait for the system to finish any currently running Child Workflows before redeploying this Task.

Pending Nexus Operations Limit Exceeded error cause

The Pending Nexus Operations Limit Exceeded error indicates that the Workflow has reached capacity for pending Nexus Operations. Therefore, the Workflow Task was failed to prevent the creation of another Nexus Operation. To resolve this, let the Workflow complete any current Nexus Operation before retrying the Task.

Pending Request Cancel Limit Exceeded error cause

The Pending Request Cancel Limit Exceeded error indicates that the Workflow Task failed after attempting to add more cancel requests. The Workflow has reached capacity for pending requests to cancel other Workflows. To resolve this, give the system time to process pending requests before retrying the Task.

Pending Signals Limit Exceeded error cause

The Pending Signals Limit Exceeded error indicates that the Workflow has reached capacity for pending Signals. Therefore, the Workflow Task was failed after attempting to add more Signals to an external Workflow. To resolve this, wait for Signals to be processed by the Workflow before retrying the Task.

Reset Sticky Task Queue error cause

The Reset Sticky Task Queue error indicates that the Sticky Task Queue needs to be reset. To resolve this, reset the Sticky Task Queue. The system will retry automatically.

Resource Exhausted Cause Concurrent Limit error cause

The Resource Exhausted Cause Concurrent Limit error indicates that the concurrent poller count has been exhausted. To resolve this, adjust the poller count per Worker.

Resource Exhausted Cause Persistence Limit error cause

The Resource Exhausted Cause Persistence Limit error indicates that the persistence rate limit has been reached.

Resource Exhausted Cause RPS Limit error cause

The Resource Exhausted Cause RPS Limit error indicates that the Workflow has exhausted its RPS limit.

Give your agent this brain