error-handling

109 notes in this subject, read out of this brain and free to use. This is page 2 of 2.

Nexus Operation failure states

Nexus Operations can end up in completed, failed, canceled, and timed out states. The Workflow-specified schedule-to-close timeout is enforced by the caller's machinery and is the only way for an Operation to transition to timed out state. Operations can end up in other three states either when the operation handler returns a synchronous response or error, or when an asynchronous Operation eventually reaches a terminal state.

Nexus Operation handler error retryability

A Nexus Operation handler can return either retryable or non-retryable errors. By default, errors are considered retryable unless specified as non-retryable. Non-retryable Application Failures, unsuccessful Operation errors, and handler errors with types BAD_REQUEST, UNAUTHENTICATED, UNAUTHORIZED, NOT_FOUND, and RESOURCE_EXHAUSTED are not retryable.

Nexus Operation Task Failure definition

A Nexus Operation Task Failure is an unexpected situation failing to process a Nexus Operation Task in a handler, triggered by throwing an unknown error in Nexus handler code. These failures cause the Nexus Operation Task to be retried.

Nexus Operation Execution Failure

A non-retryable Application Failure can be thrown by a Nexus Operation handler to fail the overall Nexus Operation Execution. Nexus Operation Execution Failures put the Nexus Operation Execution into the 'Failed' state and no more attempts will be made to complete the Nexus Operation.

Workflow errors in Nexus NewWorkflowRunOperation propagation

Application Errors thrown from a Workflow created by a Nexus NewWorkflowRunOperation handler are automatically propagated to the caller as a non-retryable error and result in a Nexus Operation Execution Failure.

Nexus handler error conversion to ApplicationFailure

Unknown errors in a Nexus Operation handler are converted to a retryable Application Failure. During conversion, the following fields are set on the Application Failure: non_retryable is set to false, type is set to the error's type name, message is set to the error message.

Retryable Nexus Operation Task failures event recording

When a Nexus Task fails with a retryable failure (such as an unknown error), the caller Workflow records an event attempt failure on the pending Nexus Operation with the following fields: state is set to the new state (e.g., BackingOff), attempt is set to an incremented count, next_attempt_schedule_time is set when the Nexus Task will be retried, last_attempt_failure is set with message and failure_info (set to the Application Failure).

Nexus Operation handler non-retryable failures

When a Nexus Operation handler throws an Application Failure with non_retryable set to true, the overall Nexus Operation Execution is failed and the error is returned to the caller's Workflow Execution as a Nexus Operation Failure.

Nexus error default mapping for Application Failures

Application Failures thrown from a Nexus Operation handler are mapped to Nexus Failures based on non_retryable flag: if non_retryable is false (default), maps to HandlerErrorTypeInternal with HTTP status 500 Internal Server Error; if non_retryable is true, maps to UnsuccessfulOperationError with HTTP status 424 Failed Dependency.

Non-retryable Nexus error types

Nexus error types that are non-retryable (non_retryable = true): HandlerErrorTypeBadRequest, HandlerErrorTypeUnauthenticated, HandlerErrorTypeUnauthorized, HandlerErrorTypeNotFound, UnsuccessfulOperationError.

CancelledFailure definition and usage

When Cancellation of a Workflow, Activity or Nexus Operation is requested, SDKs represent cancellation to the user in language-specific ways. When a Workflow, Activity or Nexus Operation is successfully Cancelled, a Cancelled Failure is the cause field of the Activity Failure, Nexus Operation Failure or 'Workflow failed' error.

Nexus Operation Failure definition

A Nexus Operation Failure is delivered to the Workflow Execution when a Nexus Operation fails. It contains information about the failure and the Nexus Operation Execution (e.g., Nexus Operation name and token). The reason for the failure is in the message and cause (typically an Application Error or a Canceled Error).

Nexus Operation Failure fields

A Nexus Operation Failure includes the following fields: Endpoint (name of the endpoint), Service (name of the service), Operation (name of the operation), Operation_token (set if async operation, can be used for additional actions like cancelling), Scheduled_event_id (caller's event id that scheduled the operation), Message (generic unsuccessful error message), Cause (underlying Application Failure with non_retryable true, type set to error's type name, message set to error message), Nexus_error_code (underlying Nexus error code).

Child Workflow Failure definition

A Child Workflow Failure is delivered to the Workflow Execution when a Child Workflow Execution fails. It contains information about the failure and the Child Workflow Execution (e.g., Workflow Type and Workflow Id). The reason for the failure is in the cause field.

Timeout Failure definition

A Timeout Failure represents the timeout of an Activity or Workflow. When an Activity times out, the last Heartbeat details it emitted is attached.

Terminated Failure definition and usage

A Terminated Failure is used as the cause of an error when a Workflow is terminated. You receive the error inside a Workflow that is waiting for the result of a Child Workflow, or when waiting for the result of a Workflow on the Client.

Server Failure definition

A Server Failure is used for errors that originate in the Temporal Service.

Payload size limit error messages

Error messages for oversized payloads include: WORKFLOW_TASK_FAILED_CAUSE_PAYLOADS_TOO_LARGE, [TMPRL1103] Attempted to upload payloads with size that exceeded the error limit, BadScheduleActivityAttributes: ScheduleActivityTaskCommandAttributes.Input exceeds size limit, Complete result exceeds size limit, CompleteWorkflowExecutionCommandAttributes.Result exceeds size limit, and WORKFLOW_TASK_FAILED_CAUSE_BAD_UPDATE_WORKFLOW_EXECUTION_MESSAGE.

Payload size limit error behavior: Python SDK 1.23.0+ and Go SDK 1.43.0+

In Python SDK 1.23.0 and later and Go SDK 1.43.0 and later, when a payload exceeds the size limit, the SDK fails the Workflow Task with cause WORKFLOW_TASK_FAILED_CAUSE_PAYLOADS_TOO_LARGE. The Workflow is not terminated and remains open, so a fix can be deployed and the Workflow can continue. For Activities, the Activity Task fails with an explicit error instead of timing out silently.

Payload size limit error behavior: Other SDK versions, input payloads

For SDK versions other than Python 1.23.0+ and Go 1.43.0+, when an oversized payload is a Workflow or Activity input, the Temporal Service rejects the command and terminates the Workflow. The issue must be resolved and the Workflow must be restarted.

Payload size limit error behavior: Other SDK versions, Activity result

For SDK versions other than Python 1.23.0+ and Go 1.43.0+, when an oversized payload is an Activity result, the Temporal Service rejects the Activity completion and the Activity fails with an error.

Payload size limit error behavior: Other SDK versions, Workflow result

For SDK versions other than Python 1.23.0+ and Go 1.43.0+, when an oversized payload is a Workflow result, the Workflow gets stuck in a retry loop. The server rejects the CompleteWorkflowExecution command, and replay produces the same oversized result.

gRPC message size limit can be hit with multiple moderate-sized payloads

A Workflow can hit the 4 MB gRPC message size limit even when every individual payload is under 2 MB. Scheduling several Activities with moderate-sized inputs, or hundreds of Activities with tiny inputs in the same Workflow Task can push the combined request past 4 MB. Activity results are also subject to this limit.

gRPC message size limit error messages

Error messages for oversized gRPC messages include: WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE and ScheduleToCloseTimeout (Activities only).

gRPC message size limit error behavior: Workflow Tasks

When a Workflow Task produces a combined request exceeding 4 MB, the Temporal Service terminates the Workflow Execution. The SDK catches the gRPC error and sends a failed Workflow Task response with cause WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE. Because replay would produce the same oversized request on every attempt, the Temporal Service ends the Execution with TERMINATED status rather than retrying it. Terminated Executions do not retry automatically, so fix the oversized payload and deploy the corrected Worker before restarting the Workflow.

gRPC message size limit error behavior: Activity Tasks

When an Activity result is too large for gRPC, the Activity gets stuck in a retry loop or exits with a ScheduleToCloseTimeout. The Activity executes successfully, but the Worker cannot deliver the oversized result over gRPC. The server never receives the completion, so it retries the Activity. Each retry completes successfully but fails to deliver the result. The Activity retries until the ScheduleToCloseTimeout expires. If no ScheduleToCloseTimeout is set, the Activity retries indefinitely until the Workflow is manually terminated. The ResourceExhausted gRPC error only appears in Worker logs.

Timing skew causing Activity timeout errors

Timing skew can cause the system clock on a Worker to drift behind the system clock of the Temporal Service. If the difference between the two clocks exceeds an Activity's Start-To-Close Timeout, an 'Activity complete after timeout' error occurs. To fix this, synchronize all clocks to an NTP server.

Check gRPC service health with grpc-health-probe

Use grpc-health-probe to check the Frontend Service, Matching Service, and History Service. Command for Frontend: ./grpc-health-probe -addr=frontendAddress:frontendPort -service=temporal.api.workflowservice.v1.WorkflowService. Command for Matching Service: ./grpc-health-probe -addr=matchingAddress:matchingPort -service=temporal.api.workflowservice.v1.MatchingService. Command for History Service: ./grpc-health-probe -addr=historyAddress:historyPort -service=temporal.api.workflowservice.v1.HistoryService

Detect resource exhausted errors causing deadline exceeded

Resource exhausted errors can cause client requests to fail and prompt deadline exceeded errors. Use the Prometheus query: sum(rate(service_errors_resource_exhausted{}[1m])) by (resource_exhausted_cause) to check for errors in RpsLimit, ConcurrentLimit and SystemOverloaded on metrics dashboard.

Invalid configuration causes deadline exceeded and connection refused

Invalid target values in Client and Worker configuration files cause deadline exceeded errors. Check for missing or incorrect server names, network or host addresses, and certificates. Invalid targets also cause 'connection refused' errors alongside deadline exceeded.

Increase frontend.keepAliveMaxConnectionAge for workflow execution

When executing or scheduling Workflows, if one or more services cannot connect to the Frontend Service and the Workflow might be unable to complete requests within the given connection time, increase the value of frontend.keepAliveMaxConnectionAge so that requests can be finished before the connection terminates. Monitor server performance for load after increasing this value.

Allow time for Temporal Service reconnection after restart

After restarting the Temporal Service, provide enough time for the service to respond and reconnect. Restart the Server, wait, and then check all services for connectivity and further errors. If the error persists, review Workflow Execution History and server logs for more specific causes.

BlobSizeLimitError maximum payload sizes

The BlobSizeLimitError occurs when the size of a blob (payloads including Workflow context and each Workflow and Activity argument and return value) is too large. The maximum payload for a single request is 2 MB, and the maximum size for any Event History transaction is 4 MB.

Context: deadline exceeded error causes

The 'Context: deadline exceeded' error occurs when requests to the Temporal Service by the Client or Worker cannot be completed. This can be due to network issues, timeouts, server overload, or Query errors.

Troubleshoot missed Schedule Actions

When a Schedule does not fire at its expected time, alert on the missed catchup window metric, then narrow down to the affected Schedule with ListSchedules and DescribeSchedule.

gRPC request failure types

Common gRPC failures from Workers and Clients to the Temporal Service include NOT_FOUND on respond operations, RESOURCE_EXHAUSTED throttling, and UNIMPLEMENTED or INTERNAL responses.

Workflow and Activity execution failure types

Common Workflow and Activity execution failures include non-determinism errors, oversized Workflow Task responses, unhandled exceptions in Workflow and Activity code, and Local Activities running past the Workflow Task heartbeat timeout.

ApplicationError for business logic validation in Updates

Raise ApplicationError with non_retryable=True in Update handlers for business logic validation errors (e.g., insufficient points, inactive account). Set appropriate error types to help clients distinguish between retryable and non-retryable failures.

WorkflowUpdateFailedError exception handling

When executing Updates, catch WorkflowUpdateFailedError to handle failures. This exception is raised when an Update encounters an error during execution, such as validation failures or application errors raised in the Update handler.

WorkflowError failure reason covers multiple failure modes

WorkflowError is the catch-all failure reason for Workflow Task failures. It covers thread pool exhaustion, unhandled exceptions thrown inside the Workflow function, and Data Converter errors.

Distinguish Code Converter error from Workflow code error

Data Converter errors that cause WorkflowError failures can result from something failing during serialization or deserialization of Workflow inputs, outputs, or Memo fields. Check Data Converter and Payload Codec configuration to diagnose these.

NOT_FOUND on respond operations causes

A Worker finished a Workflow Task or Activity Task and reported the result, and the Temporal Service replied that the task no longer exists. Three causes: (1) The task timed out—the Worker ran past the Workflow Task timeout, or past the Activity `startToClose` or `scheduleToClose` timeout, and the Service discarded the in-flight task. (2) The Workflow Execution is no longer running—it completed, was terminated, or hit its Workflow Run Timeout before the task finished. (3) The Worker restarted mid-execution—the in-flight Task Token was lost, the Service rescheduled the task, and the original Worker still attempted to respond after coming back up.

NOT_FOUND response discards Worker results

When the Temporal Service returns NOT_FOUND on a respond operation, it throws away the result your Worker just produced. For Activities, the Service has already rescheduled the Activity for retry if the Retry Policy allows it. For Workflow Tasks, the Service writes a `WorkflowTaskTimedOut` event to Event History and reschedules the task on the normal Task Queue, which forces a Sticky Execution cache eviction and a cold replay on the retry.

RESOURCE_EXHAUSTED on user-facing operations auto-retries for 60 seconds

When the Temporal Service is throttling user-facing operations (StartWorkflowExecution, SignalWithStartWorkflowExecution, SignalWorkflowExecution, UpdateWorkflowExecution, or ExecuteMultiOperation), the SDK retries these automatically for up to 60 seconds. Beyond that, the call fails and the error propagates to your caller.

RESOURCE_EXHAUSTED causes on user-facing operations

Four different causes for RESOURCE_EXHAUSTED on user-facing operations: (1) Namespace rate limit (`RESOURCE_EXHAUSTED_CAUSE_RPS_LIMIT` or `RESOURCE_EXHAUSTED_CAUSE_APS_LIMIT`), (2) concurrency limit, (3) system-wide overload (`RESOURCE_EXHAUSTED_CAUSE_SYSTEM_OVERLOADED`), and (4) open circuit breaker (`RESOURCE_EXHAUSTED_CAUSE_CIRCUIT_BREAKER_OPEN`). On the Go SDK, group `temporal_request_resource_exhausted` by its `cause` tag to identify which limit was hit.

Dropped starts and signals risk with RESOURCE_EXHAUSTED

If application code does not handle RESOURCE_EXHAUSTED errors on user-facing operations after the 60-second retry window, starts and Signals are silently dropped. A dropped start means the Workflow never runs. A dropped Signal or Update means a running Workflow never receives input it is waiting on, and may stall indefinitely. Log these failures in your application code so you can backfill starts and Signals afterward.

RESOURCE_EXHAUSTED on respond operations indicates throttling cascade risk

RESOURCE_EXHAUSTED on respond operations (RespondWorkflowTaskCompleted, RespondWorkflowTaskFailed, RespondActivityTaskCompleted, RespondActivityTaskFailed) is a leading indicator of NOT_FOUND on respond operations. If throttling goes on long enough the task times out, the Service writes a `WorkflowTaskTimedOut` or `ActivityTaskTimedOut` event, and the task gets rescheduled.

UNIMPLEMENTED indicates Service does not recognize operation

`UNIMPLEMENTED` status code means the Temporal Service does not recognize an operation the Worker called. By the time a Worker reaches steady-state polling it has already called `GetSystemInfo` and `DescribeNamespace` successfully, so this is rarely a plain version mismatch on a freshly deployed Worker. Most SDK versions treat UNIMPLEMENTED as non-retryable: the Worker surfaces it as a fatal error and may shut down.

INTERNAL error from Temporal Service

`INTERNAL` status code means the Temporal Service encountered an error it could not attribute to the request. Short bursts during Service restarts and rolling deploys are normal. The SDK retries INTERNAL, but sustained errors exhaust the retry budget and surface to callers. Workers receiving INTERNAL on poll operations back off and poll less frequently, which raises schedule-to-start latency.

Give your agent this brain