activities/error-handling

94 notes in this subject, read out of this brain and free to use. This is page 2 of 2.

Activity cancellation handling in Python

When an Activity is canceled, an asyncio.CancelledError is raised. Cleanup logic can be performed in a finally clause or inside a caught cancel error. However, for the Activity to appear canceled, the exception must be re-raised.

Activity cancellation requires Heartbeats and Heartbeat Timeout

For a non-local Activity to receive a cancellation request, the Activity Execution must send Heartbeats and set a Heartbeat Timeout. If Heartbeats are not invoked, the Activity cannot receive a cancellation request. Local Activities, however, can be canceled without sending Heartbeats because they are handled locally in the same Worker process.

Ruby SDK benign exception example

To mark an error as benign in the Ruby SDK, raise an ApplicationError with the category parameter set to Temporalio::Error::ApplicationError::Category::BENIGN. This example shows catching a StandardError from an external service call and wrapping it as a benign ApplicationError: ```ruby require 'temporalio/activity' class MyActivity < Temporalio::Activity::Definition def execute begin call_external_service rescue StandardError => e # Mark this error as benign since it's expected raise Temporalio::Error::ApplicationError.new( e.message, category: Temporalio::Error::ApplicationError::Category::BENIGN ) end end end ```

Mark Activity errors as benign to reduce observability noise

When Activities throw expected or non-severe errors, marking them as benign excludes them from logs, metrics, and OpenTelemetry traces. Benign errors have their Activity failure logs downgraded to DEBUG level, do not emit Activity failure metrics, and do not set the OpenTelemetry failure status to ERROR.

Override retry delay with next_retry_delay in Ruby

When raising an application-level error in a Ruby activity, you can override the Retry Policy's delay by specifying next_retry_delay in the ApplicationError. This allows dynamic delay calculation based on attempt number using Temporalio::Activity::Context.current.info.attempt.

Default activity retry policy in Ruby

Activities in the Ruby SDK use a system Retry Policy by default. This can be overridden by specifying a custom Retry Policy using the retry_policy parameter.

Create activity retry policy in Ruby SDK

To create an Activity Retry Policy in Ruby, set the retry_policy parameter when executing an activity. Pass Temporalio::RetryPolicy.new with configuration options such as max_interval.

ApplicationError for Temporal-aware exception handling

In the Ruby SDK, raise Temporalio::Error::ApplicationError to indicate a Workflow or Activity failure that should be handled by Temporal. Any other exceptions raised from Activity code are automatically converted to ApplicationError internally, allowing error type, severity, and details to be sent to the Temporal Service, indexed by the Web UI, and serialized across language boundaries.

CancelledError should not be raised manually

Temporal error classes like Temporalio::Error::CanceledError are used internally by Temporal for platform logic such as Workflow cancellation. You should not raise or implement these manually, as they are tied to Temporal platform logic.

ApplicationError non_retryable parameter

When raising Temporalio::Error::ApplicationError, you can set the non_retryable parameter to true to prevent Temporal from automatically retrying the activity or workflow. This is useful for deliberately failing due to bad input data rather than waiting for a timeout.

Activity exceptions converted to ApplicationError example

Raising a custom Ruby exception like 'raise MyError.new("Simulated failure")' from an Activity is equivalent to raising Temporalio::Error::ApplicationError.new('Simulated failure', type: 'MyError'). Both approaches result in the same behavior, with the type parameter allowing you to specify the original error class name.

Handling Activity Failure with rescue in Workflow

Example: wrap Temporalio::Workflow.execute_activity in a rescue block. When an Activity fails, catch the exception with rescue StandardError and then raise Temporalio::Error::ApplicationError to deliberately fail the Workflow if that Activity failure means the Workflow should not continue.

Activity error handling with ApplicationFailure in Rust

The return type of an Activity is Result<T, ActivityError>. Use ApplicationFailure::new for errors that should be retried, and ApplicationFailure::non_retryable for permanent failures that should not be retried.

Activity error handling with retryable and non-retryable errors example

#[derive(serde::Serialize, serde::Deserialize, Debug, Clone, PartialEq, Eq, Hash)] pub struct ProcessedData { pub processed: String, } #[activities] impl MyActivities { #[activity] pub async fn process_data( _ctx: ActivityContext, input: String, ) -> Result<ProcessedData, ActivityError> { // If an error should be retried if !validate_input(&input) { return Err(ApplicationFailure::builder("Invalid input format") .next_retry_delay(Duration::from_secs(5)) .build() .into()); } // If an error should not be retried if input.len() > 1000000 { return Err(ApplicationFailure::non_retryable("Input too large").into()); } let result = ProcessedData { processed: input.to_uppercase(), }; Ok(result) } }

Configure Activity Retry Policy in Rust

In Rust, configure a Retry Policy as part of Activity options when scheduling an Activity from Workflow code using ActivityOptions::with_start_to_close_timeout() chained with .retry_policy() and .build().

Override retry interval with explicit_delay in Rust

To override the next retry interval set by the current Retry Policy, return a failure from an Activity with a custom next retry delay. Return an ApplicationFailure from the Activity with .next_retry_delay(Duration) to set a custom delay that replaces the interval the Retry Policy would otherwise use. This is useful when retry timing depends on runtime state such as the current attempt number.

Activity cancellation error handling in Rust

When an Activity is canceled, an error is returned in the Activity at the next available opportunity. If cleanup logic needs to be performed, it can be done when handling the cancellation error. However, for the Activity to appear canceled, the error must be propagated.

Use benign exceptions for expected Activity failures

Use benign exceptions for Activity errors that occur regularly as part of normal operations, such as polling an external service that is not ready yet, or handling expected transient failures that will be retried.

Mark errors benign with ApplicationFailureCategory.BENIGN

To mark an error as benign, set the category field to ApplicationFailureCategory.BENIGN when creating an ApplicationFailure object in the TypeScript SDK.

Effects of benign exceptions on observability

Benign exceptions have three observability effects: Activity failure logs are downgraded to DEBUG level, Activity failure metrics are not emitted, and the OpenTelemetry failure status is not set to ERROR.

Create benign exception in Activity (TypeScript example)

import { ApplicationFailure, ApplicationFailureCategory, } from '@temporalio/common'; export async function myActivity(): Promise<string> { try { return await callExternalService(); } catch (err) { const message = err instanceof Error ? err.message : String(err); throw ApplicationFailure.create({ message, // Mark this error as benign since it's expected category: ApplicationFailureCategory.BENIGN, }); } }

Benign exceptions reduce observability noise

Benign exceptions are expected or non-severe Activity errors that create noise in logs, metrics, and OpenTelemetry traces. Marking errors as benign helps exclude them from observability data while still handling them in Workflow logic.

Override next retry delay with ApplicationFailure

An Activity can override the next retry delay determined by its Retry Policy by explicitly failing with an ApplicationFailure. To override the next retry delay, provide the nextRetryDelay property on the object argument of ApplicationFailure.create(). Example: throw ApplicationFailure.create({ nextRetryDelay: '15s', });

Configure Activity Retry Policy in TypeScript

To set an Activity's Retry Policy in TypeScript, assign the ActivityOptions.retry property when creating the Activity proxy function using proxyActivities(). Example: const { myActivity } = proxyActivities<typeof activities>({ retry: { initialInterval: '10s', maximumAttempts: 5, }, });

Activity Retry Policy defaults

Activity Executions are automatically associated with a default Retry Policy if a custom one is not provided.

ApplicationFailure for error handling in TypeScript

In the TypeScript SDK, you can raise the ApplicationFailure class to handle errors idiomatically. Any other exceptions raised from TypeScript code in a Temporal Activity will be converted to an ApplicationError internally. ApplicationFailure allows you to set a message, type, and non-retryable parameter. The non-retryable parameter lets you decide whether an error should not be retried automatically by Temporal, which is useful for deliberately failing a Workflow due to bad input data.

ApplicationFailure with nonRetryable parameter example

To prevent automatic retry of a Workflow due to bad input data, use ApplicationFailure with the nonRetryable parameter set to true. Example: ApplicationFailure.create({ message: 'Invalid charge amount: ${chargeAmount} (must be above zero)', nonRetryable: true })

CancelledFailure and built-in Temporal error classes

Temporal uses several different error classes internally, such as CancelledFailure in the TypeScript SDK, to handle a Workflow cancellation. These error classes are tied to Temporal platform logic and should not be raised or manually implemented.

High Activity retry volumes indicate underlying issues

Excessive Activity retries often indicate underlying issues like timeouts that are too short or Activities that frequently fail. Detect Activity retry frequency and if high, consider increasing retry intervals or Activity timeouts before failures occur.

Default Retry Policy can be aggressive for expensive external operations

With Temporal's default Retry Policy (1s initial interval, 2.0 backoff coefficient, unlimited maximum attempts), if an external API goes down for an extended period, the Activity retries many times before reaching the 100s maximum interval cap, then continues retrying every 100 seconds. Each retry counts as 1 Action. For expensive external operations like payment APIs, consider setting MaximumAttempts to cap total retries, increasing InitialInterval to reduce retry frequency, adding error types to NonRetryableErrorTypes for errors that won't resolve on retry (such as 4xx HTTP status codes), using next retry delay to dynamically control retry timing based on failure types, or implementing an Activity pause pattern to wait for manual intervention rather than automatic retries.

Effects of marking an Activity error as benign

When an Activity error is marked as benign, the following effects occur: Activity failure logs are downgraded to DEBUG level, no Activity failure metrics are emitted, and the OpenTelemetry failure status is not set to ERROR.

.NET explicit activity cancellation example

Example showing how to cancel an activity explicitly in a .NET Workflow: ```csharp [WorkflowRun] public async Task RunAsync() { using var cancelActivitySource = CancellationTokenSource.CreateLinkedTokenSource( Workflow.CancellationToken); var activityTask = Workflow.ExecuteActivityAsync( (MyActivities a) => a.MyNormalActivity(), new() { ScheduleToCloseTimeout = TimeSpan.FromMinutes(5), CancellationToken = cancelActivitySource.Token; }); activityTask.Start(); await Workflow.DelayAsync(TimeSpan.FromMinutes(5)); cancelActivitySource.Cancel(); await activityTask; } ```

.NET activity heartbeat and cancellation example

Example showing how to handle cancellation in a .NET Activity with heartbeating: ```csharp [Activity] public async Task MyActivityAsync() { while (true) { ActivityExecutionContext.Current.Heartbeat(); await Task.Delay(1000, ActivityExecutionContext.Current.CancellationToken); } } ```

Cancel activity explicitly in .NET Workflow

By default, Activities are automatically cancelled when the Workflow is cancelled since the workflow cancellation token is used by activities by default. To issue a cancellation explicitly, create a new cancellation token using CancellationTokenSource.CreateLinkedTokenSource(Workflow.CancellationToken) to link it to workflow cancellation, then pass the token to the activity options. You can then call cancelActivitySource.Cancel() to cancel the activity independently.

Give your agent this brain