activities/error-handling

94 notes in this subject, read out of this brain and free to use. This is page 1 of 2.

Benign exceptions reduce observability noise

By marking Activity errors as benign, you can exclude them from logs, metrics, and OpenTelemetry traces while still handling them in Workflow logic. This makes it easier to identify real issues in your observability data.

ApplicationFailureException with benign category example

Example of marking an Activity error as benign: throw new ApplicationFailureException("Service is down", inner: e, category: ApplicationErrorCategory.Benign);

HeartbeatTimeout in ActivityOptions .NET

HeartbeatTimeout is a property on ActivityOptions for ExecuteActivityAsync used to set the maximum time between Activity Heartbeats. It is specified as a TimeSpan value.

Setting Activity Timeouts with ActivityOptions in .NET

Activity timeouts are set using the ActivityOptions parameter when calling ExecuteActivityAsync. The available timeout properties are: ScheduleToCloseTimeout, StartToCloseTimeout, and ScheduleToStartTimeout, all specified as TimeSpan values.

Activity Timeout example in .NET

Example of setting StartToCloseTimeout for an activity: ```csharp return await Workflow.ExecuteActivityAsync( (MyActivities a) => a.MyActivity(param), new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) }); ```

Activity Retry Policy in .NET

A Retry Policy works with timeouts to provide fine controls for optimizing execution experience. Activity Executions are automatically associated with a default Retry Policy if a custom one is not provided. To create a custom Activity Retry Policy, set the RetryPolicy property on ActivityOptions when calling ExecuteActivityAsync.

Activity Retry Policy example in .NET

Example of setting a custom Retry Policy with MaximumInterval: ```csharp return await Workflow.ExecuteActivityAsync( (MyActivities a) => a.MyActivity(param), new() { StartToCloseTimeout = TimeSpan.FromMinutes(5), RetryPolicy = new() { MaximumInterval = TimeSpan.FromSeconds(10) }, }); ```

nextRetryDelay example in .NET activity

Example of using nextRetryDelay to scale retry delay by attempt count: ```csharp var attempt = ActivityExecutionContext.Current.Info.Attempt; throw new ApplicationFailureException( $"Something bad happened on attempt {attempt}", errorType: "my_failure_type", nextRetryDelay: TimeSpan.FromSeconds(3 * attempt)); ```

Override retry interval with nextRetryDelay in .NET

When throwing an ApplicationFailure, you can assign the nextRetryDelay field to override the Retry interval defined in the active Retry Policy. This allows dynamic calculation of the next retry delay based on context, such as scaling based on the current attempt number.

HeartbeatTimeout example in .NET

Example of setting HeartbeatTimeout in ActivityOptions: ```csharp await Workflow.ExecuteActivityAsync( (MyActivities a) => a.MyActivity(param), new() { StartToCloseTimeout = TimeSpan.FromMinutes(5), HeartbeatTimeout = TimeSpan.FromSeconds(30), }); ```

Non-retryable errors in Retry Policy

You can specify a list of errors that are non-retryable in an Activity Retry Policy as an alternative to using the nonRetryable parameter on ApplicationFailureException.

Do not raise Temporal internal error classes manually

You should not raise or implement Temporal internal error classes manually, such as CancelledFailureException in the .NET SDK used to handle Workflow cancellation. These error classes are tied to Temporal platform logic and should only be raised by the platform itself.

ApplicationFailureException for deliberate errors

ApplicationFailureException is the one Temporal error class that you will typically raise deliberately. Any other exceptions raised from C# code in a Temporal Activity will be converted to an ApplicationFailureException internally. This allows an error's type, severity, and additional details to be sent to the Temporal Service, indexed by the Web UI, and serialized across language boundaries.

ApplicationFailureException nonRetryable parameter

ApplicationFailureException allows you to set a nonRetryable parameter to decide whether an error should not be retried automatically by Temporal. This is useful for deliberately failing a Workflow due to bad input data rather than waiting for a timeout to elapse. Example: throw new ApplicationFailureException("Invalid department", nonRetryable: true);

Handle Activity failures in Workflow try-catch

You can catch ActivityFailureException in a Workflow using try-catch blocks to handle errors returned by Activities the way you would in any other program. For example, you could implement a Saga Pattern that uses try-catch blocks to unwind steps your Workflow has performed up to the point of Activity Failure.

Failing a Workflow deliberately

You will only fail a Workflow by manually raising an ApplicationFailureException from the Workflow code. This is typically done in response to an Activity Failure when the failure of that Activity means the Workflow should not continue.

Exceptions in Workflows vs Activities differ in handling

In an Activity, any C# exceptions or custom exceptions are converted to a Temporal ApplicationError. In a Workflow, any exceptions raised other than an explicit Temporal ApplicationError will only fail that particular Workflow Task and be retried. This includes typical C# RuntimeErrors. These errors are treated as bugs that can be corrected with a fixed deployment, rather than a reason for a Temporal Workflow Execution to return unexpectedly.

Custom exceptions converted to ApplicationFailureException

Custom exceptions thrown from C# code in a Temporal Activity are automatically converted to ApplicationFailureException internally. You can either throw a custom exception directly or explicitly throw ApplicationFailureException with an errorType parameter matching the exception name. Both approaches achieve the same result.

Example: catching Activity failure and failing Workflow

try { await Workflow.ExecuteActivityAsync((Activities act) => act.ValidateCreditCardAsync(order.Customer.CreditCardNumber), options); } catch (ActivityFailureException err) { logger.LogError("Unable to process credit card: {Message}", err.Message); throw new ApplicationFailureException(message: "Invalid credit card number error"); }

Activity failure handling

If an Activity fails, Temporal automatically retries it based on your retry policy configuration.

Example: Activity returning benign error with temporal.NewApplicationErrorWithOptions

import ( "go.temporal.io/sdk/activity" "go.temporal.io/sdk/temporal" ) func MyActivity(ctx context.Context) (string, error) { result, err := callExternalService() if err != nil { // Mark this error as benign since it's expected return "", temporal.NewApplicationErrorWithOptions( err.Error(), "", temporal.ApplicationErrorOptions{ Category: temporal.ApplicationErrorCategoryBenign, }, ) } return result, nil }

Mark Activity errors as benign with temporal.NewApplicationErrorWithOptions

To mark an Activity error as benign in the Go SDK, use temporal.NewApplicationErrorWithOptions() and set the Category field to temporal.ApplicationErrorCategoryBenign in the ApplicationErrorOptions. This suppresses observability noise for expected or non-severe errors.

Activity default retry policy in Go

If a custom Retry Policy is not provided, Activity Executions are automatically associated with a default RetryPolicy. The default values are: InitialInterval of 1 second, BackoffCoefficient of 2.0, MaximumInterval of 100 seconds (100 * InitialInterval), MaximumAttempts of 0 (unlimited), and NonRetryableErrorTypes as an empty list.

Activity retry policy example in Go

retrypolicy := &temporal.RetryPolicy{ InitialInterval: time.Second, BackoffCoefficient: 2.0, MaximumInterval: time.Second * 100, } activityoptions := workflow.ActivityOptions{ RetryPolicy: retrypolicy, } ctx = workflow.WithActivityOptions(ctx, activityoptions) var yourActivityResult YourActivityResult err = workflow.ExecuteActivity(ctx, YourActivityDefinition, yourActivityParam).Get(ctx, &yourActivityResult)

Setting custom activity retry policy in Go

To set a custom RetryPolicy for activities, create an instance of the temporal.RetryPolicy type with desired fields, set it in the ActivityOptions RetryPolicy field, and apply it using WithActivityOptions(). Providing a custom Retry Policy overwrites all individual field defaults.

Override retry interval with NextRetryDelay in Go

An Application Failure can be returned with the NextRetryDelay field set, which will override whatever retry interval the Retry Policy would use. This allows the Activity to customize the retry delay based on runtime conditions such as the current attempt number.

NextRetryDelay example in Go activity

attempt := activity.GetInfo(ctx).Attempt return temporal.NewApplicationErrorWithOptions(fmt.Sprintf("Something bad happened on attempt %d", attempt), "NextDelay", temporal.ApplicationErrorOptions{ NextRetryDelay: 3 * time.Second * delay, })

Timeout error types in workflows

When handling a *TimeoutError in a Workflow, you can check the timeout type using timeoutErr.TimeoutType(). The timeout types include commonpb.ScheduleToStart, commonpb.StartToClose, and commonpb.Heartbeat.

defer and recover limitations in Temporal workflows

In Temporal Workflow code, you cannot recover() from a panic inside a defer, and deferred functions that try to interact with the Temporal SDK during panic unwinding will re-panic immediately. Use defer only for local cleanup. Handle Temporal API cleanup through explicit error checks instead.

Activity error handling code example

This example shows how to handle different Activity error types in Workflow code: ```go err := workflow.ExecuteActivity(ctx, YourActivity, ...).Get(ctx, nil) if err != nil { var applicationErr *ApplicationError if errors.As(err, &applicationErr) { // retrieve error message workflow.GetLogger(ctx).Info("Application error", "error", applicationErr.Error()) // handle Activity errors (created via NewApplicationError() API) var detailMsg string // assuming Activity return error by NewApplicationError("message", true, "string details") applicationErr.Details(&detailMsg) // extract strong typed details // handle Activity errors (errors created other than using NewApplicationError() API) switch applicationErr.Type() { case "CustomErrTypeA": // handle CustomErrTypeA case CustomErrTypeB: // handle CustomErrTypeB default: // newer version of Activity could return new errors that Workflow was not aware of. } } var canceledErr *CanceledError if errors.As(err, &canceledErr) { // handle cancellation } var timeoutErr *TimeoutError if errors.As(err, &timeoutErr) { // handle timeout, could check timeout type by timeoutErr.TimeoutType() switch err.TimeoutType() { case commonpb.ScheduleToStart: // Handle ScheduleToStart timeout. case commonpb.StartToClose: // Handle StartToClose timeout. case commonpb.Heartbeat: // Handle heartbeat timeout. default: } } var panicErr *PanicError if errors.As(err, &panicErr) { // handle panic, message and call stack are available by panicErr.Error() and panicErr.StackTrace() } } ```

Error types returned by activities

When an Activity returns an error as errors.New() or fmt.Errorf(), that error is converted into *temporal.ApplicationError. When an Activity returns an error as temporal.NewNonRetryableApplicationError("error message", details), that error is returned as *temporal.ApplicationError. Other error types include *temporal.TimeoutError, *temporal.CanceledError, and *temporal.PanicError.

Handling Activity errors in workflows

Within a Workflow, an Activity or Child Workflow execution might fail. Errors can be handled differently based on the error type. Use errors.As() to check for specific error types like *ApplicationError, *CanceledError, *TimeoutError, or *PanicError and handle them accordingly.

Extracting details from ApplicationError

When handling an *ApplicationError in a Workflow, you can retrieve the error message with applicationErr.Error() and extract strong-typed details using applicationErr.Details(&detailMsg). You can also check the error type using applicationErr.Type() to differentiate between custom error types.

PanicError handling in workflows

When handling a *PanicError in a Workflow, the error message and call stack are available through panicErr.Error() and panicErr.StackTrace().

Example of throwing a benign exception in an Activity

import io.temporal.activity.ActivityInterface; import io.temporal.activity.ActivityMethod; import io.temporal.failure.ApplicationErrorCategory; import io.temporal.failure.ApplicationFailure; @ActivityInterface public interface MyActivities { @ActivityMethod String myActivity(); } public class MyActivitiesImpl implements MyActivities { @Override public String myActivity() { try { return callExternalService(); } catch (Exception e) { // Mark this error as benign since it's expected throw ApplicationFailure.newBuilder() .setMessage(e.getMessage()) .setType(e.getClass().getName()) .setCause(e) .setCategory(ApplicationErrorCategory.BENIGN) .build(); } } } This example shows how to wrap an exception in an ApplicationFailure with ApplicationErrorCategory.BENIGN to mark expected Activity errors.

Mark Activity errors as benign with ApplicationErrorCategory

To mark an Activity error as benign in Java, use the ApplicationFailure builder and set the category to ApplicationErrorCategory.BENIGN. This excludes the error from observability data while allowing it to be handled in Workflow logic.

Activity Timeout Exception includes heartbeat details

If the Activity Execution times out, the last Heartbeat details are included in the thrown ActivityTimeoutException, which can be caught by the calling Workflow. The Workflow can then use the details information to pass to the next Activity invocation if needed.

Setting Retry Options with ActivityStub example

Example of setting Retry Options with ActivityStub: ActivityOptions options = ActivityOptions.newBuilder().setStartToCloseTimeout(Duration.ofSeconds(5)).setRetryOptions(RetryOptions.newBuilder().setInitialInterval(Duration.ofSeconds(1)).setMaximumInterval(Duration.ofSeconds(10)).build()).build();

Setting Retry Options with WorkflowImplementationOptions example

Example of setting Retry Options per-Activity with WorkflowImplementationOptions: WorkflowImplementationOptions options = WorkflowImplementationOptions.newBuilder().setActivityOptions(ImmutableMap.of("EmailCustomerGreeting", ActivityOptions.newBuilder().setStartToCloseTimeout(Duration.ofSeconds(5)).setRetryOptions(RetryOptions.newBuilder().setDoNotRetry(NullPointerException.class.getName()).build()).build())).build();

Activity Retry Policy configuration

Activity Executions are automatically associated with a default Retry Policy if a custom one is not provided. To set a custom Retry Policy, use ActivityOptions.newBuilder().setRetryOptions() with a RetryOptions parameter.

Activity Retry Policy configuration in PHP

Set a Retry Policy on an Activity by using ActivityOptions::new()->withRetryOptions(RetryOptions::new()). RetryOptions supports: withInitialInterval(CarbonInterval::seconds(n)) to set the initial retry interval, withMaximumAttempts(n) to set the maximum number of attempts, and withNonRetryableExceptions([ExceptionClass::class]) to specify exceptions that should not trigger retries.

Override next Retry delay with ApplicationFailure

Throw an ApplicationFailure with the nextRetryDelay field set to override the retry interval from the retry policy. Pass a DateInterval object to customize the delay before the next attempt, for example based on the number of attempts.

Activity next Retry delay example in PHP

Example of overriding the next retry delay in PHP: ```php $attempt = \Temporal\Activity::getInfo()->attempt; throw new \Temporal\Exception\Failure\ApplicationFailure( message: "Something bad happened on attempt $attempt", type: 'my_failure_type', nonRetryable: false, nextRetryDelay: \DateInterval::createFromDateString(\sprintf('%d seconds', $attempt * 3)), ); ``` This throws an ApplicationFailure that sets the next retry delay to 3 seconds multiplied by the current attempt number.

ApplicationError with category parameter for benign exceptions

In Python, raise an ApplicationError with the category parameter set to ApplicationErrorCategory.BENIGN to mark expected or non-severe Activity errors. Example: raise ApplicationError(message=str(err), category=ApplicationErrorCategory.BENIGN)

Mark Activity errors as benign using ApplicationErrorCategory.BENIGN

To mark an Activity error as benign in the Python SDK, set the category parameter to ApplicationErrorCategory.BENIGN when raising an ApplicationError. Benign errors have their Activity failure logs downgraded to DEBUG level, do not emit Activity failure metrics, and do not set the OpenTelemetry failure status to ERROR.

Use benign exceptions for expected Activity failures

Mark Activity errors as benign when they occur regularly as part of normal operations, such as polling an external service that isn't ready yet or handling expected transient failures that will be retried. This reduces noise in logs, metrics, and OpenTelemetry traces.

RetryPolicy parameters in Python

RetryPolicy accepts the following parameters: backoff_coefficient (float), maximum_attempts (int), initial_interval (timedelta), maximum_interval (timedelta), and non_retryable_error_types (list of error type strings). The non_retryable_error_types parameter allows specifying error types that should not trigger a retry.

Override retry interval with next_retry_delay

To override the next retry interval set by the current Retry Policy, pass next_retry_delay when raising an ApplicationError in an Activity. This value replaces and overrides whatever the retry interval would normally be. The next_retry_delay accepts a timedelta value and can be set based on the Activity's attempt count.

Activity timeout example with retry policy

Example showing how to set activity timeouts and retry policy in Python: ```python from datetime import timedelta from temporalio import workflow from temporalio.common import RetryPolicy with workflow.unsafe.imports_passed_through(): from activities import your_activity, YourParams @workflow.defn class YourWorkflow: @workflow.run async def run(self, greeting: str) -> list[str]: activity_result = await workflow.execute_activity( your_activity, YourParams(greeting, "Retry Policy options"), start_to_close_timeout=timedelta(seconds=10), retry_policy=RetryPolicy( backoff_coefficient=2.0, maximum_attempts=5, initial_interval=timedelta(seconds=1), maximum_interval=timedelta(seconds=2), ), ) return activity_result ```

Activity Retry Policy defaults and configuration

Activity Executions are automatically associated with a default Retry Policy if a custom one is not provided. A Retry Policy works in cooperation with timeouts to provide fine controls for execution. The RetryPolicy class is set within execute_activity() or start_activity() functions.

Custom retry delay example

Example showing how to override retry interval with next_retry_delay based on attempt count: ```python from temporalio.exceptions import ApplicationError from datetime import timedelta from temporalio import activity @activity.defn async def my_activity(input: MyActivityInput): try: # Your activity logic goes here except Exception as e: attempt = activity.info().attempt raise ApplicationError( f"Error encountered on attempt {attempt}", next_retry_delay=timedelta(seconds=3 * attempt), ) from e ```

Non-retryable flag in ApplicationError

Set the non_retryable flag when raising an ApplicationError to mark specific errors as non-retryable. An ApplicationError with non_retryable=True will never retry, regardless of the Retry Policy. Use non-retryable errors for invalid input data that prevents the Activity from proceeding, business rule violations, and authorization failures.

When to use non_retryable vs non_retryable_error_types

Use non_retryable=True in the Activity when the Activity implementer knows the error is permanently unrecoverable. This enforces the constraint for all callers. Use non_retryable_error_types in the Retry Policy when the caller wants to decide which errors are unrecoverable based on their business logic. This lets different Workflows make different decisions about the same Activity.

ApplicationError for activity exceptions

Use ApplicationError to communicate application-specific failures from Activities. Any other exceptions raised from Python code in a Temporal Activity are converted to an ApplicationError internally, so error type, severity, and details can be sent to the Temporal Service and indexed by the Web UI.

Non-retryable ApplicationError example

from temporalio import activity from temporalio.exceptions import ApplicationError @activity.defn async def process_payment(card_number: str, amount: float): if not is_valid_card_format(card_number): # Invalid format will never become valid through retries raise ApplicationError( f"Invalid credit card format: {card_number}", type="InvalidCardFormat", non_retryable=True, ) if amount <= 0: # Invalid amount won't be fixed by retrying raise ApplicationError( f"Amount must be positive: {amount}", type="InvalidAmount", non_retryable=True, ) # Process payment...

ApplicationError example in Python activity

from temporalio import activity from temporalio.exceptions import ApplicationError @activity.defn async def validate_charge(credit_card_number: str, amount: float): if not is_valid_card(credit_card_number): raise ApplicationError( f"Invalid credit card number: {credit_card_number}", type="InvalidCreditCard", ) if amount <= 0: raise ApplicationError( f"Amount must be positive, got {amount}", type="InvalidAmount", ) return True

ActivityError wraps activity failures

When an Activity fails, Temporal wraps the exception in an ActivityError before surfacing it to the Workflow. The ActivityError provides context including: the Activity type that failed, number of retry attempts, and original cause (the ApplicationError you raised, or TimeoutError, CancelledError, etc.).

Activity retry on failure

Temporal automatically retries Activities that fail based on your configuration. Activities often interact with the outside world where failures can occur.

Cancel an Activity from a Workflow

To cancel an Activity from a Workflow Execution, call the cancel() method on the Activity handle returned from start_activity(). The Activity must send Heartbeats and have a Heartbeat Timeout set for cancellation to work. When canceled, an asyncio.CancelledError is raised in the Activity at the next available opportunity.

Example: Cancel an Activity from a Workflow in Python

@activity.defn async def cancellable_activity(input: ComposeArgsInput) -> NoReturn: try: while True: print("Heartbeating cancel activity") await asyncio.sleep(0.5) activity.heartbeat("some details") except asyncio.CancelledError: print("Activity cancelled") raise @workflow.defn class GreetingWorkflow: @workflow.run async def run(self, input: ComposeArgsInput) -> None: activity_handle = workflow.start_activity( cancellable_activity, ComposeArgsInput(input.arg1, input.arg2), start_to_close_timeout=timedelta(minutes=5), heartbeat_timeout=timedelta(seconds=30), ) await asyncio.sleep(3) activity_handle.cancel() This example shows a cancellable Activity that sends Heartbeats with a 30-second timeout, and a Workflow that starts the Activity and cancels it after 3 seconds.

Give your agent this brain