activities/execution

180 notes in this subject, read out of this brain and free to use. This is page 3 of 3.

ScheduleToCloseTimeout with WorkflowImplementationOptions example

Example of setting ScheduleToCloseTimeout with WorkflowImplementationOptions: ```java WorkflowImplementationOptions options = WorkflowImplementationOptions.newBuilder() .setActivityOptions( ImmutableMap.of( "GetCustomerGreeting", ActivityOptions.newBuilder() .setScheduleToCloseTimeout(Duration.ofSeconds(5)) .build())) .build(); ```

StartToCloseTimeout with ActivityStub example

Example of setting StartToCloseTimeout with ActivityStub: ```java GreetingActivities activities = Workflow.newActivityStub(GreetingActivities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(2)) .build()); ```

HeartbeatTimeout with ActivityStub example

Example of setting HeartbeatTimeout with ActivityStub: ```java private final GreetingActivities activities = Workflow.newActivityStub( GreetingActivities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(5)) .setHeartbeatTimeout(Duration.ofSeconds(2)) .build()); ```

TaskQueue with ActivityStub example

Example of setting TaskQueue with ActivityStub: ```java GreetingActivities activities = Workflow.newActivityStub(GreetingActivities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(5)) .setTaskQueue("yourTaskQueue") .build()); ```

RetryOptions with ActivityStub example

Example of setting RetryOptions with ActivityStub: ```java private final ActivityOptions options = ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(5)) .setRetryOptions( RetryOptions.newBuilder() .setInitialInterval(Duration.ofSeconds(1)) .setMaximumInterval(Duration.ofSeconds(10)) .build()) .build(); ```

setCancellationType with ActivityStub example

Example of setting setCancellationType with ActivityStub: ```java private final GreetingActivities activities = Workflow.newActivityStub( GreetingActivities.class, ActivityOptions.newBuilder() .setCancellationType(ActivityCancellationType.WAIT_CANCELLATION_COMPLETED) .build()); ```

Asynchronous Activity completion with doNotCompleteOnReturn example

Example of using ActivityExecutionContext.doNotCompleteOnReturn(): ```java public class FileProcessingActivitiesImpl implements FileProcessingActivities { public String download(String bucketName, String remoteName, String localName) { ActivityExecutionContext ctx = Activity.getExecutionContext(); // Used to correlate reply byte[] taskToken = ctx.getInfo().getTaskToken(); asyncDownloadFileFromS3(taskToken, bucketName, remoteName, localDirectory + localName); ctx.doNotCompleteOnReturn(); // Return value is ignored when doNotCompleteOnReturn was called. return "ignored"; } } ```

ActivityCompletionClient completion methods example

Example of using ActivityCompletionClient to complete or fail an Activity: ```java public <R> void completeActivity(byte[] taskToken, R result) { completionClient.complete(taskToken, result); } public void failActivity(byte[] taskToken, Exception failure) { completionClient.completeExceptionally(taskToken, failure); } ```

Activity reset restore-original-options and keep-paused flags

The temporal activity reset command provides --keep-paused flag to prevent unpausing a paused Activity (by default a reset unpauses). The --restore-original-options flag restores the original options of the activity. Similarly, temporal activity update-options and temporal activity unpause support --restore-original-options.

Activity cancellation transitions to CancelRequested state

When an Activity cancellation is requested via the temporal activity cancel command, the Activity's run state transitions to CancelRequested. If the Activity is heartbeating, a cancellation error will be raised when the next heartbeat response is received; if the Activity allows this error to propagate, the Activity transitions to canceled status.

Activity pause behavior for running vs non-running activities

The pause command for Activities works differently depending on the Activity's state. If the Activity is not currently running (e.g., because it previously failed), it will not be run again until unpaused. However, if the Activity is currently running, it will run until the next time it fails, completes, or times out, at which point the pause takes effect. Pause does not stop or extend the Activity's Schedule-To-Close Timeout; a paused Activity can still time out.

Activity reset operation and heartbeat handling

The reset command restarts an Activity as if it were first being scheduled, resetting both the number of attempts and the activity timeout. Activities that heartbeat will receive a Canceled failure the next time they heartbeat after a reset. If the Activity may be executing, the reset will take effect the next time it fails, heartbeats, or times out. If waiting for a retry, the reset applies immediately. The reset-heartbeats flag can clear heartbeat details.

Activity termination is not visible to Activity code

When an Activity is terminated via the temporal activity terminate command, the Activity code cannot see or respond to terminations.

ExecuteActivityAsync determines Activity execution count

An Activity defined via ExecuteActivityAsync (as indicated by the execute command blocking until completion) executes once when started. The execute command starts a new Standalone Activity and blocks until it completes, returning the result to stdout.

Activity ID reuse policies

When starting an Activity with an ID that exists and has completed, the --id-reuse-policy flag controls behavior. Accepted values are: AllowDuplicate (allow new execution with same ID), AllowDuplicateFailedOnly (allow only if previous execution failed), RejectDuplicate (reject if ID already exists).

Activity ID conflict policies for concurrent executions

The --id-conflict-policy flag controls what happens when an Activity with the same ID is currently running. Accepted values are: Fail (reject the start request) and UseExisting (use the already-running Activity).

Activity timeout configuration parameters

Activities have four timeout parameters: schedule-to-close-timeout (maximum time for the Activity Execution including all retries), start-to-close-timeout (maximum time for a single Activity attempt), schedule-to-start-timeout (maximum time an Activity task can stay in a task queue before a Worker picks it up), and heartbeat-timeout (maximum time between successful Worker heartbeats). Either schedule-to-close-timeout or start-to-close-timeout is required when starting an Activity.

Activity retry policy parameters

Activity retry policies are configured via: --retry-maximum-attempts (max attempts; 1 disables retries, 0 means unlimited), --retry-initial-interval (interval of first retry), --retry-backoff-coefficient (coefficient for calculating next retry interval, must be 1 or larger), --retry-maximum-interval (maximum interval between retries).

Activity priority key configuration

The --priority-key flag specifies Activity priority as an integer from 1-5, where lower values indicate higher priority. The default priority is 3 when not specified.

Activity input specification methods

Activity input can be specified via --input (JSON content passed directly), --input-file (path to input file), or --input-base64 (base64-encoded input). These options cannot be combined; --input and --input-file are mutually exclusive. Input can be passed multiple times to provide multiple arguments. Use --input-meta to override payload metadata such as encoding.

Activity headers and search attributes

Temporal activity headers are specified via --headers in 'KEY=VALUE' format where keys must be identifiers and values must be JSON values. Search Attributes are specified via --search-attribute in 'KEY=VALUE' format with the same constraints. Both can be passed multiple times to set multiple values.

Activity static details and summary (experimental)

The --static-details and --static-summary flags allow specification of static Activity information for human consumption in UIs. These use standard Markdown formatting excluding images, HTML, and script tags. Both flags are marked as experimental.

Activity complete command for successful completion

The temporal activity complete command marks an Activity as successfully finished. It requires the Activity ID and a JSON result value via --result. The --workflow-id flag is required for workflow Activities but must be omitted for Standalone Activities. The --run-id distinguishes between Workflow Run ID (for workflow Activities) and Activity Run ID (for Standalone Activities).

Activity fail command for error marking

The temporal activity fail command marks an Activity as having encountered an error. It requires the Activity ID. Optional parameters are --reason (failure message), --detail (failure details as JSON), --workflow-id (required for workflow Activities, omitted for Standalone Activities), and --run-id (Workflow Run ID for workflow Activities, Activity Run ID for Standalone Activities).

Activity reset supports bulk operations via query

The temporal activity reset command supports bulk activity reset using a visibility query list filter via the --query flag. When using --query, the command is marked as an experimental feature and may change in the future. Additional parameters for bulk operations include --reason (defaults to user name), --rps (requests per second limit), --jitter (random delay within specified duration), and --yes (skip confirmation prompt).

Activity unpause batch operations with reset options

The temporal activity unpause command can reset attempts via --reset-attempts and reset heartbeats via --reset-heartbeats. For batch operations using --query, additional parameters include --reason (defaults to user name), --rps (requests per second limit), --jitter (random delay), and --yes (skip confirmation).

Activity update-options incremental update behavior

The temporal activity update-options command performs incremental updates, only changing the specified options without affecting unspecified parameters. The command can update timeout values (schedule-to-close-timeout, schedule-to-start-timeout, start-to-close-timeout, heartbeat-timeout) and retry policy values (retry-initial-interval, retry-maximum-interval, retry-backoff-coefficient, retry-maximum-attempts). A subsequent temporal activity reset will apply the new values after reset.

Standalone Activities retry behavior

If a Standalone Activity fails, the Server automatically retries it according to the Retry Policy you configure when calling the Activity from your application.

Activities can be asynchronous or synchronous

Activities in the .NET SDK can be implemented as either asynchronous or synchronous methods. Both static and instance methods are supported.

DisableEagerActivityExecution always true on Lambda

DisableEagerActivityExecution is always true on Lambda and cannot be overridden. Eager Activities require a persistent connection, which Lambda invocations don't maintain.

Activity Task execution process

When a Workflow invokes an Activity, the Worker does not immediately execute that Activity code. Instead, it generates a ScheduleActivityTask Command, dispatching it to the Temporal Service. In response, the Temporal Service queues up a new Activity Task. Only when a Worker finds itself free does it collect the task and begin executing the Activity code.

Schedule-To-Start Timeout is non-retryable

The Schedule-To-Start Timeout is non-retryable by design. It does not trigger any retries regardless of the Retry Policy, because a retry would place the Activity Task back into the same Task Queue, not resolving the underlying issue.

Schedule-To-Start Timeout measurement point

The moment that an Activity Task is picked up by the Worker from the Task Queue is considered to be the start of the Activity Task Execution for the purposes of the Schedule-To-Start Timeout and associated metrics. This definition of 'Start' avoids issues that a clock difference between the Temporal Service and a Worker might create.

Schedule vs Schedule-To-Start frequency difference

The Schedule-To-Start Timeout is enforced for each Activity Task, whereas the Schedule-To-Close Timeout is enforced once per Activity Execution. Thus, 'Schedule' in Schedule-To-Start refers to the scheduling moment of every Activity Task in the sequence of Activity Tasks that make up the Activity Execution, while 'Schedule' in Schedule-To-Close refers to the first Activity Task in that sequence.

Activity Execution composed of multiple Activity Task Executions

An Activity Execution can be composed of multiple Activity Task Executions, with each Task representing a single attempt or retry. When an Activity is defined in a Workflow, that definition represents a single logical Activity Execution. If that Activity fails (for example, due to a timeout or error) and a Retry Policy is in place, Temporal will schedule another attempt to execute the same Activity. Each attempt is called an Activity Task Execution.

Start-To-Close Timeout definition

A Start-To-Close Timeout is the maximum time allowed for a single Activity Task Execution. This timeout applies to each Activity Task Execution within an Activity Execution, not the overall Activity Execution.

Activity Execution must have Start-To-Close or Schedule-To-Close Timeout

An Activity Execution must have either Start-To-Close Timeout or Schedule-To-Close Timeout set. It is strongly recommended to always set the Start-To-Close Timeout, ensuring it is set to be longer than the maximum possible time for the Activity Execution to complete. For long-running Activity Executions, also using Activity Heartbeats and Heartbeat Timeouts is recommended.

Start-To-Close Timeout use case and reliability

The main use case for the Start-To-Close Timeout is to detect when a Worker crashes after it has started executing an Activity Task. The Temporal Server doesn't detect failures when a Worker loses communication with the Server or crashes. Therefore, the Temporal Server relies on the Start-To-Close Timeout to force Activity retries.

Start-To-Close Timeout actions when reached

If the Start-To-Close Timeout is reached, the following actions occur: (1) An ActivityTaskTimedOut Event is written to the Workflow Execution's mutable state. (2) If a Retry Policy dictates a retry, the Temporal Service schedules another Activity Task, the attempt count increments by 1 in the Workflow Execution's mutable state, and the Start-To-Close Timeout timer is reset.

Schedule-To-Close Timeout definition

A Schedule-To-Close Timeout is the maximum amount of time allowed for the overall Activity Execution, from when the first Activity Task is scheduled to when the last Activity Task, in the chain of Activity Tasks that make up the Activity Execution, reaches a Closed status.

Schedule-To-Close Timeout purpose with retries

The Schedule-To-Close Timeout can be used to control the overall duration of an Activity Execution in the face of failures (repeated Activity Task Executions), without altering the Maximum Attempts field of the Retry Policy.

Activity Heartbeat definition and purpose

An Activity Heartbeat is a ping from the Worker that is executing the Activity to the Temporal Service. Each ping informs the Temporal Service that the Activity Execution is making progress and the Worker has not crashed. Activity Heartbeats work in conjunction with a Heartbeat Timeout.

Activity Heartbeat implementation and frequency

Activity Heartbeats are implemented within the Activity Definition. Custom progress information can be included in the Heartbeat which can then be used by the Activity Execution should a retry occur. A Heartbeat can be recorded as often as needed (for example, once a minute or every loop iteration). It is often a good practice to Heartbeat on anything but the shortest Activity Execution. Temporal SDKs control the rate at which Heartbeats are sent to the Temporal Service.

Local Activities do not require heartbeating

Heartbeating is not required from Local Activities, and does nothing if attempted.

Activity Heartbeat progress persistence during retries

A Heartbeat can include an application layer payload that can be used to save Activity Execution progress. If an Activity Task Execution times out due to a missed Heartbeat, the next Activity Task can access and continue with that payload.

Activity Cancellation delivery via Heartbeats

Activity Cancellations are delivered to Activities from the Temporal Service when they Heartbeat. Activities that don't Heartbeat cannot receive a Cancellation. Heartbeat throttling may lead to Cancellation being delivered later than expected.

Heartbeat throttling mechanism and intervals

Heartbeats may not always be sent to the Temporal Service—they may be throttled by the Worker. The throttle interval is the smaller of the following: (1) If heartbeatTimeout is provided, heartbeatTimeout * 0.8; otherwise, defaultHeartbeatThrottleInterval. (2) maxHeartbeatThrottleInterval. The defaultHeartbeatThrottleInterval is 30 seconds by default, and maxHeartbeatThrottleInterval is 60 seconds by default. Each can be set in Worker options.

Heartbeat throttling implementation

Throttling is implemented as follows: After sending a Heartbeat, the Worker sets a timer for the throttle interval. The Worker stops sending Heartbeats, but continues receiving Heartbeats from the Activity and remembers the most recent one. When the timer fires, the Worker sends the most recent Heartbeat and sets the timer again.

Heartbeat throttling exceptions

Throttling does not apply to the final Heartbeat message in the case of Activity Failure. If an Activity fails just after recording progress information in a Heartbeat message, that progress information will be available during the next retry attempt, provided that the Worker itself did not crash before delivering it to the Temporal Service.

Heartbeat decision criteria

Heartbeating is best thought about not in terms of time, but in terms of 'How do you know you are making progress?' For short-term operations, progress updates are not a requirement. However, checking the progress and status of Activity Executions that run over long periods is almost always useful.

Suitable scenarios for Activity Heartbeating

Suitable scenarios for Activity Heartbeating include: (1) Reading a large file from Amazon S3, and (2) Running an ML training job on some local GPUs.

Heartbeat Timeout recommendations for long-running activities

For long-running Activities, it is recommended to use a relatively short Heartbeat Timeout and a frequent Heartbeat. That way if a Worker fails it can be handled in a timely manner.

Monitoring alternative to Schedule-To-Start Timeout

In most cases, monitoring the temporal_activity_schedule_to_start_latency metric is recommended to know when Workers slow down picking up Activity Tasks, instead of setting the Schedule-To-Start Timeout.

Schedule-To-Start Timeout recommendation for production

If the Schedule-To-Start Timeout is used, it is recommended to set this timeout to the maximum time a Workflow Execution is willing to wait for an Activity Execution in the presence of all possible Worker outages, and have a concrete plan in place to reroute Activity Tasks to a different Task Queue.

Heartbeat considerations for Activity Execution progress

When considering Activity Heartbeats, note that the underlying task must be able to report definite progress. The Workflow cannot read this progress information while the Activity is still executing (or it would have to store it in Event History). Progress can be reported to external sources if it needs to be exposed to the user.

Schedule-To-Close Timeout default value

The default Schedule-To-Close Timeout is ∞ (infinity).

Unsuitable scenarios for Activity Heartbeating

Unsuitable scenarios for Activity Heartbeating include: (1) Making a quick API call, and (2) Reading a small file from disk.

Heartbeat Timeout definition

A Heartbeat Timeout is the maximum time between Activity Heartbeats. If this timeout is reached, the Activity Task fails and a retry occurs if a Retry Policy dictates it.

Schedule-To-Start Timeout definition and purpose

A Schedule-To-Start Timeout is the maximum amount of time allowed from when an Activity Task is scheduled (placed in a Task Queue) to when a Worker starts executing that Activity Task (picks it up from the Task Queue). It is a limit for how long an Activity Task can be enqueued. The default Schedule-To-Start Timeout is ∞ (infinity). This timeout has two primary use cases: (1) detect whether an individual Worker has crashed, and (2) detect whether the fleet of Workers polling the Task Queue cannot keep up with the rate of Activity Tasks.

Activities handle external world interactions

Activities handle everything that interacts with the outside world, including API calls, database queries, LLM invocations, and file I/O. When a Workflow calls an Activity, the Activity runs once and its result is recorded in the Event History. During replay, that result is reused rather than recomputed, so Activities are not executed again during replay.

Give your agent this brain