Set Activity priority in Python
In Python, override Activity priority in `workflow.execute_activity()` using the `priority` parameter with `Priority(priority_key=N)` to run an individual Activity at a different priority than its parent Workflow.
activities & execution
23 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.
Set Activity priority in Go
In Go, override Activity priority by setting `Priority: temporal.Priority{PriorityKey: N}` in `workflow.ActivityOptions` and applying it with `workflow.WithActivityOptions(ctx, ao)`.
Set Activity priority in Java
In Java, override Activity priority using `ActivityOptions.newBuilder().setPriority(Priority.newBuilder().setPriorityKey(N).build())` when creating the Activity stub.
DisableEagerActivityExecution always true on Lambda
DisableEagerActivityExecution is always true on Lambda and cannot be overridden. Eager Activities require a persistent connection, which Lambda invocations do not maintain.
ActivityTaskScheduled event indicates activity scheduled
ActivityTaskScheduled event indicates that an Activity Task was scheduled. The SDK client should pick up this Activity Task and execute. It has fields: activity_id (Identifier assigned to Activity by Worker or user), activity_type (Type of Activity scheduled), namespace (Namespace of Workflow that Activity resides in), task_queue (Task Queue that Activity Task was enqueued in), header (Information passed by sender of Signal copied into Workflow Task), input (Deserialized to provide arguments to Workflow function), schedule_to_close_timeout (Amount of time caller will wait for Activity completion, limits time retries attempted), schedule_to_start_timeout (Limits time Activity Task can stay in Task Queue, cannot be retried), start_to_close_timeout (Maximum execution time allowed after being picked up by Worker, retryable), heartbeat_timeout (Maximum time allowed between successful Worker heartbeats), workflow_task_completed_event_id (Id of WorkflowTaskCompleted Event reported with), retry_policy (Amount of retries as determined by service's dynamic configuration).
ActivityTaskStarted event indicates task started
ActivityTaskStarted event indicates that an Activity Task Execution was started. The SDK Worker picked up the Activity Task and started processing the Activity invocation. ActivityTaskStarted is generated by the server when the Task is dispatched to the Worker, not when the Worker starts executing the Task. This Event is not written to History until the terminal Event (like ActivityTaskCompleted or ActivityTaskFailed) occurs. It has fields: scheduled_event_id (Id of ActivityTaskScheduled Event this Task corresponds to), identity (Identifies Worker that started Task), request_id (Identifies Activity Task request), attempt (Number of attempts made to complete Task), last_failure (Details from most recent failure Event, only assigned if Task previously failed and been retried).
ActivityTaskCompleted event indicates successful completion
ActivityTaskCompleted event indicates that the Activity Task has completed. The SDK client has picked up and successfully completed the Activity Task. It has fields: result (Serialized result of completed Activity), scheduled_event_id (Id of ActivityTaskScheduled Event this completion Event corresponds to), started_event_id (Id of ActivityTaskStarted Event this Task corresponds to), identity (Identity of Worker that completed Task).
ActivityTaskFailed event indicates task failure
ActivityTaskFailed event indicates that the Activity Task has failed. The SDK client picked up the Activity Task but unsuccessfully completed it. It has fields: failure (Serialized result of Workflow failure), scheduled_event_id (Id of ActivityTaskScheduled Event this failure Event corresponds to), started_event_id (Id of ActivityTaskStarted Event this failure corresponds to), retry_state (Reason provided for whether Task should or shouldn't be retried).
ActivityTaskCancelRequested and ActivityTaskCanceled events
ActivityTaskCancelRequested event indicates that a request to cancel the Activity has occurred. It has fields: scheduled_event_id (Id of ActivityTaskScheduled Event this cancel Event corresponds to), workflow_task_completed_event_id (Id of WorkflowTaskCompleted Event reported with). ActivityTaskCanceled event indicates that the Activity has been canceled. It has fields: details (Additional information reported by Activity upon confirming cancelation), latest_cancel_requested_event_id (Id of most recent ActivityTaskCancelRequested Event referring to same Activity), scheduled_event_id (Id of ActivityTaskScheduled Event this cancelation corresponds to), started_event_id (Id of ActivityTaskStarted Event this cancelation corresponds to), identity (Identifies Worker that requested cancelation).
Local Activity timeout causes re-execution from scratch
If a Workflow Task timeout causes a Local Activity to re-execute, the Local Activity runs from scratch on the retried Task because its results are not checkpointed between Workflow Task heartbeats.
Unregistered Activity gets retried indefinitely
When a Workflow schedules an Activity that the Worker has no registered implementation for, the Activity keeps getting retried against a Worker that has no implementation for it until the Activity's scheduleToClose timeout expires, or forever if no timeout is set.
Local Activity Workflow Task heartbeat timeout default
The Workflow Task heartbeat timeout defaults to 30 minutes. If a Local Activity runs past this timeout, the Temporal Service times out the heartbeating Workflow Task and reschedules it on the normal Task Queue.
How Workflow Task heartbeating works
Local Activities execute inside the Workflow Task rather than as separately scheduled Activity Tasks. To keep the Task alive longer than the Workflow Task timeout, the SDK sends Workflow Task heartbeats: repeated RespondWorkflowTaskCompleted calls that tell the Temporal Service work is still going and ask for more time. The Service honors this up to the Workflow Task heartbeat timeout.
Local Activity results not recorded between heartbeats
Local Activity results are not recorded in Event History between Workflow Task heartbeats. When a heartbeating Workflow Task is rescheduled, every Local Activity in it runs again from the beginning.
Local Activity non-idempotence creates duplicate side effects
When the Temporal Service times out a heartbeating Workflow Task, any non-idempotent Local Activity in it re-executes from scratch, producing duplicate side effects with real business impact.
Local Activity blocked slot causes cascading delays
Local Activities occupy an executor slot for their entire duration. Several Local Activities running past the heartbeat timeout at once can occupy every available slot, blocking new Local Activities from starting.
Local Activity design is for short fast operations
Local Activities are designed for short, fast operations. A single Local Activity running for 30 minutes is a design problem, not a tuning problem.
Convert long-running Local Activity to regular Activity
If Local Activity work genuinely takes longer than the Workflow Task heartbeat timeout, convert it to a regular Activity with heartbeating, which is the correct primitive for long-running work.
Local Activity timeout handling with scheduleToCloseTimeout
If a Local Activity must stay a Local Activity but could run past the Workflow Task heartbeat timeout, set a scheduleToCloseTimeout below the Workflow Task heartbeat timeout so it fails with a timeout error the Workflow can handle, rather than having the entire Workflow Task re-executed.
temporal_activity_execution_latency spike causes
temporal_activity_execution_latency measures time from when Worker starts processing an Activity Task until it reports to service that Task is complete or failed. Causes for high latency: Activity implementation performing time-consuming operations or making slow external API calls (most common cause), external dependencies constraining Activity, under-resourced Worker nodes or high CPU utilization, and network latency between Workers and external services or Temporal service. Diagnosis and remediation: monitor activity_execution_latency metric filtered by Activity type and Task queue, optimize Activity implementation especially for external services or database interactions, check Worker CPU and memory utilization, and examine Worker configuration particularly (Max)ConcurrentActivityExecutionSize and (Max)WorkerActivitiesPerSecond to ensure they are not limiting activity execution.
Depletion of temporal_worker_task_slots_available for ActivityWorker
temporal_worker_task_slots_available{worker_type="ActivityWorker"} metric indicates number of available slots for executing Activity Tasks on a Worker. Slots go to zero for reasons: Blocked Activities and Zombie Activities (most common cause where activities blocked or not returning on time; Zombie Activities occur when Activity times out at StartToClose or HeartbeatTimeout and has stopped Heartbeating but continues to run occupying slots as more retries occur, happens when Activity code blocks on downstream service call or infinite loop or mismatch between Activity's StartToClose timeout and client-side timeouts for external calls), and Resource Utilization (high CPU or memory usage on Workers causes activities to block and not release slots). Prevention: monitor Worker CPU and Memory usage while increasing (Max)ConcurrentActivityExecutionSize to add more execution slots, add client-side timeout to downstream API client, and review Task code to ensure Tasks complete within reasonable time measured by temporal_activity_execution_latency.
Activity response blob size limit
The blob size limit for Activity responses is 2MB. If a Worker tries to return an Activity response larger than this limit, the service will reject it causing a request failure and increasing temporal_request_failure_total counter.
Local Activity execution and slot holding behavior
Local Activities run inside the Workflow Task execution loop, so a blocked slot holds up the whole Workflow Task. The SDK keeps the Task alive by sending repeated Workflow Task heartbeats. If blocking extends past the Workflow Task heartbeat timeout (30 minutes by default), the Temporal Service times the Task out and reschedules it, causing every Local Activity in it to run again from the start.