workflow-operations

11 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

workflow cancel event history and cleanup

Canceling a running Workflow Execution records a WorkflowExecutionCancelRequested event in the Event History. The Service schedules a new Command Task, and the Workflow Execution performs any cleanup work supported by its implementation.

workflow delete async with optional termination

The `temporal workflow delete` command deletes a Workflow Execution and its Event History asynchronously. If the Execution is running, the Service terminates it before deletion. In global Namespaces, deletion removes executions from all replicas.

Schedule missed catchup window metric (Cloud)

Temporal Cloud emits the counter `temporal_cloud_v1_schedule_missed_catchup_window_count` when a scheduled Action is skipped because it could not run within the configured Catchup Window. Alert on any non-zero value grouped by `temporal_namespace`. Example PromQL: `sum by (temporal_namespace) (increase(temporal_cloud_v1_schedule_missed_catchup_window_count[5m])) > 0`

Schedule missed catchup window metric (self-hosted)

Self-hosted Temporal emits the counter `schedule_missed_catchup_window` when a scheduled Action is skipped because it could not run within the configured Catchup Window. Alert on any non-zero value grouped by `namespace`. Example PromQL: `sum by (namespace) (increase(schedule_missed_catchup_window[5m])) > 0`

Missed catchup window metric scope

The missed catchup window metric is scoped to the Namespace, not to individual Schedules. A non-zero value indicates that at least one Schedule in the Namespace missed an Action, but not which Schedule.

DescribeSchedule info block fields

The `DescribeSchedule` API returns an `info` block with cumulative counters and state: `missedCatchupWindow` (Actions skipped because they could not run within the Catchup Window), `overlapSkipped` (Actions skipped because the previous run was still in progress and Overlap Policy is Skip), `bufferDropped` (Buffered Actions dropped because the buffer was full), `bufferSize` (current depth of the Action buffer), `recentActions` (most recent Action times and results), and `runningWorkflows` (Workflow Executions currently running for the Schedule).

Default Catchup Window duration

The default Catchup Window for Schedules is one year. The minimum Catchup Window is ten seconds.

Common root causes of missed Schedule Actions

Missed Schedule Actions are typically caused by: (1) Service or Namespace outage longer than the Catchup Window, (2) Namespace rate limiting that causes scheduled starts to queue past the Catchup Window (cross-check `temporal_cloud_v1_schedule_rate_limited_count` on Cloud or `schedule_rate_limited` on self-hosted), or (3) Buffer overruns under `BufferAll` where long-running Workflow Executions push buffered Actions past the Catchup Window (cross-check `temporal_cloud_v1_schedule_buffer_overruns_count` on Cloud or `schedule_buffer_overruns` on self-hosted).

Investigate missed Schedule Actions workflow

To investigate which Schedule missed an Action: (1) Run `temporal schedule list --namespace <your-namespace>` to enumerate all Schedules in the alerting Namespace, (2) For each Schedule Id, run `temporal schedule describe --schedule-id <your-schedule-id> --namespace <your-namespace>` and check the `missedCatchupWindow` field in the `info` block. A non-zero value identifies the Schedule responsible for the alert.

Assess impact of missed Schedule Actions

To assess impact: compare `recentActions` to the Schedule's Spec to determine how many Actions were skipped and over what time period. If the Schedule uses the `Skip` Overlap Policy and the preceding run was long-running, the miss may reflect that run exceeding the Catchup Window rather than a Service outage. For business-critical Schedules, Backfill the skipped interval once the underlying cause is resolved.

Remediate missed Schedule Actions

To remediate missed Schedule Actions: (1) Widen the Catchup Window if the current value is tighter than your Service's worst-case unavailability (trade-off is that more late Actions will fire during recovery), (2) Revisit the Overlap Policy if runs routinely exceed the Spec interval (BufferAll and Skip have different failure modes under sustained delay), (3) Increase Namespace throughput limits if rate limiting is the contributing factor, (4) Backfill the missed interval if the skipped Actions need to run.

Give your agent this brain