workflow basics

54 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Dynamic Workflow definition in .NET SDK

A Dynamic Workflow in Temporal is a Workflow that is invoked dynamically at runtime if no other Workflow with the same name is registered. In the .NET SDK, a Workflow can be made dynamic by setting Dynamic as true on the [Workflow] attribute. Only one Dynamic Workflow can be present on a Worker.

Dynamic Workflow registration requirement

A Dynamic Workflow must be registered with the Worker before it can be invoked.

Dynamic Workflow parameter type in .NET SDK

The Workflow Definition for a Dynamic Workflow must accept a single argument of type Temporalio.Converters.IRawValue[]. The Workflow.PayloadConverter property is used to convert an IRawValue object to the desired type using extension methods in the Temporalio.Converters namespace.

Dynamic Workflow example in .NET SDK

Example of a Dynamic Workflow in .NET SDK: ```csharp [Workflow(Dynamic = true)] public class DynamicWorkflow { [WorkflowRun] public async Task<string> RunAsync(IRawValue[] args) { var name = Workflow.PayloadConverter.ToValue<string>(args.Single()); var param = MyActivityParams("Hello", name); return await Workflow.ExecuteActivityAsync( (MyActivities a) => a.MyActivity(param), new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) }); } } ``` This example shows how to define a Dynamic Workflow that accepts raw arguments, converts them using PayloadConverter, and executes an activity.

Dynamic Workflow definition and invocation

A Dynamic Workflow in Temporal is a Workflow that is invoked dynamically at runtime if no other Workflow with the same name is registered. A Workflow can be registered as dynamic by using worker.RegisterDynamicWorkflow(). You must register the Workflow with the Worker before it can be invoked. Only one Dynamic Workflow can be present on a Worker.

Dynamic Workflow function signature requirement

A Dynamic Workflow Definition must accept a single argument of type converter.EncodedValues. This allows the dynamic workflow to receive arguments that are decoded at runtime.

Dynamic Workflow example with argument decoding and activity execution

package main func DynamicWorkflow(ctx workflow.Context, args converter.EncodedValues) (string, error) { var result string info := workflow.GetInfo(ctx) var arg1, arg2 string err := args.Get(&arg1, &arg2) if err != nil { return "", fmt.Errorf("failed to decode arguments: %w", err) } if info.WorkflowType.Name == "dynamic-activity" { ctx = workflow.WithActivityOptions(ctx, workflow.ActivityOptions{StartToCloseTimeout: 10 * time.Second}) err := workflow.ExecuteActivity(ctx, "random-activity-name", arg1, arg2).Get(ctx, &result) if err != nil { return "", err } } else { result = fmt.Sprintf("%s - %s - %s", info.WorkflowType.Name, arg1, arg2) } return result, nil } This example shows a Dynamic Workflow that decodes two string arguments, checks the workflow type name, and either executes an activity with those arguments or formats them as a string result.

WorkflowStream library location

The Workflow Streams library ships as go.temporal.io/sdk/contrib/workflowstreams and is available in the Go SDK.

Eager Workflow Start latency savings

Eager Workflow Start eliminates the Matching Service round-trip, saving approximately 30–50 ms per Workflow start. When combined with Local Activities, this pattern achieves approximately 265 ms total-workflow latency compared to approximately 850 ms baseline.

Eager Workflow Start mechanism

Eager Workflow Start bypasses the Temporal Matching Service by dispatching the first Workflow Task directly to a co-located Worker. The server returns the first Workflow Task inline in the StartWorkflowExecution response, and the co-located Worker processes it immediately without a separate polling round-trip.

Eager Workflow Start co-location requirement

The Worker and the client that starts the Workflow must share the same process and server connection for Eager Workflow Start to function.

Eager Workflow Start SDK support

Eager Workflow Start is supported in Go, Java, and Python SDKs. The TypeScript SDK does not support Eager Workflow Start.

Eager Workflow Start configuration by SDK

Enable Eager Workflow Start using: EnableEagerStart: true (Go), setDisableEagerExecution(false) (Java), or request_eager_start=True (Python) on the StartWorkflowOptions.

Eager Workflow Start fallback behavior

If the server cannot fulfill the eager request (for example, no local slot is available), it falls back silently to normal dispatch. The code does not need to handle this case explicitly.

Self-hosted Temporal server Eager Workflow Start flag

On self-hosted Temporal Server, Eager Workflow Start may require enabling the dynamic config flag: --dynamic-config-value system.enableEagerWorkflowStart=true. Temporal Cloud and recent versions of the open-source server may enable this by default.

Eager Workflow Start Python example

import asyncio from temporalio.client import Client from temporalio.worker import Worker from workflows import TransactionWorkflow from activities import validate_transaction, settle_transaction from shared import TASK_QUEUE, TransactionRequest async def main(): client = await Client.connect("localhost:7233") async with Worker( client, task_queue=TASK_QUEUE, workflows=[TransactionWorkflow], activities=[validate_transaction, settle_transaction], ): result = await client.execute_workflow( TransactionWorkflow.run, TransactionRequest(amount=100.00, currency="USD"), id="eager-workflow-start-demo", task_queue=TASK_QUEUE, request_eager_start=True, ) print(f"Transaction complete: ID={result.id} Status={result.status}") if __name__ == "__main__": asyncio.run(main())

Eager Workflow Start Go example

func main() { c, err := client.Dial(client.Options{}) if err != nil { log.Fatalln("Unable to create Temporal client:", err) } defer c.Close() w := worker.New(c, TaskQueue, worker.Options{}) w.RegisterWorkflow(TransactionWorkflow) w.RegisterActivity(ValidateTransaction) w.RegisterActivity(SettleTransaction) if err := w.Start(); err != nil { log.Fatalln("Unable to start worker:", err) } defer w.Stop() run, err := c.ExecuteWorkflow(context.Background(), client.StartWorkflowOptions{ ID: "eager-workflow-start-demo", TaskQueue: TaskQueue, EnableEagerStart: true, }, TransactionWorkflow, TransactionRequest{Amount: 100.00, Currency: "USD"}) if err != nil { log.Fatalln("Failed to start workflow:", err) } var result Transaction if err := run.Get(context.Background(), &result); err != nil { log.Fatalln("Workflow failed:", err) } fmt.Printf("Transaction complete: ID=%s Status=%s\n", result.ID, result.Status) }

Eager Workflow Start Java example

public class Starter { public static void main(String[] args) { WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs(); WorkflowClient client = WorkflowClient.newInstance(service); WorkerFactory factory = WorkerFactory.newInstance(client); io.temporal.worker.Worker worker = factory.newWorker(Shared.TASK_QUEUE); worker.registerWorkflowImplementationTypes(TransactionWorkflow.Impl.class); worker.registerActivitiesImplementations(new Activities.Impl()); factory.start(); TransactionWorkflow workflow = client.newWorkflowStub( TransactionWorkflow.class, WorkflowOptions.newBuilder() .setTaskQueue(Shared.TASK_QUEUE) .setWorkflowId("eager-workflow-start-demo") .setDisableEagerExecution(false) .build() ); Shared.Transaction result = workflow.processTransaction( new Shared.TransactionRequest(100.00, "USD")); System.out.printf("Transaction complete: ID=%s Status=%s%n", result.id(), result.status()); factory.shutdown(); } }

Eager Workflow Start pitfall: Worker start timing

If the Worker is not registered and running before the eager start call, no local slot exists and the request falls back to normal dispatch.

Eager Workflow Start pitfall: distributed deployments

If the process that calls ExecuteWorkflow is not the same process running the Worker, eager dispatch will never succeed. The call still works, but it provides no latency benefit.

Eager Workflow Start pitfall: self-hosted server configuration

If the server dynamic config flag is not set on a self-hosted server, eager dispatch requests are silently ignored and the execution falls back to normal dispatch. Verify the flag system.enableEagerWorkflowStart=true is set if expected latency improvement is not observed.

Entity Workflow best practice: keep state minimal

Store large data externally and reference it in the Workflow. State is kept in Workflow memory, so external storage should be used for large data.

Entity Workflow best practice: implement deletion handling

Implement an explicit deletion or decommission Signal to handle when an entity lifecycle ends.

Entity Workflow pattern definition and use case

The Entity Workflow pattern models long-lived business entities (users, accounts, devices, orders) as individual Workflows that persist for the entity's entire lifetime, potentially months or years. Each entity gets its own Workflow instance identified by the entity ID, handling all state transitions and operations through Signals and Updates.

Entity Workflow: use entity ID as Workflow ID

Use the entity ID as the Workflow ID. This ensures uniqueness and enables idempotent starts.

Entity Workflow: not suitable for high-frequency updates

Entity Workflow pattern is not a good fit for high-frequency updates of more than 100 per second per entity. A single Workflow handles all operations for one entity, which limits throughput.

Entity Workflow: not suitable for short-lived or CRUD-only processes

Entity Workflow pattern is not a good fit for short-lived processes (use regular Workflows), stateless operations (use Activities), or entities with only CRUD operations (use a database).

Entity Workflow: benefits include complete audit trail

The Workflow history provides a complete audit trail of all state changes. All entity logic lives in one place, and state survives process crashes and restarts. Temporal provides exactly-once execution and automatic retries.

Entity Workflow: Python implementation example

Python Entity Workflow implementation: ```python @dataclass class UserState: status: str = "ACTIVE" profile: ProfileData | None = None pending_email: str | None = None created_at: datetime | None = None updated_at: datetime | None = None @dataclass class UserAccountInput: user_id: str state: UserState | None = None @workflow.defn class UserAccountWorkflow: def __init__(self) -> None: self.state = UserState(created_at=datetime.utcnow()) self.deleted = False self.operation_count = 0 @workflow.run async def run(self, input: UserAccountInput) -> None: if input.state is not None: self.state = input.state await workflow.wait_condition( lambda: self.deleted or workflow.info().is_continue_as_new_suggested() ) if not self.deleted and workflow.info().is_continue_as_new_suggested(): await workflow.wait_condition(workflow.all_handlers_finished) workflow.continue_as_new( UserAccountInput(user_id=input.user_id, state=self.state) ) self.state.status = "DELETED" @workflow.update async def update_profile(self, data: ProfileData) -> None: if self.deleted: raise ValueError("User account is deleted") await workflow.execute_activity( validate_profile, data, start_to_close_timeout=timedelta(seconds=30), ) self.state.profile = data self.state.updated_at = datetime.utcnow() self.operation_count += 1 @workflow.signal def delete(self) -> None: self.deleted = True @workflow.query def get_state(self) -> UserState: return self.state ```

Entity Workflow: Go implementation example

Go Entity Workflow implementation: ```go type UserState struct { Status string Profile ProfileData PendingEmail string CreatedAt time.Time UpdatedAt time.Time } type UserAccountInput struct { UserID string State *UserState } func (w *UserAccountWorkflow) Run(ctx workflow.Context, input UserAccountInput) error { var state UserState if input.State != nil { state = *input.State } else { state = UserState{ Status: "ACTIVE", CreatedAt: workflow.Now(ctx), } } deleted := false operationCount := 0 err := workflow.SetUpdateHandler(ctx, "updateProfile", func(ctx workflow.Context, data ProfileData) error { if deleted { return errors.New("user account is deleted") } if err := workflow.ExecuteActivity(ctx, ValidateProfile, data).Get(ctx, nil); err != nil { return err } state.Profile = data state.UpdatedAt = workflow.Now(ctx) operationCount++ return nil }) if err != nil { return err } for { selector := workflow.NewSelector(ctx) selector.AddReceive(workflow.GetSignalChannel(ctx, "delete"), func(c workflow.ReceiveChannel, more bool) { c.Receive(ctx, nil) deleted = true }) selector.Select(ctx) if deleted { state.Status = "DELETED" return nil } if workflow.GetInfo(ctx).GetContinueAsNewSuggested() { return workflow.NewContinueAsNewError(ctx, w.Run, UserAccountInput{UserID: input.UserID, State: &state}) } } } ```

Entity Workflow: Java implementation example

Java Entity Workflow implementation: ```java public record UserAccountInput(String userId, Optional<UserState> state) {} @WorkflowInterface public interface UserAccountWorkflow { @WorkflowMethod void run(UserAccountInput input); @UpdateMethod void updateProfile(ProfileData data); @SignalMethod void delete(); @QueryMethod UserState getState(); } public class UserAccountWorkflowImpl implements UserAccountWorkflow { private String userId; private UserState state = new UserState(); private boolean deleted = false; private int operationCount = 0; @Override public void run(UserAccountInput input) { this.userId = input.userId(); if (input.state().isPresent()) { this.state = input.state().get(); } else { state.setStatus("ACTIVE"); state.setCreatedAt(Workflow.currentTimeMillis()); } Workflow.await(() -> deleted || Workflow.getInfo().isContinueAsNewSuggested()); if (!deleted && Workflow.getInfo().isContinueAsNewSuggested()) { Workflow.continueAsNew(new UserAccountInput(userId, Optional.of(state))); } state.setStatus("DELETED"); } @Override public void updateProfile(ProfileData data) { if (deleted) throw new IllegalStateException("User account is deleted"); Activities.validateProfile(data); state.setProfile(data); state.setUpdatedAt(Workflow.currentTimeMillis()); operationCount++; } @Override public void delete() { deleted = true; } @Override public UserState getState() { return state; } } ```

Entity Workflow: TypeScript implementation example

TypeScript Entity Workflow implementation: ```typescript interface UserState { status: string; profile?: ProfileData; pendingEmail?: string; createdAt: number; updatedAt: number; } interface UserAccountInput { userId: string; state?: UserState; } export const updateProfileUpdate = defineUpdate<void, [ProfileData]>('updateProfile'); export const deleteSignal = defineSignal('delete'); export const getStateQuery = defineQuery<UserState>('getState'); export async function userAccountWorkflow(input: UserAccountInput): Promise<void> { const state: UserState = input.state ?? { status: 'ACTIVE', createdAt: Date.now(), updatedAt: Date.now(), }; let deleted = false; let operationCount = 0; setHandler(updateProfileUpdate, async (data: ProfileData) => { if (deleted) { throw new Error('User account is deleted'); } await validateProfile(data); state.profile = data; state.updatedAt = Date.now(); operationCount++; }); setHandler(deleteSignal, () => { deleted = true; }); setHandler(getStateQuery, () => state); await condition(() => deleted || workflowInfo().continueAsNewSuggested); if (!deleted && workflowInfo().continueAsNewSuggested) { await condition(allHandlersFinished); await continueAsNew<typeof userAccountWorkflow>({ userId: input.userId, state }); } state.status = 'DELETED'; } ```

Entity Workflow suitable entities and domains

The Entity Workflow pattern is a good fit for user accounts and profiles, IoT devices and sensors, customer relationships (CRM), shopping carts and orders, financial accounts, subscription management, device provisioning and lifecycle, and multi-tenant resources.

Entity Workflow best practice: set timeouts

Use Workflow execution timeout as a safety net.

Python Resumable Activity example with correction and approval signals

from dataclasses import dataclass from datetime import timedelta from temporalio import workflow from temporalio.common import RetryPolicy, SearchAttributeKey from temporalio.exceptions import ActivityError import activities TRANSFER_STATUS_KEY = SearchAttributeKey.for_keyword("TransferStatus") @dataclass class TransferInput: from_account: str to_account: str amount: float @workflow.defn class TransferWorkflow: def __init__(self) -> None: self._status = "PENDING" self._corrected_account: str | None = None self._approval: bool | None = None @workflow.run async def run(self, transfer: TransferInput) -> str: account = transfer.to_account correction_attempts = 0 while True: self._status = "TRANSFERRING" try: result = await workflow.execute_activity( activities.execute_transfer, TransferInput(transfer.from_account, account, transfer.amount), start_to_close_timeout=timedelta(seconds=30), retry_policy=RetryPolicy(maximum_attempts=3), ) break except ActivityError: correction_attempts += 1 if correction_attempts > 5: self._status = "FAILED" workflow.upsert_search_attributes([TRANSFER_STATUS_KEY.value_set(self._status)]) raise self._status = "AWAITING_CORRECTION" workflow.upsert_search_attributes([TRANSFER_STATUS_KEY.value_set(self._status)]) workflow.logger.warning( "Transfer failed — waiting for account correction", extra={"to_account": account}, ) await workflow.wait_condition( lambda: self._corrected_account is not None ) account = self._corrected_account self._corrected_account = None self._status = "AWAITING_APPROVAL" await workflow.wait_condition(lambda: self._approval is not None) if self._approval: self._status = "COMPLETED" return f"Transfer of {transfer.amount} to {account} completed" self._status = "REJECTED" return "Transfer rejected by client" @workflow.signal def retry_with_correction(self, corrected_account: str) -> None: self._corrected_account = corrected_account @workflow.signal def approve(self, approved: bool) -> None: self._approval = approved @workflow.query def get_status(self) -> str: return self._status

Go Resumable Activity example with correction and approval signals

package transfer import ( "fmt" "time" "go.temporal.io/sdk/temporal" "go.temporal.io/sdk/workflow" ) type TransferInput struct { FromAccount string ToAccount string Amount float64 } func TransferWorkflow(ctx workflow.Context, input TransferInput) (string, error) { status := "PENDING" if err := workflow.SetQueryHandler(ctx, "getStatus", func() (string, error) { return status, nil }); err != nil { return "", err } correctionCh := workflow.GetSignalChannel(ctx, "retryWithCorrection") approvalCh := workflow.GetSignalChannel(ctx, "approve") ao := workflow.ActivityOptions{ StartToCloseTimeout: 30 * time.Second, RetryPolicy: &temporal.RetryPolicy{MaximumAttempts: 3}, } actCtx := workflow.WithActivityOptions(ctx, ao) account := input.ToAccount correctionCount := 0 for { status = "TRANSFERRING" err := workflow.ExecuteActivity(actCtx, ExecuteTransfer, TransferInput{ FromAccount: input.FromAccount, ToAccount: account, Amount: input.Amount, }).Get(actCtx, nil) if err == nil { break } correctionCount++ if correctionCount > 5 { status = "FAILED" _ = workflow.UpsertSearchAttributes(ctx, map[string]interface{}{"TransferStatus": status}) return "", err } status = "AWAITING_CORRECTION" _ = workflow.UpsertSearchAttributes(ctx, map[string]interface{}{"TransferStatus": status}) workflow.GetLogger(ctx).Warn("Transfer failed — waiting for account correction", "to_account", account) var corrected string _ = workflow.Await(ctx, func() bool { return correctionCh.ReceiveAsync(&corrected) }) account = corrected } status = "AWAITING_APPROVAL" var approved bool _ = workflow.Await(ctx, func() bool { return approvalCh.ReceiveAsync(&approved) }) if approved { status = "COMPLETED" return fmt.Sprintf("Transfer of %.2f to %s completed", input.Amount, account), nil } status = "REJECTED" return "Transfer rejected by client", nil }

Java Resumable Activity example with correction and approval signals

import io.temporal.activity.ActivityOptions; import io.temporal.common.RetryOptions; import io.temporal.common.SearchAttributeKey; import io.temporal.failure.ActivityFailure; import io.temporal.workflow.SignalMethod; import io.temporal.workflow.QueryMethod; import io.temporal.workflow.WorkflowInterface; import io.temporal.workflow.WorkflowMethod; import io.temporal.workflow.Workflow; import java.time.Duration; @WorkflowInterface public interface TransferWorkflow { @WorkflowMethod String run(TransferInput input); @SignalMethod void retryWithCorrection(String correctedAccount); @SignalMethod void approve(boolean approved); @QueryMethod String getStatus(); } public class TransferWorkflowImpl implements TransferWorkflow { private String status = "PENDING"; private String correctedAccount; private Boolean approval; private final TransferActivities activities = Workflow.newActivityStub( TransferActivities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(30)) .setRetryOptions(RetryOptions.newBuilder() .setMaximumAttempts(3) .build()) .build() ); @Override public String run(TransferInput input) { String account = input.getToAccount(); int correctionCount = 0; while (true) { status = "TRANSFERRING"; try { activities.executeTransfer( new TransferInput(input.getFromAccount(), account, input.getAmount()) ); break; } catch (ActivityFailure e) { correctionCount++; if (correctionCount > 5) { status = "FAILED"; Workflow.upsertTypedSearchAttributes( SearchAttributeKey.forKeyword("TransferStatus").valueSet(status) ); throw e; } status = "AWAITING_CORRECTION"; Workflow.upsertTypedSearchAttributes( SearchAttributeKey.forKeyword("TransferStatus").valueSet(status) ); Workflow.getLogger(getClass()).warn( "Transfer failed — waiting for account correction: " + account ); Workflow.await(() -> correctedAccount != null); account = correctedAccount; correctedAccount = null; } } status = "AWAITING_APPROVAL"; Workflow.await(() -> approval != null); if (approval) { status = "COMPLETED"; return String.format("Transfer of %.2f to %s completed", input.getAmount(), account); } status = "REJECTED"; return "Transfer rejected by client"; } @Override public void retryWithCorrection(String account) { this.correctedAccount = account; } @Override public void approve(boolean decision) { this.approval = decision; } @Override public String getStatus() { return status; } }

TypeScript Resumable Activity example with correction and approval signals

import * as wf from '@temporalio/workflow'; import type * as activities from './activities'; export interface TransferInput { fromAccount: string; toAccount: string; amount: number; } export const retryWithCorrectionSignal = wf.defineSignal<[string]>('retryWithCorrection'); export const approveSignal = wf.defineSignal<[boolean]>('approve'); export const getStatusQuery = wf.defineQuery<string>('getStatus'); const { executeTransfer } = wf.proxyActivities<typeof activities>({ startToCloseTimeout: '30s', retry: { maximumAttempts: 3 }, }); export async function transferWorkflow(input: TransferInput): Promise<string> { let status = 'PENDING'; let correctedAccount: string | undefined; let approval: boolean | undefined; wf.setHandler(retryWithCorrectionSignal, (account: string) => { correctedAccount = account; }); wf.setHandler(approveSignal, (decision: boolean) => { approval = decision; }); wf.setHandler(getStatusQuery, () => status); let account = input.toAccount; let correctionCount = 0; while (true) { status = 'TRANSFERRING'; try { await executeTransfer({ ...input, toAccount: account }); break; } catch (err) { correctionCount++; if (correctionCount > 5) { status = 'FAILED'; wf.upsertSearchAttributes({ TransferStatus: [status] }); throw err; } status = 'AWAITING_CORRECTION'; wf.upsertSearchAttributes({ TransferStatus: [status] }); wf.log.warn('Transfer failed — waiting for account correction', { account }); await wf.condition(() => correctedAccount !== undefined); account = correctedAccount!; correctedAccount = undefined; } } status = 'AWAITING_APPROVAL'; await wf.condition(() => approval !== undefined); if (approval) { status = 'COMPLETED'; return `Transfer of ${input.amount} to ${account} completed`; } status = 'REJECTED'; return 'Transfer rejected by client'; }

gRPC message too large terminates Workflow Execution immediately

When a Workflow Task response is bigger than the gRPC message size limit, the Temporal Service terminates the Workflow Execution immediately with TERMINATED status and no retry. Affected Executions end permanently and someone must restart them by hand.

gRPC message too large failure reason

When the Worker tries RespondWorkflowTaskCompleted and the response is rejected for being too large by the gRPC library, proxy, load balancer, or the Temporal Service, the SDK follows up with RespondWorkflowTaskFailed with cause WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE.

Oversized payload mitigation strategies

To handle oversized Activity inputs or outputs, move the payload out of band by putting it in blob storage and passing a reference through Event History. For accumulated Signals or Updates, rate-limit senders or batch Signals. For too many commands in one response, break the fan-out into smaller batches across multiple Workflow Tasks.

Workflow Task execution latency default timeout

The default Workflow Task timeout is 10 seconds. When Workflow Task execution latency reaches or exceeds this value, the Temporal Service actively times out Workflow Tasks.

TemporalReportedProblems Search Attribute tracks repeated Workflow Task failures

The Temporal Service sets the TemporalReportedProblems Search Attribute on Executions experiencing repeated Workflow Task failures. This can be used in the Temporal UI to find affected Executions.

temporal_request_failure metric increments on non-OK gRPC status

The metric `temporal_request_failure` increments when the Temporal Service returns a non-OK gRPC status code on a standard operation. It carries `namespace`, `operation`, and `status_code` tags.

temporal_long_request_failure metric for poll and history operations

The metric `temporal_long_request_failure` covers poll operations and long-poll `GetWorkflowExecutionHistory`. It carries `namespace`, `operation`, and `status_code` tags.

Request latency alert threshold for user-facing operations

The recommended alert for `temporal_request_latency` on user-facing operations (StartWorkflowExecution, SignalWithStartWorkflowExecution, SignalWorkflowExecution, ExecuteMultiOperation) fires when p99 goes above 2 seconds, held for 5 minutes, routed Critical by default.

User-facing operations block application code

Operations that applications call synchronously (StartWorkflowExecution, SignalWithStartWorkflowExecution, SignalWorkflowExecution, UpdateWorkflowExecution, ExecuteMultiOperation) block your application code while they wait on the Temporal Service, so the latency is felt directly by your users and by anything downstream of the call completing.

Request latency includes serialization and network time

Request latency metric on user-facing operations includes serialization and network time. Large Workflow inputs or Signal payloads, or an expensive Payload Codec, raise it without any Service-side slowdown.

Temporal Cloud namespace service limits

On Temporal Cloud, Namespaces have service limits on throughput and concurrency. When RESOURCE_EXHAUSTED fires with a rate limit cause on user-facing operations, compare current throughput against your Namespace's service limits and open a support request if you need them raised.

Check temporal_workflow_task_execution_latency for Workflow Task timeout diagnosis

For Workflow Task NOT_FOUND on respond operations, check `temporal_workflow_task_execution_latency`. If p99 is at or above the corresponding timeout, that is the direct cause of the timeout.

Identity field identifies which Worker executed a task

Look at the `identity` field in the `WorkflowTaskStarted` or `ActivityTaskStarted` event to identify which Worker ran the task, then check that pod for CPU saturation and cold-start delays.

temporal_workflow_endtoend_latency spike causes

temporal_workflow_endtoend_latency represents total Workflow Execution time from Schedule to closure for a single Workflow Run. Unexpected spikes can be caused by: complex Workflows with many Activities or long-running Activities, Workflow and Activity retries with frequent failures, insufficient Worker capacity or overloaded Workers, external dependencies (databases, APIs) that are slow or unreliable, and network latency with Workers in different region from cluster. Diagnosis: review Workflow and Activity designs for efficiency, monitor Workers for sufficient capacity, monitor external dependencies performance, and ensure Workers in same region as cluster.

Workflow Task execution latency high metric

The metric temporal_workflow_task_execution_latency is used to confirm that something is holding slots when task slots are exhausted. Sustained high values indicate Workflow Task execution is being delayed.

Workflow Task execution failed metric

The metric temporal_workflow_task_execution_failed counts Workflow Task failures. If polling is active but completions are zero, check this metric as Workers may be failing every Task.

Give your agent this brain