Child Workflow versus Activity comparison
A Workflow models composite operations that consist of multiple Activities or other Child Workflows. An Activity usually models a single operation on the external world. When in doubt, use an Activity.
Temporal · Concepts · all subjects
design-patterns
826 notes in this subject, read out of this brain and free to use. This is page 9 of 14.
Child Workflows more expensive than Activities in Event History
Child Workflow Executions result in more overall Events recorded in Event Histories than Activities. Because each entry in an Event History is a cost in terms of compute resources, this could become a factor in very large workloads. It is recommended to start with a single Workflow implementation that uses Activities until there is a clear need for Child Workflows.
Do not use Child Workflows for code organization alone
There is no reason to use Child Workflows just for code organization. Object oriented structure and other code organization techniques can be used to deal with complexities. It is typically recommended to start from a single Workflow Definition if the problem has bounded size in terms of the number of Activity Executions and processed Signals.
Child Workflow state tracking in Event History
Temporal tracks all state changes within a Child Workflow Execution in Event History. Only the input, output, and retry attempts of an Activity Execution are tracked.
Parent Close Policy per-child configuration
Each Child Workflow Execution may have its own Parent Close Policy. This policy applies only to Child Workflow Executions and has no effect otherwise. You can set policies per child, which means you can opt out of propagating terminates or cancels on a per-child basis.
Parent Close Policy for asynchronous child workflows
Parent Close Policy can be used to start Child Workflows asynchronously by setting different policies per child to prevent the parent terminating or cancelling the child.
Parent Close Policy values
There are three possible values for Parent Close Policy: Abandon (the Child Workflow Execution is not affected), Request Cancel (a Cancellation request is sent to the Child Workflow Execution), and Terminate (the Child Workflow Execution is forcefully Terminated, and is the default).
Codec Server Namespace-level authorization
Each request from the Web UI includes an X-Namespace header identifying the Namespace. To enforce Namespace-level access control, your Codec Server must validate whether the authenticated user has permissions for the requested Namespace. This applies regardless of which authentication approach you use.
Codec Server vs Payload Codec
A Payload Codec encodes and decodes Payloads by applying encryption, compression, or other byte-level transformations; it runs in-process inside Workers and Clients and also inside the Codec Server; it is called by the Temporal SDK automatically on every serialization and deserialization. A Codec Server hosts a Payload Codec as an HTTP service so the Web UI and CLI can encode and decode Payloads remotely; it runs as a standalone HTTP service in your environment with a Payload Codec inside it; it is called by the Web UI and CLI over HTTP when viewing or submitting Payload data. Both have access to encryption keys.
Codec Server HTTPS requirement
Use HTTPS for any Codec Server deployment that is not strictly loopback (localhost).
Codec Server network-level security
Restrict network access to the Codec Server. The Web UI can communicate with a Codec Server that is only accessible on localhost, so running the Codec Server locally is a viable security pattern. For team access, place the Codec Server behind a VPN.
Codec Server authentication approaches
The Web UI supports two authentication approaches: (1) Include cross-origin credentials (recommended) - enable in Web UI Codec Server settings, browser sends cookies scoped to the Codec Server's domain, your Codec Server must have its own authentication mechanism so the user independently authenticates; (2) Pass access token - enable in Web UI settings, Web UI includes the same JWT used to log into Temporal UI in the Authorization header, your Codec Server validates the token signature against the OpenID Connect (OIDC) provider's JSON Web Key Set (JWKS) endpoint.
Codec Server solves encrypted payload visibility
When custom Payload Codecs apply encryption or compression, data stored in the Temporal Service is encoded and the Service cannot decode it. Without a Codec Server, the Web UI and CLI display raw encoded payloads. A Codec Server gives the Web UI and CLI a way to decode payloads on demand without exposing keys to the Temporal Service.
Temporal Cloud JWKS endpoint
For token validation on Temporal Cloud, verify the JWT against the Temporal Cloud JWKS endpoint at https://login.tmprl.cloud/.well-known/jwks.json. The audience claim should be validated against https://saas-api.tmprl.cloud.
Codec Server endpoints
A Codec Server exposes two HTTP POST endpoints: /encode accepts plaintext payloads and returns encoded payloads for sending; /decode accepts encoded payloads and returns decoded payloads for retrieving. Both endpoints receive and respond with a JSON body containing a payloads array of Payload objects.
Codec Server use cases
Common reasons to run a Codec Server include: debugging Workflows by viewing decoded inputs, outputs, and Event History in the Web UI instead of reading base64-encoded or encrypted blobs; operating from the CLI with readable data even when payloads are encrypted at rest; encoding inputs from the UI and CLI so the Temporal Service never sees plaintext; and providing compliance and access control by running in your environment to control who can decode payloads and under what conditions.
Codec Server consistency with Workers
Your Codec Server should use the same Payload Codec implementation as your Workers to ensure consistent encoding and decoding.
Codec Server with External Storage endpoints
When using External Storage, create a handler with NewPayloadHTTPHandler and PayloadHTTPHandlerOptions. The handler applies storage drivers and codecs in the correct order. Three endpoints become available: /decode still decodes encoded payloads and handles storage references (uses ?preserveStorageRefs=true to return storage references as-is without retrieval); /download retrieves actual payload data from external storage and decodes it through the Payload Codec; /encode applies the Payload Codec, then uploads payloads exceeding the size threshold to external storage and replaces them with reference tokens.
NewPayloadHTTPHandler vs NewPayloadCodecHTTPHandler
Do not use NewPayloadHTTPHandler as a target for a remote Data Converter or remote codec on your Workers. NewPayloadHTTPHandler runs the full encode-store-encode and decode-retrieve-decode pipeline. For remote codecs on Workers, use NewPayloadCodecHTTPHandler separately. If you need both, set up NewPayloadHTTPHandler for the Web UI and CLI alongside NewPayloadCodecHTTPHandler for your Workers, and configure both with the same codecs.
Codec Server security: treat like a Worker
Because a Codec Server can decode sensitive data, treat it with the same trust as a Worker. Anyone who can call it has effective decrypt access.
Recommended approach to time-based workflow actions
If you need to perform an action inside your Workflow after a specific period of time, use a Timer instead of setting a Workflow Timeout.
Non-deterministic operations allowed in Activities, not Workflows
While Workflows themselves must be deterministic, Temporal applications do not need to be entirely deterministic. Non-deterministic operations that interact with the external world are absolutely supported through Activities: calling LLM APIs, querying databases, reading or writing files, and making HTTP requests to external services. This gives Workflow determinism with Activity flexibility and built-in retry support.
Nexus operation error propagation from underlying workflow
Errors from an asynchronous Operation's underlying Workflow propagate back to the caller.
Nexus multi-level calls across namespaces
Nexus supports end-to-end execution debugging across caller Workflows, Nexus Operations, and handler Workflows, even across multi-level calls spanning multiple Namespaces.
Nexus bi-directional linking across namespaces
Nexus supports bi-directional links connecting Nexus Operation events in the caller's Event History to corresponding events in the handler's Event History. Forward links go from a caller's Nexus Operation event to the handler's Workflow. Backward links go from the handler's Workflow back to the caller's Nexus Operation event. SDK builder functions like New-Workflow-Run-Operation automatically wire these links, enabling click-through navigation across Namespaces, regions, and clouds in the Temporal UI.
Pending Nexus Operations visibility
Pending Nexus Operations are displayed in the UI on the Workflow details page and can be listed from the CLI using the `temporal workflow describe` command. The output shows Endpoint, Service, Operation, OperationToken, State, Attempt, ScheduleToCloseTimeout, NextAttemptScheduleTime, LastAttemptCompleteTime, and LastAttemptFailure fields.
Retryable vs non-retryable Nexus errors
Retryable errors surface in the Pending Operation display. Non-retryable errors resolve the Operation with a Failed, TimedOut, or Canceled event.
Pending Callbacks for asynchronous Nexus operations
Nexus completion callbacks are sent from the handler's Namespace to the caller's Namespace for asynchronous Operations. These can be viewed in the UI or from the CLI using the `temporal workflow describe` command. The output shows URL, Trigger, State, Attempt, and RegistrationTime fields.
Temporal tracing with OpenTelemetry and OpenTracing
Temporal integrates with OpenTelemetry and OpenTracing to visualize call graphs across Activities, Nexus Operations, and Child Workflows. Tracing is enabled by installing an interceptor on the Client or Worker. Sample implementations are available in the temporalio samples repositories for Go, Java, Python, TypeScript, and .NET SDKs.
Nexus router-queue pattern: dedicated routing worker
The router-queue pattern separates Nexus routing from Workflow execution by using a dedicated Nexus Worker on a dedicated router Task Queue to route Operations to Workflows on other Task Queues in the same Namespace. The Nexus Endpoint's target Task Queue points to the router Task Queue. In each Nexus Operation handler, you specify a different target Task Queue in the Workflow start options. Existing Workers continue to poll their own Task Queues and execute the Workflows started by the router.
Nexus router-queue pattern: when to use
Use the router-queue pattern when you need to scale Nexus routing independently from Workflow execution, want a single Nexus Worker routing requests to multiple Workflow types on different Task Queues, worker fleets have different IAM permissions for different underlying resources, or need to add a router Worker to a Namespace without changing existing Workers or Workflows.
Nexus router-queue pattern: production usage
The router-queue pattern is used in production by organizations running self-service platforms where a central gateway routes requests to domain-specific Namespaces and Task Queues. The router Worker is lightweight and only handles routing logic.
Nexus collocated pattern: single worker and task queue
The collocated pattern runs Nexus Operation handlers in the same Worker and on the same Task Queue as the underlying Workflows. The Nexus Endpoint targets the same Task Queue used by the underlying Workflows. A single Worker registers both Nexus Services and Workflow types, so everything runs together. This is the default and simplest deployment.
Nexus collocated pattern: use Eager Workflow Start
When a Nexus handler starts a Workflow in the same Worker using the collocated pattern, you can use Eager Workflow Start to execute the first Workflow Task locally without an extra call to the Temporal Server while still recording durable state. If the process crashes, the Workflow resumes on another Worker.
Nexus collocated pattern: when to use
Use the collocated pattern by default: when getting started with Nexus, the same team owns both the Nexus Service and underlying Workflows, you don't need to scale Nexus routing separately from Workflow execution, or when setting up a simple test environment.
Nexus Services are named collections of Nexus Operations
Nexus Services are named collections of arbitrary-duration Nexus Operations that provide a contract for sharing across team boundaries.
Services typically run alongside Workflows or in dedicated router Worker
Services typically run alongside the Workflows they abstract, or in a dedicated router Worker using the router-queue pattern.
Callers reference Service by name when executing Nexus Operation
Callers reference a Service by name when executing a Nexus Operation.
Nexus Endpoint exposes Services
A Nexus Endpoint exposes Services for callers to use.
Automatic retries for Nexus Operations
Once the caller Workflow schedules an Operation with the caller's Temporal Service, the caller's Nexus Machinery keeps trying to start the Operation. If a retryable Nexus error is returned, the Nexus Machinery will retry until the Nexus Operation's Schedule-to-Start timeout or Schedule-to-Close timeout is exceeded. If a Nexus handler returns a retryable error or an upstream timeout is encountered by the caller, the Nexus request will be retried up to the default Retry Policy's max attempts and expiration interval.
Schedule-to-Close timeout for Nexus Operations
The Schedule-to-Close timeout limits the total duration from when the Operation is scheduled to when it completes. This is the overall timeout for the entire Operation. The Nexus Machinery automatically retries failed requests internally until this timeout is exceeded, at which point the Operation fails with a NexusOperationTimedOut event. This timeout covers the full Nexus Operation lifecycle. In Temporal Cloud, the maximum Schedule-to-Close timeout is 60 days.
Schedule-to-Start timeout for Nexus Operations
The Schedule-to-Start timeout limits how long the caller is willing to wait for the Operation to be started (or completed, if synchronous) by the handler. If the Operation is not started within this timeout, it fails with TIMEOUT_TYPE_SCHEDULE_TO_START. If not set or set to zero, no Schedule-to-Start timeout is enforced. This timeout requires Temporal Server version 1.31.0 or later.
Start-to-Close timeout for Nexus Operations
The Start-to-Close timeout limits how long the caller is willing to wait for an asynchronous Operation to complete after it has been started. If the Operation does not complete within this timeout after starting, it fails with TIMEOUT_TYPE_START_TO_CLOSE. This timeout only applies to asynchronous Operations. Synchronous Operations ignore this timeout because they complete as part of the start request. If not set or set to zero, no Start-to-Close timeout is enforced. This timeout requires Temporal Server version 1.31.0 or later.
Circuit breaker activation and thresholds
Nexus implements circuit breaking per caller-Namespace/Endpoint pair. By default, the circuit breaker activates after 5 consecutive retryable errors.
Circuit breaker state transitions
After tripping, the circuit breaker enters the open state and stops sending requests. After 60 seconds, it transitions to half-open, allowing a single probe request. If the probe succeeds, the circuit breaker returns to closed (normal operation). If it fails, the circuit breaker returns to open for another 60 seconds.
Circuit breaker trip conditions
Different Operations within the same destination pair contribute to the trip count. Worker availability affects the circuit breaker as well. If no workers are polling the handler task queue due to a deployment issue, crash, or scale-down, Nexus requests will time out. Consecutive timeouts count as retryable errors and will trip the circuit breaker just as application-level errors do.
At-least-once execution semantics for Nexus Operations
The Nexus Machinery provides reliable execution with at-least-once execution semantics for a Nexus Operation, until the caller's Schedule-to-Close timeout is exceeded. The Machinery retries on handler timeouts or retryable errors, so a handler may be invoked multiple times for the same Operation. Nexus Operation handlers should be idempotent, similar to Activities.
Nexus Operation cancellation propagation
Cancelling a caller Workflow automatically propagates to all pending Nexus Operations and their underlying handler Workflows. A canceled handler Workflow reports a Canceled Failure to the caller.
Nexus Operation termination behavior
Terminating a caller Workflow abandons all pending Nexus Operations. Unlike cancellation, no cancel request is sent to the handler Namespace, so handler Workflows continue running indefinitely, consuming resources until they time out or are manually stopped. Because the handler runs in a separate Namespace, it has no signal that the caller is gone, making orphaned Operations difficult to detect and correlate. If the Nexus Operation was part of a multi-step process, termination also leaves no opportunity to run compensation logic, potentially leaving the system in a partially completed state. Cancellation is preferred when possible.
Versioning Nexus services with Task Routing
Task Routing is the simplest way to version Nexus service code. For backward-incompatible changes, use a different Service name and Task Queue (for example, prod.payments.v2). Callers migrate to the new version on their own deployment schedule.
Multiple callers attaching to a handler Workflow
Operations started with New-Workflow-Run-Operation automatically attach a completion Callback to the handler Workflow. Additional callers can attach to the same handler Workflow using a Conflict-Policy of Use-Existing. Each handler Workflow has a Callback limit (configurable for self-hosted, see Cloud limits for Temporal Cloud).
Continue-As-New with Nexus completion Callbacks
When a handler Workflow uses Continue-As-New, existing completion Callbacks are copied to the new Execution. The previous Execution's Callbacks remain in Standby state indefinitely.
Exactly-once execution semantics for Nexus Operations
To upgrade to exactly-once semantics, back your Operation with a Workflow that uses a WorkflowIDReusePolicy of RejectDuplicates. This allows only one Workflow Execution per Workflow ID within a Namespace for the Retention Period.
Nexus Operation types: synchronous vs asynchronous
Nexus Operations can be synchronous or asynchronous. Unlike a traditional RPC, an asynchronous Nexus Operation has an operation token that can be used to re-attach to a long-running Operation backed by a Workflow.
When to use synchronous Nexus Operations
Use a synchronous Nexus Operation only when the complete execution path is highly reliable, has predictably low latency, and finishes well within the 10-second handler deadline.
When to use asynchronous Nexus Operations
Use an asynchronous Nexus Operation when latency or availability is uncertain, the work might exceed the handler deadline, or execution depends on a potentially unreliable service or database.
Synchronous Operation handler deadline
Synchronous Operations must complete within the 10-second handler deadline, as measured from the caller's Nexus Machinery. Timed-out handlers are retried until the Operation's Schedule-to-Close timeout is exceeded.
Asynchronous Operation maximum duration
Asynchronous Operations can run up to 60 days, which is the maximum Schedule-to-Close timeout in Temporal Cloud.
Nexus Operation lifecycle overview
An Operation's lifecycle spans scheduling, reliable delivery with retries, handler execution, and result or callback completion. When a caller Workflow executes a Nexus Operation, the command is atomically handed off to the Nexus Machinery, which ensures at-least-once execution with automatic retries and reliable result delivery.
New-Workflow-Run-Operation handler function
New-Workflow-Run-Operation starts a Workflow as an asynchronous Operation. It is used on the handler side to define asynchronous Nexus Operations.