architecture/services

36 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Temporal Service components: Frontend, History, and Matching

The Temporal Service is a group of backend services forming the core Temporal system. It exposes a Frontend Service, which is the network API that everything connects to. Backend services include the History Service, which keeps the full Event History and state for each Workflow Execution, and the Matching Service, which manages Task Queues that hold pending work items for Workers.

Nexus Endpoint description field

The Endpoint description field supports markdown for documenting available Operations, contact information, or schema links.

Deploying a Nexus Endpoint

Adding an Endpoint to the Nexus Registry deploys it immediately. The Endpoint is available at runtime as soon as it is registered.

Nexus Endpoint target type: Worker

The EndpointSpec supports a target type of Worker, which routes to a target Namespace and Task Queue.

Nexus Endpoint is not a general-purpose proxy

A Nexus Endpoint acts as a reverse proxy for a single Nexus Service, routing requests to one target Namespace and Task Queue. Unlike general-purpose proxies, it does not route to multiple backends. Instead, it provides a secure, managed connection to a specific upstream target, which can be in any region or cloud.

Nexus OSS Cluster metrics from History, Concurrency Limiter, and Frontend Services

OSS Cluster Nexus metrics are emitted from History Service, Concurrency Limiter, and Frontend Service.

Nexus Cloud metrics for caller and handler namespaces

Temporal Cloud emits Nexus metrics including: RespondWorkflowTaskCompleted (schedule a Nexus Operation) in the caller namespace, and PollNexusTaskQueue (get a Nexus Task to process), RespondNexusTaskCompleted (report success), and RespondNexusTaskFailed (report failure) in the handler namespace.

Nexus RPC protocol

Nexus Machinery uses Nexus RPC on the wire—a protocol supporting arbitrary-duration Operations. Users interact only with the Temporal SDK, not Nexus RPC directly.

Nexus 10-second handler deadline

Synchronous Nexus Operations must complete within a 10-second handler deadline.

Nexus 60-day asynchronous operation limit

Asynchronous Nexus Operations can run up to 60 days.

Nexus Service and Operations

A Nexus Service is a named collection of Nexus Operations that a team exposes. Operations abstract the underlying implementation—callers do not need to know whether an Operation starts a Workflow, sends a Signal, runs a Query, or executes other reliable code.

Nexus Operation lifecycle modes

Nexus Operations support two modes: Asynchronous Operations start a Workflow (same or different Task Queue, optionally with Eager Start) and can run up to 60 days. Synchronous Operations complete within the 10-second handler deadline and are used for Signals, Queries, Updates, or other reliable low-latency calls using the Temporal SDK Client.

When to use Synchronous vs Asynchronous Nexus Operations

Use Synchronous Operations only when the complete execution path is highly reliable, has predictably low latency, and finishes well within the 10-second handler deadline. Otherwise, use Asynchronous Operations.

Nexus Endpoint definition and purpose

A Nexus Endpoint is a reverse proxy that decouples callers from handlers. Callers reference an Endpoint by name. The Endpoint routes requests to a target Namespace and Task Queue. Callers never need to know the handler's Namespace, Task Queue, or internal implementation.

Nexus Registry

Endpoints are managed in the Nexus Registry using the UI, CLI, or Cloud Ops API.

Nexus Machinery execution semantics

When a caller Workflow executes a Nexus Operation, the Nexus Machinery handles delivery with at-least-once execution semantics. It provides automatic retries with exponential backoff, rate limiting and concurrency limiting, circuit breaking (which trips after 5 consecutive retryable errors), and automatic load balancing.

Dynamic configuration is optional

Setting dynamic configuration for your Temporal Service is optional. Dynamic configuration keys are used to fine-tune your self-deployed Temporal Service setup.

Temporal emits Prometheus-format metrics by default

Temporal emits metrics by default in a format that is supported by Prometheus. Any metrics software that supports the same format can be used. Currently tested with Prometheus >= v2.0 and Grafana >= v2.5.

Temporal Cloud emits metrics through Prometheus HTTP API

Temporal Cloud emits metrics through a Prometheus HTTP API endpoint, which can be directly used as a Prometheus data source in Grafana or to query and export Cloud metrics to any observability platform.

Self-hosted Temporal Service requires exposing Prometheus endpoints

On the self-hosted Temporal Service, you must expose Prometheus endpoints in your Temporal Service configuration and configure Prometheus to scrape metrics from the endpoints. You can then set up your observability platform (such as Grafana) to use Prometheus as a data source.

Temporal Service configuration is YAML-based

Temporal Service configuration is the setup and configuration details of your self-hosted Temporal Service, defined using YAML. It must be defined when setting up a self-hosted Temporal Service.

Static configuration read once at startup

Static configuration contains details of how the Temporal Service should be set up and is read just once to configure service nodes at startup. Static configuration values cannot be changed at runtime.

Static configuration covers services, persistence, security, and observability

Static configuration must contain details for setting up Temporal Services (Frontend, History, Matching, Worker), membership ports, persistence (including History Shard count), Visibility, Archival store setups, TLS, authentication, authorization, server log level, metrics, Temporal Service metadata, and Dynamic config Client.

Some static configuration changes require service restart

Some values in static configuration, such as Metrics configuration or Server log level, can be changed in the static configuration but require restarting the Temporal Service for the changes to take effect.

Dynamic configuration can be updated without restart

Dynamic configuration contains configuration keys that you can update in your Temporal Service setup without having to restart the server processes. All dynamic configuration keys provided by Temporal have default values, which can be overridden by setting different values in a YAML file and configuring the dynamic configuration client to poll this file for updates.

Dynamic configuration changes effective immediately or require restart

Setting overrides for some dynamic configuration keys updates the Temporal Service configuration immediately. However, for configuration fields that are checked at startup (such as thread pool size), you must restart the server for the changes to take effect.

Temporal provides ClaimMapper and Authorizer plugins for authentication and authorization

Temporal offers two plugin interfaces for authentication and authorization of API calls: ClaimMapper and Authorizer. The logic of both plugins can be customized to fit a variety of use cases. When plugins are provided, the Frontend Service invokes their implementation before running the requested operation.

History Service responsibilities

The History Service is responsible for persisting Workflow Execution state to the Event History. When a Workflow Execution is able to progress, the History Service adds a Task with the Workflow's updated history to the Task Queue. From there, a Worker can poll for work, receive this updated history, and resume execution. It uses grpcPort 7234 for the service handler and port 6934 for membership-related communication.

Frontend Service responsibilities

The Frontend Service is a stateless gateway service that exposes a Proto API. It is responsible for rate limiting, authorizing, validating, and routing all inbound calls. Inbound request rate limiting is applied per host and per namespace. It handles Namespace CRUD, external events, worker polls, Visibility requests, Temporal CLI operations, and calls from remote Temporal Services related to Multi-Cluster Replication. It uses grpcPort 7233 for the service handler and port 6933 for membership-related communication.

Frontend Service routing

Every inbound request related to a Workflow Execution must have a Workflow Id, which is hashed for routing purposes. The Frontend Service has access to hash rings that maintain service membership information, including how many nodes (instances of each service) are in the Temporal Service.

History Service and History Shard ratio

The total number of History Service processes can be between 1 and the total number of History Shards. An individual History Service can support many History Shards. Temporal recommends starting at a ratio of 1 History Service process for every 500 History Shards. Although the total number of History Shards remains static for the life of the Temporal Service, the number of History Service processes can change.

Matching Service responsibilities

The Matching Service is responsible for hosting user-facing Task Queues for Task dispatching. It is responsible for matching Workers to Tasks and routing new Tasks to the appropriate queue. It uses grpcPort 7235 for the service handler and port 6935 for membership-related communication.

Temporal Cloud as alternative to self-hosted service

Temporal Cloud is a software as a service (SaaS) offering that provides an alternative to hosting the Temporal Service yourself.

Temporal Service internal workers for system workflows

The Temporal Service itself has internal workers for system Workflow Executions, but these internal workers are not visible to the developer.

gRPC endpoints for Temporal Cloud Namespaces

Every Temporal Namespace uses a gRPC endpoint for communication. Temporal Clients communicate with a Temporal Server via gRPC, a Remote Procedure Call framework featuring low latency and high performance. The gRPC endpoint appears on the detail page for each Cloud Namespace and can be copied from the Connect button on the Namespace detail page.

Temporal Service as choreographer

The Temporal Service functions more as a choreographer than a conductor. Rather than directly assigning tasks to Workers, the Temporal Service arranges the Tasks into a Task Queue while Workers poll the Task Queue. Developers may create a fleet of Workers and tune them so that a Task is picked up as soon as it is available. If a Worker goes down, Tasks can wait until the next Worker is available.

Give your agent this brain