architecture/persistence

28 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Server persists all state to database for durability

The Server records all state to a database, so the Server can remember what has happened even if processes or machines fail. Users can run a Server themselves (self-hosted cluster) or use Temporal Cloud, which is the same kind of service but operated for the user.

Default Data Converter encoding order

The default Data Converter in each Temporal SDK converts objects to bytes using a series of Payload Converters and supports binary, Protobufs, and JSON formats. Values are encoded in the following order: Null, Byte array, Protobuf JSON, JSON.

Protobuf encoding in default Data Converter

If a value is an instance of a Protobuf message, it is encoded with proto3 JSON. In SDKs that cannot determine parameter types at runtime (for example, TypeScript), Protobufs are not included in the default converter.

JSON encoding in default Data Converter

If a value is not null, binary, or a Protobuf, it is encoded as JSON. Most common input types including strings, integers, floating point numbers, and booleans are serializable as JSON. If any part of it is not serializable as JSON, an error is thrown.

Default Data Converter serializes by root type

The default Data Converter serializes objects based on their root type, rather than nested types. The JSON serializers of some SDKs cannot process lists with Protobuf children objects without implementing a custom Data Converter.

Custom Data Converter definition

A custom Data Converter extends the default Data Converter with custom logic for Payload conversion or encoding. Custom Data Converters can be created to alter formats (for example, using MessagePack instead of JSON) or add compression and encryption.

Payload Codec bytes-to-bytes conversion

A Payload Codec encodes and decodes Payloads with bytes-to-bytes conversion. To use custom encryption or compression logic, create a custom Payload Codec with encryption/compression logic in the encode function and decryption/decompression logic in the decode function.

Custom Data Converter implementation methods

To implement a custom Payload Codec, you can override the default Data Converter or create a customized Data Converter that defines its own Payload Converter. A customized Data Converter can have three components: Payload Converter, Payload Codec, and Failure Converter.

Search Attributes not encrypted by custom Data Converter

Custom Data Converters are not applied to all data. Search Attributes are persisted unencoded so they can be indexed for searching, and thus are not subject to custom Data Converter encryption or compression.

Failure Converter definition and purpose

A Failure Converter transforms error messages and call stacks generated by Workflows into encoded formats. The default Failure Converter copies error messages and call stacks as plain text, making them directly accessible in the Message field of Workflow Executions. This may be undesirable if errors contain privileged or sensitive information.

Failure messages not encoded by default

Failure messages and call stacks are not encoded as codec-capable Payloads by default. You must explicitly enable encoding these common attributes on failures if you need to protect sensitive information.

Encrypting failure messages and call stacks

To encrypt the message and call stack, configure the default Failure Converter to use your encoding. This moves the message and stack_trace fields to a Payload that is run through your codec.

Custom FailureConverter implementation

You can replace the NewDefaultFailureConverter with a custom method if you need different Converter logic for your Failures. This is useful when working with highly sensitive data and using sophisticated logging or observability implementations that require different encryption methods.

Persistence database role in Temporal Service

The Temporal Persistence store is a database used by the Temporal Server to persist events generated and processed in a Temporal Service and SDK. A Temporal Service's only required dependency for basic operation is the Persistence database.

Data types stored in Persistence database

The Persistence database stores four types of data: Tasks (to be dispatched), State of Workflow Executions (including an Execution table capturing mutable state and a History table with append-only Workflow Execution History Events), Namespace metadata, and Visibility data.

Supported databases for Persistence

Multiple types of databases are supported for Temporal Persistence. Temporal tests compatibility with Cassandra v3.11, v4.0, and v5.0.4 and later; PostgreSQL 13.18, 14.15, 15.10, and 16.6; MySQL v5.7 and v8.0 (specifically 8.0.19+). SQLite v3.x is also supported but only for development and testing, not production.

Advanced Visibility database requirements

For production environments, Elasticsearch is recommended as the Visibility store. With Temporal Server version 1.19.1 and earlier, Elasticsearch is required for advanced Visibility. With version 1.20 and later, advanced Visibility features are available on SQL databases (MySQL 8.0.17+, PostgreSQL 12+, SQLite v3.31.0+) and Elasticsearch.

Temporal database compatibility testing approach

Temporal tests compatibility by spanning the minimum and maximum stable major versions for each supported database. The versions listed in the documentation are used in test pipelines and actively tested before any Temporal version release.

Cassandra version constraints

Cassandra versions 5 earlier than 5.0.4 are not supported. Supported versions are Cassandra v3.11, v4.0, and v5.0.4 and later.

MySQL version constraint

For MySQL, specifically version 8.0.19 or later is required within the 8.0 series due to a bug in earlier 8.0 releases.

Database compatibility stability

Because Temporal Server primarily relies on core database functionality, compatibility is not expected to break often. Temporal has no opinions on database upgrade paths; as long as you can upgrade your database according to each project's specifications, Temporal should work with any version within supported ranges.

Untested database vendors

Temporal does not run tests with vendors like Vitess and CockroachDB.

History Shard purpose and scaling

A History Shard is a unit within a Temporal Service by which concurrent Workflow Execution throughput can be scaled. Each History Shard maps to a single persistence partition and assumes only one concurrent operation can be within a partition at a time. The number of History Shards represents the number of concurrent database operations that can occur for a Temporal Service, playing a significant role in the performance of a Temporal Application.

History Shard count is immutable

Before integrating a database, the total number of History Shards for the Temporal Service must be chosen and set in the Temporal Service's configuration. After the Shard count is configured and the database integrated, the total number of History Shards for the Temporal Service cannot be changed.

History Shard assignment algorithm

A History Shard is represented as a hashed integer. Each Workflow Execution is automatically assigned to a History Shard. The assignment algorithm hashes Workflow Execution metadata such as Workflow Id and Namespace and uses that value to match a History Shard.

History Shard internal Task Queues

Each History Shard maintains the Workflow Execution Event History, Workflow Execution mutable state, and the following internal Task Queues: Internal Transfer Task Queue (transfers internal tasks to the Matching Service when a new Workflow Task needs to be scheduled), Internal Timer Task Queue (durably persists Timers), Internal Replicator Task Queue (asynchronously replicates Workflow Executions from active Clusters to passive Clusters, relies on experimental Multi-Cluster feature), and Internal Visibility Task Queue (pushes data to the Advanced Visibility index).

History Shard deployment scales

A Temporal Service can operate with anywhere from 1 to 128K History Shards, with each Shard responsible for tens of thousands of Workflow Executions. One Shard is useful only in small scale setups designed for testing, while 128K Shards is useful only in very large scale production environments. Each History Shard adds compute overhead to the Temporal Service. The correct number of History Shards depends entirely on the Temporal Application being supported and the type of database.

Data security: Temporal Service reads Event History at rest

While data transferred in Event Histories is secured by mTLS in transit, it is still readable at rest in the Temporal Service by default. Temporal SDKs offer a Data Converter API to customize serialization of data going out of and coming back into a Worker Entity, which can guarantee that the Temporal Service cannot read sensitive business data.

Give your agent this brain