Data encoding and decoding flow in Temporal
Data encoding is performed by the client using converters and codecs when passing input to the Temporal Cluster. Plain text input is usually serialized into JSON, and can then be compressed or encrypted. Data decoding may be performed during Workflows or Activities as necessary, but decoded Workflow results are never persisted back to the Cluster. Instead, they are stored encoded on the Cluster, and additional parameters are needed when using the temporal workflow show CLI command or browsing the Web UI to view output.
Codec Server for decoding encrypted payloads
A Codec Server is an HTTP server that uses custom Codec logic to decode data remotely. It is independent of the Temporal Cluster and decodes encrypted payloads through predefined endpoints. You create, operate, and manage access to your Codec Server in your own environment. The Temporal CLI and Web UI provide built-in hooks to call the Codec Server to decode encrypted payloads on demand.
PayloadCodec interface in Ruby SDK
The Payload Codec needs to extend Temporalio::Converters::PayloadCodec and implement encode and decode methods. The encode method should convert payloads into new payloads by setting the 'encoding' metadata field. The decode method should convert encoded payloads back, also using the 'encoding' metadata field. Do not mutate the existing payloads.
Setting custom Payload Codec on Temporal Client in Ruby
When creating a Ruby client, the default DataConverter can be updated with a custom payload codec by passing it to the data_converter parameter:
```ruby
my_client = Temporalio::Client.connect(
'localhost:7233',
'my-namespace',
data_converter: Temporalio::Converters::DataConverter.new(payload_codec: Base64Codec.new)
)
```
PayloadCodec purpose and default
PayloadCodec has the purpose of transforming encoded payloads such as encryption and compression. By default, PayloadCodec is None, which means it operates as a passthrough without transformation.
PayloadConverter purpose and default
PayloadConverter has the purpose of serializing application data to bytes. The default PayloadConverter uses JSON serialization.
PayloadCodec interface for custom encryption
To implement client-side encryption of Payloads in the TypeScript SDK, create a class that implements the PayloadCodec interface. This codec is then provided as a custom dataConverter to both the Client and Worker.
EncryptionCodec example implementation
This example shows a complete EncryptionCodec class implementing PayloadCodec for the TypeScript SDK:
import { webcrypto as crypto } from 'node:crypto';
import { METADATA_ENCODING_KEY, Payload, PayloadCodec, ValueError } from '@temporalio/common';
import { temporal } from '@temporalio/proto';
import { decode, encode } from '@temporalio/common/lib/encoding';
import { decrypt, encrypt } from './crypto';
const ENCODING = 'binary/encrypted';
const METADATA_ENCRYPTION_KEY_ID = 'encryption-key-id';
export class EncryptionCodec implements PayloadCodec {
constructor(
protected readonly keys: Map<string, crypto.CryptoKey>,
protected readonly defaultKeyId: string,
) {}
static async create(keyId: string): Promise<EncryptionCodec> {
const keys = new Map<string, crypto.CryptoKey>();
keys.set(keyId, await fetchKey(keyId));
return new this(keys, keyId);
}
async encode(payloads: Payload[]): Promise<Payload[]> {
return Promise.all(
payloads.map(async (payload) => ({
metadata: {
[METADATA_ENCODING_KEY]: encode(ENCODING),
[METADATA_ENCRYPTION_KEY_ID]: encode(this.defaultKeyId),
},
data: await encrypt(
temporal.api.common.v1.Payload.encode(payload).finish(),
this.keys.get(this.defaultKeyId)!,
),
})),
);
}
async decode(payloads: Payload[]): Promise<Payload[]> {
return Promise.all(
payloads.map(async (payload) => {
if (!payload.metadata || decode(payload.metadata[METADATA_ENCODING_KEY]) !== ENCODING) {
return payload;
}
if (!payload.data) {
throw new ValueError('Payload data is missing');
}
const keyIdBytes = payload.metadata[METADATA_ENCRYPTION_KEY_ID];
if (!keyIdBytes) {
throw new ValueError('Unable to decrypt Payload without encryption key id');
}
const keyId = decode(keyIdBytes);
let key = this.keys.get(keyId);
if (!key) {
key = await fetchKey(keyId);
this.keys.set(keyId, key);
}
const decryptedPayloadBytes = await decrypt(payload.data, key);
return temporal.api.common.v1.Payload.decode(decryptedPayloadBytes);
}),
);
}
}
async function fetchKey(_keyId: string): Promise<crypto.CryptoKey> {
const key = Buffer.from('test-key-test-key-test-key-test!');
const cryptoKey = await crypto.subtle.importKey(
'raw',
key,
{
name: 'AES-GCM',
},
true,
['encrypt', 'decrypt'],
);
return cryptoKey;
}
The example uses AES-GCM encryption with 256-bit keys. In production, keys should be fetched from a key management system (KMS). The encode method encrypts entire payloads while preserving metadata and storing the encryption key ID. The decode method decrypts payloads by looking up the appropriate key using the stored key ID.
Use @ronomon/crypto-async for CPU-intensive encryption
Because encryption is CPU-intensive and doing AES with the crypto module built into Node.js blocks the main thread, use @ronomon/crypto-async, which uses the Node.js thread pool for encryption and decryption operations.
Provide dataConverter to Client for encryption
When creating a Temporal Client in the TypeScript SDK, provide the custom dataConverter with the EncryptionCodec to the Client constructor. This ensures Payloads are encrypted before being sent to the server.
Provide dataConverter to Worker for decryption
When creating a Temporal Worker in the TypeScript SDK, provide the custom dataConverter with the EncryptionCodec to the Worker.create() method. This ensures Payloads are decrypted when received by the worker.
Payload encryption example flow
When a Client sends data to a Workflow, it gets encrypted on the Client and decrypted in the Worker. The Workflow receives the decrypted message and can process it. When the Workflow returns data, the string gets encrypted by the Worker and decrypted by the Client.
When to customize Data Converter layers
Customize the PayloadCodec layer when your application requires encryption or compression. Customize the ExternalStorage layer when you need to offload large payloads. The default PayloadConverter with JSON serialization is sufficient for most applications that do not require non-JSON types.
Data Converter - three layers overview
All data sent to and from the Temporal Service passes through the Data Converter. The Data Converter has three layers: PayloadConverter, PayloadCodec, and ExternalStorage. Only the PayloadConverter is required; the PayloadCodec and ExternalStorage layers are optional.
ExternalStorage - optional layer
The ExternalStorage is an optional layer of the Data Converter. Its purpose is to offload large payloads to an external store. By default, all payloads are stored in Event History.
PayloadCodec - optional layer
The PayloadCodec is an optional layer of the Data Converter. Its purpose is to transform encoded payloads through operations like encryption or compression. The default behavior is passthrough (no transformation).
PayloadConverter - required layer
The PayloadConverter is the required layer of the Data Converter. Its purpose is to serialize application data to bytes. Temporal uses a default PayloadConverter that handles JSON serialization.
StorageDriver store method receives serialized Payloads
In the store() method, application data has already been serialized by the Payload Converter and Payload Codec before it reaches the driver. You receive an array of serialized Payload protobuf messages and serialize each to bytes before writing to storage.
External Storage runs outside Workflow sandbox
External Storage runs outside the Workflow sandbox, so you can pass the driver object directly to the Data Converter rather than referencing it by path the way a custom Payload Converter requires.
External Storage npm package installation GCS
For Google Cloud Storage external storage in TypeScript SDK, install:
```sh
npm install @temporalio/external-storage-gcs \
@temporalio/external-storage-gcs-google-sdk \
@google-cloud/storage
```
External Storage npm package installation S3
For Amazon S3 external storage in TypeScript SDK, install:
```sh
npm install @temporalio/external-storage-s3 \
@temporalio/external-storage-s3-aws-sdk \
@aws-sdk/client-s3
```
External Storage driver prerequisites
Prerequisites for using External Storage: (1) A bucket with read and write access; refer to lifecycle management to ensure payloads remain available for the entire Workflow lifetime; for multi-region durability see Durable External Storage; (2) Credentials with permission to write objects on components that store payloads and read objects on components that retrieve them; components that only retrieve payloads do not need write access and vice versa.
Dynamic bucket routing in External Storage
To route payloads to different buckets at runtime, pass a function as `bucket` parameter instead of a string. The function receives the store context and the payload, and returns a bucket name.
S3 MRAP driver configuration example
Example showing how to configure S3StorageDriver with a Multi-Region Access Point ARN:
```ts
const driver = new S3StorageDriver({
client: new AwsSdkS3StorageDriverClient(s3Client),
bucket: 'arn:aws:s3::123456789012:accesspoint/mfzwi23gnjvgw.mrap',
});
```
SigV4A signer installation for S3 MRAP
MRAP requests are signed with SigV4A. The AWS SDK for JavaScript does not bundle a SigV4A signer, so install it alongside existing dependencies:
```sh
npm install @aws-sdk/signature-v4a
```
Import the signer once during application startup to register it with the AWS SDK:
```ts
import '@aws-sdk/signature-v4a';
```
`@aws-sdk/signature-v4-crt` is an alternative implementation backed by AWS Common Runtime. When both are installed, the AWS SDK prefers the CRT implementation.
S3 Multi-Region Access Point with Cross-Region Replication
To make S3-backed External Storage tolerant of regional failures, configure the AWS side with Cross-Region Replication (CRR) and an S3 Multi-Region Access Point (MRAP), then point the driver at the MRAP ARN instead of a bucket name.
Custom storage driver filesystem for local development
A shared filesystem works for local development and for Workers that mount the same volume. For production or distributed scenarios, use a storage system that every Client and Worker can reach.
Multiple drivers configuration example
Example showing how to register two drivers with a selector:
```ts
const preferredDriver = new S3StorageDriver({
client: new AwsSdkS3StorageDriverClient(s3Client),
bucket: 'my-bucket',
});
const legacyDriver = new LegacyStorageDriver();
const externalStorage = new ExternalStorage({
drivers: [preferredDriver, legacyDriver],
driverSelector: () => preferredDriver,
});
```
Multiple S3 drivers require distinct names
Because S3StorageDriver defaults its `name` to 'aws.s3driver', registering two S3 drivers requires setting the `driverName` option on at least one of them to ensure each driver has a distinct name.
External Storage multiple drivers with driverSelector
When registering multiple drivers, you must provide a `driverSelector` function that chooses which driver stores each payload. ExternalStorage throws an error if more than one driver is registered without a selector. Returning null from the selector keeps a specific payload inline in Event History. All registered drivers remain available for retrieval, useful when migrating between storage backends.
Built-in S3 and GCS driver payload handling
Both built-in S3 and GCS drivers upload and download payloads concurrently, address objects by SHA-256 hash of their contents so identical payloads are stored once, and verify that hash on retrieve. Each driver rejects any single payload larger than `maxPayloadSize`, which defaults to 50 MiB, and includes diagnostic metadata in error messages.
StorageDriver store context provides target identity
The `context.target` provided to store() is a discriminated union with a `kind` property distinguishing 'workflow' from 'activity'. It contains namespace, id, runId, and type information. This identity information can be included in storage keys to identify which Workflow owns each payload.
StorageDriver name vs type distinction
The `name` property must be unique per driver instance and is stored in claim checks so it cannot change after payloads are stored. The `type` property must be the same across all instances of the same driver implementation regardless of configuration. For example, two S3 drivers named 's3-primary' and 's3-archive' would both report 'aws.s3driver' as their type.
StorageDriver interface properties and methods
A custom StorageDriver implements: name (unique string identifying the driver instance, stored in claim check so retrieval requests route correctly), type (string identifying driver implementation reported in Worker heartbeat), store() method (receives array of payloads, returns one StorageDriverClaim per payload), and retrieve() method (receives claims and returns original payloads).
External Storage payloadSizeThreshold configuration
Example showing how to configure the payload size threshold:
```ts
const dataConverter = {
externalStorage: new ExternalStorage({
drivers: [driver],
payloadSizeThreshold: 0,
}),
};
```
Set `payloadSizeThreshold` to any byte value. A value of 0 externalizes all payloads regardless of size.
External Storage Data Converter configuration
Example showing how to configure ExternalStorage on a Data Converter and pass it to Client and Worker:
```ts
import { Client, Connection } from '@temporalio/client';
import { ExternalStorage } from '@temporalio/common';
import { Worker } from '@temporalio/worker';
async function createClientAndWorker() {
const dataConverter = {
externalStorage: new ExternalStorage({ drivers: [driver] }),
};
const connection = await Connection.connect();
const client = new Client({ connection, dataConverter });
const worker = await Worker.create({
workflowsPath: require.resolve('./workflows'),
taskQueue: 'my-task-queue',
dataConverter,
});
}
```
External Storage GCS driver setup example
Example showing how to create a GcsStorageDriver with Google Cloud SDK:
```ts
import { Storage } from '@google-cloud/storage';
import { GcsStorageDriver } from '@temporalio/external-storage-gcs';
import { GoogleCloudGcsStorageDriverClient } from '@temporalio/external-storage-gcs-google-sdk';
const storage = new Storage();
const driver = new GcsStorageDriver({
client: new GoogleCloudGcsStorageDriverClient(storage),
bucket: 'my-temporal-payloads',
});
```
External Storage S3 driver setup example
Example showing how to create an S3StorageDriver with AWS SDK:
```ts
import { S3Client } from '@aws-sdk/client-s3';
import { S3StorageDriver } from '@temporalio/external-storage-s3';
import { AwsSdkS3StorageDriverClient } from '@temporalio/external-storage-s3-aws-sdk';
const s3Client = new S3Client({ region: 'us-east-2' });
const driver = new S3StorageDriver({
client: new AwsSdkS3StorageDriverClient(s3Client),
bucket: 'my-temporal-payloads',
});
```
External Storage default payload size threshold
By default, payloads of 256 KiB or larger are offloaded to external storage in the TypeScript SDK. Smaller payloads remain inline in Event History.
External Storage feature status
External Storage APIs and configuration may change before General Availability. This is a prerelease feature. Join the #large-payloads Slack channel to provide feedback or ask for help.
Google Cloud SDK credentials for GCS driver
The Google Cloud SDK reads Application Default Credentials when creating a Storage client for the GCS driver.
AWS SDK credentials for S3 driver
The AWS SDK reads environment variables, an IAM role, or your AWS config file for credentials when creating an S3Client.
Custom driver data converter configuration example
Example showing how to create a Data Converter with a custom storage driver:
```ts
export function createDataConverter(rootDir: string = STORAGE_ROOT): DataConverter {
return {
externalStorage: new ExternalStorage({
drivers: [new FileSystemStorageDriver({ rootDir })],
payloadSizeThreshold: PAYLOAD_SIZE_THRESHOLD,
// driverSelector: (context, _payload) =>
// context.target?.type === 'processDocument' ? coldDriver : hotDriver,
}),
};
}
```
StorageDriver retrieve method returns reconstructed Payloads
In the retrieve() method, download the bytes using the claim data, then reconstruct the Payload protobuf message. The Payload Converter handles deserializing the application data after the driver returns the payload.
Data handling with converters and encryption in TypeScript SDK
The TypeScript SDK includes best practices documentation on converters and encryption for data handling.
Default .NET payload converter supported types
The default payload converter in the .NET SDK supports: null, byte[], Google.Protobuf.IMessage instances, anything that System.Text.Json supports, and IRawValue as unconverted raw payloads.
Data Converter architecture has three layers
The Temporal Data Converter architecture consists of three components: payload converters (convert .NET values to/from serialized bytes), payload codecs (convert bytes to bytes, for example for compression or encryption), and failure converters (convert exceptions to/from serialized failures). Payload converters and codecs can be customized independently.
Set DataConverter when creating a .NET client
A custom data converter is set via the DataConverter option when creating a TemporalClient. The DataConverter is passed in the client configuration object, for example: DataConverter = DataConverter.Default with { PayloadConverter = new CustomPayloadConverter() }.
Custom Payload Converter example for camel case
To create a custom payload converter in .NET that converts property names to camel case: class CamelCasePayloadConverter : DefaultPayloadConverter { public CamelCasePayloadConverter() : base(new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.CamelCase }) { } }. Then pass it to the client via DataConverter = DataConverter.Default with { PayloadConverter = new CamelCasePayloadConverter() }.
Data converters namespace in .NET
Data converters in the .NET SDK are located in the Temporalio.Converters namespace.
PreferredSelector for multiple storage drivers example
To register multiple drivers and always select a preferred driver for new payloads, implement StorageDriverSelector interface with SelectDriver() method returning the preferred driver, while registering both drivers to allow retrieval of legacy payloads from the other driver.
Default payload size limit enforced by Temporal Service
The Temporal Service enforces a 2 MB per-payload limit by default. This limit is configurable on self-hosted deployments.
External Storage uses claim check pattern to offload large payloads
When workflows or activities handle data larger than the 2 MB limit, payloads can be offloaded to external storage such as Amazon S3 or Google Cloud Storage, with a small reference token passed through the Event History instead.
Default payload size threshold for external storage in Go SDK
By default, payloads of 256 KiB or larger are offloaded to external storage. This can be adjusted with the PayloadSizeThreshold option, or set to 1 to externalize all payloads regardless of size. A value of 0 is interpreted as the default (256 KiB).
External storage driver maximum payload size
Each storage driver rejects any single payload larger than MaxPayloadSize, which defaults to 50 MiB.
StorageDriver interface methods in Go SDK
A custom storage driver implements the converter.StorageDriver interface with four methods: Name() returns a unique string identifying the driver instance; Type() returns a string identifying the driver implementation (must be same across all instances of same type); Store() receives a slice of payloads and returns one StorageDriverClaim per payload; Retrieve() receives the claims and returns the original payloads.
StorageDriver Name vs Type distinction
Name() must be unique per driver instance and should not change after payloads are stored, as the SDK stores this name in claim checks for routing. Type() identifies the driver implementation and must be the same across all instances of the same driver type. For example, two S3 drivers named 's3-primary' and 's3-archive' would both return 'aws.s3driver' as their type.
Payload marshaling for external storage
In the Store() method, marshal each Payload protobuf message to bytes with proto.Marshal(payload) and write the bytes to the storage system. The application data has already been serialized by the Payload Converter and Payload Codec before it reaches the driver.
Payload retrieval from external storage
In the Retrieve() method, download the bytes using the claim data, then reconstruct the Payload protobuf message with proto.Unmarshal(data, payload). The Payload Converter handles deserializing the application data after the driver returns the payload.
Storage key generation for external storage payloads
The hash forms part of the object key, along with the Namespace, Workflow Id, and Run Id. When one Workflow Run passes the same payload to several Activities, the key is identical each time, so the driver uploads the payload once and later writes reuse that object. A different Run, Workflow, or Namespace produces a different key, so it stores its own copy even when the bytes are identical.