SHA-256 hash verification in external storage drivers
Both the S3 and GCS drivers upload and download payloads concurrently, and verify a SHA-256 hash of the contents on retrieval.
Temporal · Develop · all subjects
data-conversion
179 notes in this subject, read out of this brain and free to use. This is page 3 of 3.
Configuring external storage in Go SDK client
Pass an ExternalStorage struct with your driver in the Client options using client.Dial(client.Options{ExternalStorage: converter.ExternalStorage{Drivers: []converter.StorageDriver{driver}}}).
Worker inherits external storage from client configuration
A Worker inherits external storage configuration from the Client it is created with. When Workers run in their own process, repeat the external storage setup there. All Workflows and Activities running on the Worker use the storage driver automatically without changes to business logic.
Multiple storage drivers with DriverSelector
When registering multiple drivers, provide a DriverSelector that implements the StorageDriverSelector interface to choose which driver stores each payload. Any driver in the list not selected for storing is still available for retrieval. Return nil from the selector to keep a specific payload inline in Event History.
Multiple storage drivers use cases
Multiple drivers are useful for driver migration (Worker retrieves payloads created by clients using different drivers) and multi-cloud storage (route payloads to different backends based on cloud environment, e.g., S3 for AWS, GCS for Google Cloud).
Default driver names for S3 and GCS in Go SDK
The S3 driver defaults to name 'aws.s3driver' and the GCS driver defaults to 'gcp.gcsdriver'. You can register one of each without extra configuration. Registering two drivers of the same type requires setting the DriverName option on at least one of them.
Go SDK S3 driver setup with AWS SDK v2
To set up the S3 driver: load AWS config with config.LoadDefaultConfig(), create an S3 client with s3.NewFromConfig(cfg), wrap it with awssdkv2.NewClient(), then pass to s3driver.NewDriver() with Options containing the client and bucket.
Go SDK GCS driver setup
To set up the GCS driver: create a GCS client with storage.NewClient(context.Background()), wrap it with gcssdk.NewClient(), then pass to gcsdriver.NewDriver() with Options containing the client and bucket.
Dynamic bucket selection in external storage driver
To route payloads to different buckets at runtime, pass a BucketFunc as Bucket instead of using StaticBucket. The function receives the store context and the payload, and returns a bucket name.
StorageDriverStoreContext provides identity information
The ctx.Target in Store() provides identity information (namespace, Workflow ID) depending on the operation. Use a type switch on StorageDriverWorkflowInfo and StorageDriverActivityInfo to access concrete values.
StorageDriverWorkflowInfo vs StorageDriverActivityInfo
StorageDriverActivityInfo is used for standalone (non-workflow-bound) activities. Activities started by a workflow use StorageDriverWorkflowInfo.
Custom storage driver example with local disk
A custom driver implementing converter.StorageDriver with local disk backing store should marshal payloads to bytes, store them in directories organized by namespace and workflow/activity ID, generate unique keys, and return StorageDriverClaim with ClaimData containing the file path. For retrieval, read the file and unmarshal the payload.
Lifecycle management requirements for external storage
A bucket must have read and write access. Ensure lifecycle management allows payloads to remain available for the entire lifetime of the Workflow. For multi-region durability, see Durable External Storage pattern.
Bucket credentials location for external storage
Bucket credentials are needed on both the Temporal Client and Workers, since each reaches the bucket directly.
External storage permissions requirements
Permission to write objects is required on processes that store payloads, and to read objects on processes that retrieve them. Storing also requires read permission, because the drivers check whether an object already exists before uploading.
S3 driver diagnostic metadata in error messages
The S3 driver includes diagnostic metadata, such as the AWS region, in error messages to help troubleshoot storage failures.
Multi-region durability with S3 and MRAP
To make S3-backed External Storage tolerant of regional failures, configure AWS with Cross-Region Replication (CRR) and an S3 Multi-Region Access Point (MRAP), then point the driver at the MRAP ARN instead of a bucket name. The AWS SDK for Go v2 automatically uses SigV4A signing when the bucket value is an MRAP ARN, requiring no additional client configuration.
PayloadSizeThreshold configuration example
Set PayloadSizeThreshold to 1 to externalize all payloads regardless of size: client.Dial(client.Options{ExternalStorage: converter.ExternalStorage{Drivers: []converter.StorageDriver{driver}, PayloadSizeThreshold: 1}}).
Payload size compared against threshold includes metadata
The size compared against the PayloadSizeThreshold is that of the serialized Payload, which includes its metadata, not just the application data.
Default supported types in Python Payload Converter
The default Data Converter supports: None, bytes, google.protobuf.message.Message (as JSON when encoding, but can decode binary proto from other languages), and anything JSON-serializable including types that json.dump supports natively, dataclasses, iterables like set, IntEnum and StrEnum based enumerates, and UUID.
DefaultPayloadConverter structure
The default Data Converter is a CompositePayloadConverter that tries each encoding converter in order until one handles the value. Upon serialization, each EncodingPayloadConverter is used in order until one succeeds.
How to use Pydantic Data Converter in Python
To use Pydantic model instances, install Pydantic and set the Pydantic Data Converter when creating Client instances by passing data_converter=pydantic_data_converter to the Client constructor. Import it from temporalio.contrib.pydantic.
Pydantic Data Converter supported types
The Pydantic Data Converter supports conversion of all types supported by Pydantic to and from JSON, including everything that json.dumps() supports by default, standard library types like dataclasses, datetime module types, sets, UUID, and custom types composed of these with any degree of nesting, such as a list of Pydantic models with datetime fields.
Pydantic v1 not supported
Pydantic v1 is not supported by the Pydantic Data Converter. If unable to upgrade from Pydantic v1, see https://github.com/temporalio/samples-python/tree/main/pydantic_converter/v1 for limited v1 support.
datetime types only work with Pydantic Data Converter
datetime.date, datetime.time, and datetime.datetime can only be used with the Pydantic Data Converter, not with the default converter.
Use single dataclass or Pydantic model parameter for workflow definitions
Workflows, Updates, Signals, and Queries can be defined with multiple input parameters, but users are strongly encouraged to use a single dataclass or Pydantic model parameter so that fields with defaults can be easily added without breaking compatibility. Similar advice applies to return values.
Classes with generics may not resolve properly
Classes with generics may not have the generics properly resolved. The current implementation does not have generic type resolution. Users should use concrete types instead.
Create custom EncodingPayloadConverter for custom types
To handle custom data types, create a new EncodingPayloadConverter by subclassing it and implementing the encoding property returning a string encoding name, to_payload method for serialization, and from_payload method for deserialization. Then wrap it in a CompositePayloadConverter with the default converters.
Custom IPv4Address Payload Converter example
Example of custom EncodingPayloadConverter for IPv4Address types: ```python class IPv4AddressEncodingPayloadConverter(EncodingPayloadConverter): @property def encoding(self) -> str: return "text/ipv4-address" def to_payload(self, value: Any) -> Optional[Payload]: if isinstance(value, ipaddress.IPv4Address): return Payload( metadata={"encoding": self.encoding.encode()}, data=str(value).encode(), ) else: return None def from_payload(self, payload: Payload, type_hint: Optional[Type] = None) -> Any: assert not type_hint or type_hint is ipaddress.IPv4Address return ipaddress.IPv4Address(payload.data.decode()) class IPv4AddressPayloadConverter(CompositePayloadConverter): def __init__(self) -> None: super().__init__( IPv4AddressEncodingPayloadConverter(), *DefaultPayloadConverter.default_encoding_payload_converters, ) my_data_converter = dataclasses.replace( DataConverter.default, payload_converter_className=IPv4AddressPayloadConverter, ) ```
Customize JSON converter for custom types in collections
If custom types need to work in lists, unions, and other collections, customize the existing JSON converter instead of adding a new encoding converter. The JSON converter is the last in the list, so it handles any otherwise unknown type. Customize serialization with a custom json.JSONEncoder and deserialization with a custom JSONTypeConverter.
Custom JSON encoder and type converter example for IPv4Address
Example of customizing JSON converter for IPv4Address in collections: ```python class IPv4AddressJSONEncoder(AdvancedJSONEncoder): def default(self, o: Any) -> Any: if isinstance(o, ipaddress.IPv4Address): return str(o) return super().default(o) class IPv4AddressJSONTypeConverter(JSONTypeConverter): def to_typed_value( self, hint: Type, value: Any ) -> Union[Optional[Any], _JSONTypeConverterUnhandled]: if issubclass(hint, ipaddress.IPv4Address): return ipaddress.IPv4Address(value) return JSONTypeConverter.Unhandled class IPv4AddressPayloadConverter(CompositePayloadConverter): def __init__(self) -> None: json_converter = JSONPlainPayloadConverter( encoder=IPv4AddressJSONEncoder, custom_type_converters=[IPv4AddressJSONTypeConverter()], ) super().__init__( *[ c if not isinstance(c, JSONPlainPayloadConverter) else json_converter for c in DefaultPayloadConverter.default_encoding_payload_converters ] ) my_data_converter = dataclasses.replace( DataConverter.default, payload_converter_className=IPv4AddressPayloadConverter, ) ``` This allows IPv4Address to be used in type hints including collections and optionals.
Three layers of Data Converter in Python
The three layers of Data Converter are: the Payload Converter layer (handles serialization/deserialization to Payload objects), the Encoding Converter layer (handles specific encodings), and the JSON converter layer (handles JSON types and custom type converters). The CompositePayloadConverter tries each EncodingPayloadConverter in order until one succeeds.
Temporal converter architecture layers
Temporal's converter architecture consists of three layers: payload converters (convert Ruby values to/from serialized bytes), payload codecs (convert bytes to bytes for compression or encryption), and failure converters (convert exceptions to/from serialized failures). These three components together make up a data converter.
Data Converter keyword argument in Ruby client
A custom data converter can be set via the `data_converter` keyword argument when creating a client in the Ruby SDK.
Default Ruby payload converter supported types
The default payload converter in Ruby supports: nil, bytes (String with Encoding::ASCII_8BIT encoding), Google::Protobuf::MessageExts instances, and uses the JSON module for everything else. Normal Ruby objects are serialized with JSON.generate and deserialized with JSON.parse using create_additions: true by default.
Ruby object deserialization to hash with string keys
When Ruby objects are deserialized, they will often appear as a hash. Hashes that are passed in with symbol keys end up with string keys when deserialized because JSON serialization converts symbol keys to strings.
Encoding payload converters sequence in Ruby
The default payload converter is a collection of encoding payload converters. On serialize, each encoding converter is tried in order until one accepts, with a default fallback to the JSON converter. The encoding converter sets an `encoding` metadata value which is used to determine which converter to use on deserialize.
Payload Converter ordering in Data Converter
The order in which encoding Payload Converters are applied depends on the order given to the Data Converter. When the Data Converter receives a value for conversion, it passes through each Payload Converter in sequence until the converter that handles the data type performs the conversion.
Payload Converters and Codecs independence
Payload Converters can be customized independently of a Payload Codec. The two components work separately in the converter architecture.
Data Converter location in Ruby SDK
Data converters in the Ruby SDK are located in the `Temporalio::Converters` module.
ActiveModel JSON support for Temporal data conversion
By default, ActiveModel objects do not natively support the JSON module. A mixin can be created to add JSON support by including ActiveModel::Serializers::JSON and implementing as_json and json_create methods. The as_json method should merge the fully qualified class name as the JSON create_id key, and json_create should reconstruct the object from this data with symbolized keys.
ActiveModel JSON support mixin implementation
Here is an example mixin for ActiveModel JSON support: ```ruby module ActiveModelJSONSupport extend ActiveSupport::Concern include ActiveModel::Serializers::JSON included do def as_json(*) super.merge(::JSON.create_id => self.class.name) end def to_json(*args) as_json.to_json(*args) end def self.json_create(object) object = object.dup object.delete(::JSON.create_id) new(**object.symbolize_keys) end end end ``` When included on an ActiveModel class, serialization uses to_json which calls as_json to include the fully qualified class name, and deserialization uses this key to call json_create on the correct class.
JSON Additions cross-SDK compatibility
While JSON Additions are supported in the Ruby SDK's default payload converter, they are not cross-SDK-language compatible because JSON Additions are a Ruby-specific construct.
Payload definition and role in Temporal
A Payload is a binary form with metadata that Temporal uses to transport data. Payload Converters serialize application objects into a Payload and deserialize them back.
Default Data Converter supported types in TypeScript
The default Data Converter in TypeScript SDK supports: undefined, null, Uint8Array, and JSON-serializable values. For protobufs, use DefaultPayloadConverterWithProtobufs.
PayloadCodec interface
The PayloadCodec interface has two methods: encode(payloads: Payload[]): Promise<Payload[]> to encode payloads for sending over the wire, and decode(payloads: Payload[]): Promise<Payload[]> to decode payloads received from the wire.
Payload Converter ordering during serialization
During serialization, the Data Converter tries Payload Converters in sequence until one returns a non-null Payload. The order of Payload Converters is important because the first converter that can handle a data type performs the conversion.
Three layers of Data Converter in TypeScript
The default Data Converter is implemented as a Composite Data Converter with three layers applied in order: UndefinedPayloadConverter (handles null and undefined), BinaryPayloadConverter (handles byte arrays), and JsonPayloadConverter (handles JSON-serializable values).
PayloadConverter interface methods
The PayloadConverter interface requires two methods: toPayload<T>(value: T): Payload to convert a value to a Payload for serialization, and fromPayload<T>(payload: Payload): T to convert a Payload back to a value for deserialization.
Configure custom Data Converter for Worker in TypeScript
To configure a custom Data Converter for a Worker, pass it in WorkerOptions with: dataConverter: { payloadConverterPath: require.resolve('./payload-converter') }.
Configure custom Data Converter for Client in TypeScript
To configure a custom Data Converter for a Client, pass it in Client constructor options with: dataConverter: { payloadConverterPath: require.resolve('./payload-converter') }.
EJSON custom Payload Converter example
The EJSON custom PayloadConverter implements PayloadConverterWithEncoding. It sets encodingType to 'json/plain', implements toPayload by calling EJSON.stringify and encoding the result with metadata, and implements fromPayload by decoding and calling EJSON.parse.
Protobuf setup with protobufjs in TypeScript
To use protobufs with protobufjs: use runtime-loaded messages (not generated classes) and MessageClass.create (not new MessageClass()). Generate json-module.js with: pbjs -t json-module --workflow-id commonjs -o protos/json-module.js protos/*.proto. Generate root.d.ts with: pbjs -t static-module protos/*.proto | pbts -o protos/root.d.ts -.
Patch protobuf root in TypeScript
Patch json-module.js by creating a root.js file that calls patchProtobufRoot on the unpatchedRoot from json-module: const { patchProtobufRoot } = require('@temporalio/common/lib/protobufs'); module.exports = patchProtobufRoot(unpatchedRoot);
DefaultPayloadConverterWithProtobufs configuration
To create a DefaultPayloadConverterWithProtobufs, instantiate it with: new DefaultPayloadConverterWithProtobufs({ protobufRoot: root }) where root is the patched protobuf root.
ProtobufBinaryPayloadConverter for binary encoding
Use ProtobufBinaryPayloadConverter to encode protobufs as binary instead of proto3 JSON, which saves space but cannot be viewed in the Web UI. Instantiate with: new ProtobufBinaryPayloadConverter(root).
Composite Data Converter with protobufs example
To support binary-encoded Protobufs alongside default types, create a CompositePayloadConverter with: new CompositePayloadConverter(new UndefinedPayloadConverter(), new BinaryPayloadConverter(), new ProtobufBinaryPayloadConverter(root), new JsonPayloadConverter()).
Custom Payload Converter for non-JSON-serializable types
Create a custom Payload Converter to handle data types that are not natively JSON-serializable, such as BigInt, Date, or binary data. Implement the PayloadConverter interface with toPayload and fromPayload methods.
Default Data Converter encoding order in Nexus
Each Temporal SDK includes a default Data Converter that encodes payloads in the following order: Null, Byte array, and JSON. In polyglot environments (multiple languages/SDKs), JSON is commonly used. The TypeScript SDK does not support Protobuf JSON encoding by default; use ProtobufJsonPayloadConverter instead for Protobuf payloads.