authentication/mtls

28 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Passing mTLS certificates inline to Temporal Cloud CLI

To pass mTLS certificates inline to the Temporal CLI: `temporal workflow list --address <namespace>.<account>.tmprl.cloud:7233 --namespace <namespace>.<account> --tls-cert-path /path/to/client.pem --tls-key-path /path/to/client.key`.

TEMPORAL_TLS_CLIENT_KEY_PATH and TEMPORAL_TLS_CLIENT_KEY_DATA environment variables

TEMPORAL_TLS_CLIENT_KEY_PATH specifies the filesystem path to the client's private TLS key and cannot be combined with TEMPORAL_TLS_CLIENT_KEY_DATA. TEMPORAL_TLS_CLIENT_KEY_DATA provides raw PEM data for the client's private TLS key and cannot be combined with TEMPORAL_TLS_CLIENT_KEY_PATH. TOML keys: profile.<name>.tls.client_key_path and profile.<name>.tls.client_key_data respectively. CLI flags: --tls-key-path and --tls-key-data respectively. Read by: every client.

TEMPORAL_TLS_SERVER_CA_CERT_PATH and TEMPORAL_TLS_SERVER_CA_CERT_DATA environment variables

TEMPORAL_TLS_SERVER_CA_CERT_PATH specifies the filesystem path to the Certificate Authority certificate used to verify the server and cannot be combined with TEMPORAL_TLS_SERVER_CA_CERT_DATA. TEMPORAL_TLS_SERVER_CA_CERT_DATA provides raw PEM data for the Certificate Authority certificate used to verify the server and cannot be combined with TEMPORAL_TLS_SERVER_CA_CERT_PATH. TOML keys: profile.<name>.tls.server_ca_cert_path and profile.<name>.tls.server_ca_cert_data respectively. CLI flags: --tls-ca-path and --tls-ca-data respectively. Read by: every client.

TEMPORAL_TLS_SERVER_NAME environment variable

TEMPORAL_TLS_SERVER_NAME overrides the server name used for Server Name Indication (SNI) in the TLS handshake. TOML key: profile.<name>.tls.server_name. CLI flag: --tls-server-name. Read by: every client.

TEMPORAL_TLS_DISABLE_HOST_VERIFICATION environment variable

TEMPORAL_TLS_DISABLE_HOST_VERIFICATION disables server hostname verification. Use with caution. Not every SDK applies this setting. TOML key: profile.<name>.tls.disable_host_verification. CLI flag: --tls-disable-host-verification. Read by: every client.

TEMPORAL_TLS environment variable

TEMPORAL_TLS is set to 'true' to enable TLS or 'false' to disable it. The TOML key inverts the value: disabled = true turns TLS off. TOML key: profile.<name>.tls.disabled. CLI flag: --tls. Read by: every client.

tls configuration for Web UI Server

TLS configuration for the Temporal Server includes the following fields: caFile (path to CA certificate file), certFile (path to certificate file), keyFile (path to key file), caData (CA certificate data), certData (certificate data), keyData (key data), enableHostVerification (boolean, defaults to true, enables verification of the server hostname), and serverName (name of the TLS server).

Check certificates match between Frontend and internode after enabling mTLS

After enabling mTLS, check the health of the Temporal Service with temporal operator cluster health command. Server names and certificates must match between Frontend and internode. Add any missing environment variables to the configuration files and correct any incorrect values.

Troubleshoot persistent connection errors

If the 'failed reaching server: last connection error' persists after updating certificates, verify that the Client is using the correct TLS certification and that Client requests reach the server after roles are fully initialized.

Failed reaching server error cause: expired TLS certificates

The error message 'Failed reaching server: last connection error' often results from an expired TLS certificate or occurs during Server startup when Client requests reach the Server before roles are fully initialized.

Verify TLS certificate expiration with tcld

Use the command 'tcld namespace accepted-client-ca list --namespace <namespace_id>.<account_id> | jq -r '.[0].notAfter'' to list the expiration date of the TLS certification. If the returned date is in the past, the certificate has expired.

Verify TLS certificate expiration with OpenSSL

Run the command 'openssl s_client -connect <namespace_grpc_endpoint> -showcerts -cert ~/certs/path.pem -key ~/certs/path.key -tls1_2' to verify the TLS connection using existing certificate management infrastructure.

Verify TLS certificate expiration with Temporal CLI for self-signed certificates

Run the command 'temporal namespace describe --namespace <namespace_id>.<account_id> --address <namespace_grpc_endpoint> --tls-cert-path <path-to-mTLS-pem-file> --tls-key-path <path-to-mTLS-key-file>' to verify expiration for self-signed certificates. The Namespace gRPC endpoint is available on the details page for your Temporal Cloud Namespace.

Renew TLS certificate process

If the certificate has expired or is about to expire, contact the certificate authority (CA) that issued the certificate and request a renewal. For existing certificate management infrastructure, contact the administrator of the infrastructure. For self-signed certificates, generate a new certificate using OpenSSL, step CLI, or similar tools.

Update renewed CA certificate in server configuration

After renewing the TLS certificate, update the new CA certificate in the Temporal Cloud server configuration using either the Temporal Cloud UI or the tcld command-line tool. After updating, retry the connection.

Certificate Filters for shared CA access control

Certificate Filters are an additional way of validating using the client certificate presented during client authentication. When using the same CA for dev and prod environments with a common name convention matching the namespace, Certificate Filters can prevent access to production.

SDK support for Worker mTLS certificate rotation without restart

Per-SDK code for rotating a Worker's mTLS client certificate without a restart is documented on each SDK's Temporal Client page in the 'Connect to Temporal Cloud' section. Go, Java, Python, .NET, Ruby, and TypeScript are all supported. PHP and Rust are not yet supported.

Let's Encrypt not recommended for mTLS

Using Let's Encrypt for mutual TLS implementations is not recommended because it is designed primarily for public-facing services and lacks support for internal certificate requirements.

Valid certificate management options for mTLS

Several valid options exist for managing certificates in mTLS implementations. Vendor solutions such as AWS Private CA, Sectigo, Microsoft Certification Authority, or DigiCert offer robust integration and lifecycle features. Alternatively, self-signed certificates are valid and commonly used even in production environments. Tools like OpenSSL, CFSSL, or step CLI can help generate and manage certificates effectively.

Certificate common name convention for multi-environment access control

A convention for multi-environment setups is to give certificates a common name that matches the namespace. When using the same CA for dev and prod environments, this allows leveraging Certificate Filters to prevent access to production environments.

mTLS certificate authentication and RBAC

Unlike API keys tied to users or service accounts, mTLS certificate authentication is not tied to Temporal Cloud RBAC identities. Namespace access is based on CA trust, with optional Certificate Filters to narrow access by Common Name.

Enable mutual TLS (mTLS) for Temporal Cloud application authentication

Temporal Cloud secures its gRPC endpoint per Namespace via mutual TLS. You provide a Certificate Authority (CA) certificate for your Namespace, and all your Temporal clients and workers must present client certificates signed by that CA. This ensures only systems holding a valid certificate issued by your trusted CA can connect.

Proactively manage and rotate mTLS certificates

Track the expiration dates of your client and Certificate Authority certificates. Temporal Cloud trusts the uploaded CA; if it expires, all client authorizations will fail. Establish and automate a certificate rotation schedule—for example, rotate client certificates quarterly and CA certificates annually, well before expiry. Temporal supports uploading a new CA certificate alongside the old one to allow seamless rollover. Always test new certificates in a staging environment if possible.

Mutual TLS certificate requirements for internode communication

When client authentication is enabled, the internode.server certificate is used as the client certificate among services. Requirements: the internode.server certificate must be specified on all roles even for frontend-only configuration; internode server certificates must be minted with either no Extended Key Usages or both ServerAuth and ClientAuth EKUs; if Certificate Authorities are untrusted, the internode server CA must be specified in internode.server.clientCaFiles, internode.client.rootCaFiles, and frontend.server.clientCaFiles.

TLS internode and frontend subsections

The tls section controls SSL/TLS settings for network communication and contains two subsections: internode (governs internal service communication among roles) and frontend (governs SDK client communication to the Frontend Service role). Each subsection contains a server section and a client section.

TLS server configuration parameters

TLS server sections contain: certFile (path to PEM-encoded public key certificate), keyFile (path to PEM-encoded private key), requireClientAuth (boolean, requires clients to authenticate with a certificate for mutual TLS), and clientCaFiles (list of paths to PEM-encoded Certificate Authority public keys for trusting client authentication, ignored if requireClientAuth is not enabled).

TLS client configuration parameters

TLS client sections contain: serverName (expected DNS SubjectName in the presented server certificate, needed since Temporal uses IP to IP communication), and rootCaFiles (paths to Root CA files when the client's host does not trust the Root CA used by the server).

WithTLSConfigFactory server option

WithTLSConfigFactory overrides the default TLS configuration provider. TLSConfigProvider is defined in the go.temporal.io/server/common/rpc/encryption package.

Give your agent this brain