cost-optimization/best-practices

47 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Preventing cost overruns with automatic capacity deprovisioning

Permanently allocating peak capacity for temporary spikes results in unnecessary costs. Manually adjusting capacity quotas is prone to human error, risking either workflow throttling if limits are not raised in time, or runaway costs if operators forget to reduce limits after the workload subsides. Automating deprovisioning with durable Timers guarantees that capacity reverts to its original state after the specified duration.

Publisher TTL deduplication window

At each Continue-As-New, deduplicate entries whose last-seen time is older than the publisher TTL are dropped. The last-seen time is updated on each successful publish (not on each retry attempt), so a publisher that retries through a long partition without success can still age out. A publisher that returns after a longer pause may produce a duplicate. stream.NewContinueAsNewError(...) snapshots with a 15-minute default; tune it by using the explicit recipe and passing your value to GetState(publisherTTL).

Temporal Cloud consumption-based pricing components

Temporal Cloud uses consumption-based pricing with two primary cost components: Actions and Storage.

Build workflows following best practices before optimizing

Build Workflows following best practices first, then optimize based on observed costs. Premature optimization can compromise observability and create operational challenges.

Activity consolidation anti-pattern

Combining Activities before understanding failure modes reduces observability and retry control. Activities should be split based on failure boundaries and retry requirements, not cost optimization alone.

Local Activities limitations and semantics

Local Activities do not provide Worker-level isolation and have different retry behavior compared to regular Activities. Using Local Activities for all operations without understanding these limitations is an anti-pattern.

Continue-As-New requirement for long-running workflows

Long-running Workflows that do not implement Continue-As-New accumulate large Event Histories, increasing storage costs and impacting performance. Workflows running days or weeks or processing thousands of events require Continue-As-New.

High activity retry volume as cost and quality indicator

Excessive Activity retries often indicate underlying issues like timeouts that are too short or Activities that frequently fail. Detecting high Activity retry frequency and addressing root causes reduces costs and improves reliability.

Large payloads in event history anti-pattern

Passing multi-megabyte payloads through Workflows when external storage (S3, blob storage) is more appropriate is an anti-pattern. Use compression or the claim check pattern for large data instead.

Cost distribution in typical workloads

For most workloads, Actions represent the majority of total costs, with storage typically accounting for 10% or less of a monthly bill.

High Actions costs drivers

High Actions costs generally indicate: many Activities per Workflow, frequent Signals/Queries/Updates, long-running Activities with Heartbeats, high Activity retry rates, or extensive Query usage.

High Storage costs drivers

High Storage costs generally indicate: large payloads in Workflow inputs/outputs/Activity results, long retention periods with high Workflow volume, long-running Workflows without Continue-As-New, or Workflows accumulating large Event Histories.

Optimization priority order

Optimization priority should be: 1) Actions optimization (usually provides the largest cost reduction opportunity), 2) Active Storage optimization (relevant for long-running Workflows or large payloads), 3) Retained Storage optimization (relevant for high volume combined with long retention periods).

Establishing baseline metrics before optimization

Before optimizing, establish baseline metrics including: Actions consumption (per Workflow, per day/month, by Namespace), Storage consumption (Active and Retained), monthly costs (total, per Namespace, per Workflow Type), and observability metrics (time to debug, incident detection).

Active vs Retained Storage cost difference

Storage costs are divided into Active Storage (open Workflows) and Retained Storage (closed Workflow History during retention period). Active Storage is significantly more expensive than Retained Storage.

Continue-As-New for active storage reduction

For long-running Workflows with extended sleep/wait periods, calling Continue-As-New before sleeping closes the current execution (moving to cheaper Retained Storage) and starts fresh when work resumes, reducing Active Storage costs.

Compression strategy for payload optimization

Implement a custom Data Converter with compression for moderately large payloads (100KB-1MB) to reduce Active Storage costs.

Claim check pattern for large payloads

For very large payloads or binary data, store data externally (S3 or GCS) and pass references through Workflows.

Default namespace retention period

The default Namespace retention is 30 days, configurable between 1 and 90 days.

Retention period adjustment considerations

When adjusting retention periods: shorter retention reduces costs but limits historical analysis, audit investigation patterns before shortening retention, and ensure compliance requirements are met.

Workflow export cost and use

Temporal Cloud supports exporting Event Histories to external storage for compliance while maintaining shorter retention periods. Workflow export costs one Action per export.

Validation approach for optimizations

Validate optimizations through: 1) Test in non-production to validate functional correctness before production deployment, 2) Monitor comprehensively using the Usage dashboard in the Cloud UI to track impact on Actions and Storage, 3) Progressive rollout by deploying to a small percentage, validate, then expand using Worker Versioning, and 4) Continuous review by re-evaluating optimization effectiveness quarterly.

Success criteria for cost optimization

Cost optimization success criteria include: cost reduced without increasing mean time to repair (MTTR), workflow success rates maintained or improved, and reduced observability does not increase mean time to detect (MTTD) incidents.

Over-optimization at expense of observability anti-pattern

Aggressively optimizing costs without maintaining sufficient visibility for debugging and operational needs is an anti-pattern. Balance cost reduction with your team's observability requirements. For example, merging five Activities into one reduces Actions but you lose per-step visibility, independent retry control, and inability to filter Workflows by failure stage.

Removing heartbeats observability tradeoff

Removing Heartbeats from a long-running data processing Activity saves Actions, but you cannot detect a stuck Worker until the full Activity timeout expires and you lose progress tracking (for example, 'processed 500 of 1,000 records').

Real-Time SLAs and Deadline Management APS impact

Businesses with strict service level agreements often implement active monitoring and escalation in Workflows by setting Timers every [x] minutes to determine if an SLA deadline is approaching. Each Timer/monitoring action affects APS. When thousands of in-flight Workflows actively monitor their own SLAs, the background load becomes significant—consuming substantial APS capacity even when Workflows aren't doing their primary work.

What is an Action in Temporal

An action is any operation that modifies Workflow state or interacts with the Temporal service. Actions include: starting, completing, and resetting Workflows; starting Child Workflows, Schedules, and Timers; starting and retrying Activities; Heartbeating; and Signals, Updates, and Queries. Nearly everything that happens in Temporal—state changes, decision points, interactions—is counted as an action.

APS limit definition and throttling behavior

A Namespace's Actions Per Second (APS) limit controls how many operations can happen per second across all Workflows within that Namespace. When the APS limit is reached, Temporal begins to throttle requests. In Temporal Cloud, the effect of rate limiting is increased latency, not lost work—Workers might take longer to complete Workflows.

Action Multiplier Effect

When a single Workflow starts, it performs multiple actions because a Workflow is not a single atomic operation but a series of events that Temporal orchestrates. Each Activity at the start of the Workflow is an Action, creating a burst of Activities at Workflow startup. Additionally, there are often business reasons to start multiple Workflows at the same time, all contributing to the multiplier effect.

Common APS limit patterns—bursty traffic

Bursty traffic creates APS challenges because Temporal Cloud enforces limits at the per-second level. Common bursty patterns include: calendar-driven spikes (month-end financial close, quarterly reporting, payroll on specific dates, batch jobs at midnight); event-driven surges (product launches, marketing campaigns, flash sales, seasonal events); recovery scenarios (thundering herd when downstream dependency recovers); geographic/business hours concentration; retry storms (many Workflows stuck on failing Activity with short retry delay); and timer storms (many Workflows setting Timers for the exact same time).

Mitigating bursty traffic APS consumption

To reduce bursty APS spikes: implement application-level queuing or rate limiting to smooth predictable spikes; for scheduled batch operations, stagger start times rather than triggering everything at once and implement jitter in high-volume Schedules; implement jitter when starting Workflows, such as with Start Delay; accept rate limiting; or use Provisioned Capacity.

Cascading Workflows and Fan-Out Patterns APS impact

Decomposing complex processes into parent and Child Workflows or using Nexus multiplies APS costs dramatically with depth and fan-out. Each Child Workflow goes through its full action lifecycle (start, tasks, activities, completion), and all actions count toward Namespace APS limits. This pattern appears frequently in batch processing (parent spawning Child for each record), map-reduce patterns (fanning out to process partitions in parallel), and multiple levels of nesting. The challenge compounds when multiple levels of nesting exist.

Mitigating cascading workflows and fan-out APS consumption

To reduce APS impact from cascading workflows and fan-out: evaluate whether Child Workflows are necessary and consider Activities or Workflows in another Namespace via Nexus instead; when using Child Workflows, limit fan-out size by designing a Child Workflow to process work in batches rather than one Child per work item; consider flattening deeply nested hierarchies into shallower structures.

Human-in-the-Loop Processes at Scale APS impact

Workflows incorporating human decision-making (approvals, reviews, manual data entry, quality checks) tend to be long-running and interaction-intensive, creating sustained APS load. These Workflows can involve Queries from UIs to display current state and pending tasks. At small scale this is manageable, but running thousands simultaneously—such as content moderation queues, loan approval systems, or support ticket systems—creates cumulative APS load from all long-running Workflows.

Mitigating human-in-the-loop process APS consumption

To reduce APS impact from human-in-the-loop processes: avoid polling patterns where UIs constantly query Workflow state. Instead, push state changes to a database that UIs can read.

Mitigating real-time SLA monitoring APS consumption

To reduce APS impact from real-time SLA monitoring: use longer monitoring intervals where possible (e.g., check SLAs every 30 minutes rather than every 1 minute); where possible, consolidate Timers (rather than 10 Timers checking 10 tasks, have 1 Timer then check those 10 tasks); where possible, have an external system signal your Workflow rather than using short-lived Timers to poll; for retries, use exponential backoff with reasonable initial intervals.

Many Small Activities APS impact

Processing work through many small Activities consumes significantly more APS than batching. For example, a Workflow spawning 1,000 separate Activities (one per record) consumes more APS than spawning 10 Activities each processing 100 records in a batch. This pattern appears everywhere: processing individual transactions versus batches, sending individual notifications versus bulk operations, or making separate API calls versus batch endpoints. Each separate Activity adds Action overhead.

Mitigating many small activities APS consumption

To reduce APS impact from many small activities: consider if you can combine multiple external calls within a single Activity; if processing a large amount of data, process it in chunks.

Multiple use cases in one Namespace APS impact

When multiple Temporal use cases run in the same Namespace, the APS limit applies across all use cases. Multiple use cases with multiple traffic patterns in the same Namespace can exhaust the limit quickly because the limit is set per Namespace.

Mitigating multiple use cases APS consumption

To manage APS across multiple use cases: plan for a set of Namespaces (one per environment) per use case. This gives each use case its own APS envelope and reduces the blast radius when one workload spikes or misbehaves.

Temporal Cloud Capacity Modes

Temporal Cloud offers two Capacity Modes: On-Demand mode (default) where your Namespace automatically scales based on your trailing 7-day usage, working well for steady, predictable workloads; and Provisioned mode where you reserve capacity by adding Temporal Resource Units (TRUs), giving guaranteed headroom for traffic spikes.

When to use Provisioned Capacity mode

Use Provisioned capacity when the on-demand model can't respond quickly enough for: planned spikes (promotions, holiday traffic, product launches—pre-provision TRUs before the event); unplanned spikes (sudden traffic surges, viral events—react instantly via UI/CLI/API when throttling is seen); load testing (provision TRUs for the test, deprovision after); batch jobs (automate TRU scaling via API around job schedules); migrations (bridge with TRUs for approximately 7 days while the on-demand envelope catches up).

On-demand APS limit reset after Provisioned mode

When switching back to on-demand mode, your APS limit resets to the running average from the last 7 days. If Temporal Support has set a custom limit for your Namespace, that limit is preserved across the switch. Plan for this if your workload is sensitive to the transition.

Provisioned Capacity cost optimization

To minimize Provisioned Capacity costs: provision only when you need extra capacity; deprovision promptly after spikes end; for predictable patterns, automate scaling to minimize time in provisioned mode.

Automation best practices for TRU scaling

Build your own TRU automation rather than relying on generic auto-scaling: use the Cloud Ops API, Terraform Provider, or tcld CLI to programmatically scale capacity based on your application's signals; set utilization thresholds (for example, scale up when hitting 70-80% of limit, scale down after sustained low usage); schedule capacity changes using Temporal Schedules or Workflows to increase TRUs before known events; react to leading indicators (if your application has upstream signals like incoming order queue depth or marketing campaign start, use those to trigger capacity changes proactively).

Monitoring APS consumption and throttling

To monitor if approaching or exceeding APS limits in Temporal Cloud: track the metric `temporal_cloud_v1_total_action_throttled_count` to detect throttling events; set alerts at 70-80% utilization to give time to provision TRUs before hitting limits; use OpenMetrics for real-time visibility into APS consumption integrated with your observability stack; analyze historical patterns to understand traffic patterns and decide between reactive TRU provisioning and proactive automation.

Questions to ask when designing Workflows for APS efficiency

When designing Temporal Workflows with APS limits in mind, ask: How many actions will a single execution of this Workflow consume? How many Workflows will typically be running at the same time? What happens to APS consumption when the number of Actions multiplied by number of active Workflows scales to 100x current volume? Are there natural opportunities to combine operations—combine activities or process chunks of data together? Am I polling when I could be using Signals? Does this Workflow need to run continuously, or can it be event-driven?

Give your agent this brain