testing-and-debugging

164 notes in this subject, read out of this brain and free to use. This is page 3 of 3.

Replay testing during development

During development, replay testing provides early feedback on whether changes are compatible. Integration tests can run Workflows against the Temporal Test Server to produce histories, which can be checked in and used for replay tests to verify no breaking changes were made.

Deployment-time replay test approach

A safe deployment can be broken into two phases: a verification phase where a replay test runs, followed by actual deployment of new Worker code. This can be accomplished by wrapping the Worker application with code that chooses to run in verification mode or production mode. It is recommended not to deploy Workers side-by-side with other application code.

Encrypted payloads may prevent fetching histories

If Workflows use encrypted Payloads (typical in production), fetched histories may not be decryptable for use in replay tests. If Workflows contain PII that is encrypted, this information should be scrubbed for testing purposes, or fetched histories should not be used.

Bimodal Worker deployment strategy

The most straightforward way to deploy a bimodal Worker is to deploy one instance in verify mode at the beginning of the deployment process. Once verification passes, proceed to deploy the rest of the new workers in run mode.

Python Worker entry point for verify or run mode

A Python Worker can be implemented with a single entry point that accepts 'verify' or 'run' mode arguments. In verify mode, it runs replay tests against historical Workflow executions. In run mode, it creates a Worker instance with the task queue, workflows, and activities, then calls await worker.run() to process Workflows from the task queue.

Replay testing during pre-deployment validation

During pre-deployment validation, replay testing can be performed in a more representative environment by fetching histories from a live Temporal environment (production or pre-production) and using them in replay tests.

Node.js built-in modules not allowed in Workflow code

Importing Node.js built-in modules or calling non-deterministic operations in Workflow code is invalid. For example, reading from the filesystem using fs.readFileSync('config.json', 'utf8') will fail because reading from the filesystem is a non-deterministic operation: the file may change from the time of the original Workflow Execution to when the Workflow is replayed. Move this code to an Activity instead. You'll typically see errors in the form 'Module not found: Error: Can't resolve 'fs' in '/path/to/src''. The solution is to move the non-deterministic code to an Activity.

Workflow/Activity registration errors and solutions

If Workflows or Activities are not imported or spelled correctly, you may see errors like 'ApplicationFailure: 'MyFunction' is not a function' or 'Workflow did not register a handler for MyQuery'. Double-check that your Workers are registering the right Workflow and Activity Definitions (function names) on the right Task Queues. In monorepos, node_modules may be in a different location than expected, resulting in errors like 'Module not found: Error: Can't resolve '@temporalio/workflow/lib/worker-interface.js''. When passing a workflowsPath, the Webpack config expects to find node_modules in the same or a parent/ancestor directory. If custom bundling Workflows, you may get 'ReferenceError: exports is not defined' errors. Temporal Workflow Bundles need to export a set of methods that fit the compiled worker-interface.ts from @temporalio/workflow as an entry point. The SDK offers a bundleWorkflowCode method to assist with this.

Stale Workflows cause confusion during development

If old Workflows have the same name and are on the same Task Queue, Temporal will try to continue executing them on your new code. This can produce confusing errors because Temporal is trying to execute old Workflow code that no longer exists in your codebase, or your new Client code is expecting Temporal to execute old Workflow/Activity code it doesn't yet know about. You can terminate old stale Workflows using Temporal Web or the Temporal CLI.

Two locations to check for errors in TypeScript SDK

Workflow errors are reflected in Temporal Web. Worker errors and logs are reflected in the terminal. If something is not behaving as expected, check both locations for helpful error messages.

Debugging tools for production environments

In production environments, you can debug Workflows using the Web UI, Temporal CLI, Replay, Tracing, and Logging. Worker performance can be debugged and tuned with metrics and the Worker performance guide. Server performance can be debugged with Cloud metrics or self-hosted Server metrics.

ESbuild configuration to keep function names

To prevent ESbuild from stripping Workflow function names, use the keepNames option: require('esbuild').buildSync({ entryPoints: ['app.js'], minify: true, keepNames: true, outfile: 'out.js', });

gRPC context deadline exceeded errors

The opaque 'context deadline exceeded' error comes from gRPC with code 4 and DEADLINE_EXCEEDED status. Several conditions can cause this error, including network hiccups, timeouts that are too short, and an overloaded server. Querying a Workflow Execution whose query handler causes an error can result in the query call timing out. Troubleshooting actions include: verify the connection from Worker to Temporal Server is working and doesn't have unusually high latency, check server metrics to ensure it's not overloaded, and if a query is timing out, check Worker logs to see if they are having issues handling the query.

Resetting Workflows to deal with logical bugs

You can 'rewind time' using the Temporal CLI, resetting Event History to some previous point in time. You can reset specific Workflows by ID or reset all Workflows by binary checksum identifier using the Temporal CLI. For programmatic resets, the TypeScript SDK does not have high-level APIs for this, but you can make raw gRPC calls to resetWorkflowExecution. Resetting should only be used to deal with serious logical bugs in your code and is not for handling transient failures like a downstream service being unreachable. It should not be used in the course of normal application flows.

TransportError in production connection

If you are trying to connect to Temporal Server in production and getting 'TransportError: transport error', it is a sign that something is wrong with your Cert/Key pair. Log it out and make sure it is an exact match with what is expected. Often, the issue can be whitespace when injecting from your production secrets management environment.

Webpack errors during bundling

The TypeScript SDK's Worker bundles Workflows based on workflowsPath with Webpack and runs them inside v8 isolates. If Webpack fails to create the bundle, the SDK will throw an error and emit webpack logs using the SDK's logger. If you do not see Webpack output in your terminal, make sure that you have not disabled SDK logging.

Webpack TerserPlugin configuration to keep function names

To prevent Webpack from stripping Workflow function names, configure webpack.config.js as follows: module.exports = { optimization: { minimize: true, minimizer: [ new TerserPlugin({ terserOptions: { keep_fnames: true, // don't strip function names in production }, }), ], }, };

Webpack function name stripping in production

Webpack can change the Workflow's function name to something shorter during bundling. Temporal relies on Workflow function names to set the 'Workflow Type' in the Service Web UI. When names are stripped, you may see errors like 'Error: 3 INVALID_ARGUMENT: WorkflowType is not set on request.' or see shorter names in the Web UI. To prevent the build process from shortening Workflow function names, modify the webpack.config.js to set keep_fnames to true in the TerserPlugin configuration.

Cannot directly import and call Activities from Workflow code

Importing and calling Activities directly from Workflow code is invalid. Activity implementations should not be directly referenced by Workflow code because Activities are used by Workflows to make network calls and read from the filesystem, operations which are non-deterministic by nature because they rely on external state. Temporal records Activity results in the Event history and in case your Workflow is replayed, completed Activities will not be rerun, instead their recorded result will be delivered to the Workflow. Instead, use proxyActivities and import only the Activity types: import type * as activities from './activities'; const { makeHTTPRequest } = proxyActivities<typeof activities>();

Debugging tools for development environments

In development environments, you can debug Workflows using normal development tools like logging and a debugger, in addition to the Web UI and Temporal CLI.

Testing best practices available in TypeScript SDK

Testing best practices documentation is available for the TypeScript SDK under the Testing section.

Debugging best practices available in TypeScript SDK

Debugging best practices documentation is available for the TypeScript SDK.

Testing recommendation: integration tests preferred

The majority of tests should be written as integration tests rather than end-to-end or unit tests.

Test server for time skipping

The test server supports skipping time and should be used for both end-to-end and integration tests with Workers. Time is a global property of a TestWorkflowEnvironment instance, so skipping time applies to all currently running tests.

Jest and Mocha support in TypeScript SDK

TypeScript SDK provides sample tests for Jest (minimum version 27.0.0) and Mocha. Jest tests must use testEnvironment: 'node'; testEnvironment: 'jsdom' is not supported. Mocha tests can use @temporalio/nyc-test-coverage for test coverage library.

Skip time in mock Activities

Call TestWorkflowEnvironment.sleep() from the mock Activity to skip time during Activity execution in tests. This allows testing Workflows that send notifications or perform actions after an Activity completes.

Example: Creating Worker with TestWorkflowEnvironment

import { Worker } from '@temporalio/worker'; import { v4 as uuid4 } from 'uuid'; import { workflowFoo } from './workflows'; test('workflowFoo', async () => { const worker = await Worker.create({ connection: testEnv.nativeConnection, taskQueue: 'test', }); const result = await worker.runUntil( testEnv.client.workflow.execute(workflowFoo, { workflowId: uuid4(), taskQueue: 'test', }) ); expect(result).toEqual('foo'); });

TestWorkflowEnvironment properties for Worker setup

TestWorkflowEnvironment has client and nativeConnection properties for creating Workers. The client property connects to the test server, and nativeConnection is used as the connection parameter when creating a Worker.

Example: Setting up TestWorkflowEnvironment

import { TestWorkflowEnvironment } from '@temporalio/testing'; let testEnv: TestWorkflowEnvironment; beforeAll(async () => { testEnv = await TestWorkflowEnvironment.createTimeSkipping(); }); afterAll(async () => { await testEnv?.teardown(); });

TestWorkflowEnvironment.createTimeSkipping() setup

TestWorkflowEnvironment.createTimeSkipping() starts the test server. A typical test suite should set up a single instance of the test environment to be reused in all tests, for example in a Jest beforeAll hook or a Mocha before() hook.

Install @temporalio/testing package

Install the @temporalio/testing package for TypeScript SDK to access the TestWorkflowEnvironment and other testing utilities: npm install @temporalio/testing

Example: Using workflowInterceptorModules for assert handling

import { TestWorkflowEnvironment, workflowInterceptorModules } from '@temporalio/testing'; const worker = await Worker.create({ connection: testEnv.nativeConnection, interceptors: { workflowModules: workflowInterceptorModules, }, workflowsPath: require.resolve('./workflows/file-with-workflow-function-to-test'), }); await worker.runUntil( testEnv.client.workflow.execute(functionToTest, workflowOptions), ); // throws WorkflowFailedError

Pitfall: AssertionError in Workflow causes task retry

By default, a failed assert statement throws AssertionError, which causes a Workflow Task to fail and be indefinitely retried. To prevent this, use workflowInterceptorModules from @temporalio/testing to catch AssertionError and turn it into an ApplicationFailure that fails the entire Workflow Execution instead of just the Workflow Task.

Use assert statement in Workflow

The assert statement from the Node.js assert module is included in Workflow bundles and provides a convenient way to insert debugging assertions into the Workflow context in TypeScript and Python.

Example: Testing function in Workflow context

// workflows/file-with-workflow-function-to-test.ts import { sleep } from '@temporalio/workflow'; export async function functionToTest(): Promise<number> { await sleep('1 day'); return 42; } // test.ts const worker = await Worker.create({ connection: testEnv.nativeConnection, workflowsPath: require.resolve('./workflows/file-with-workflow-function-to-test'), }); const result = await worker.runUntil( testEnv.client.workflow.execute(functionToTest, workflowOptions), ); assert.equal(result, 42);

Execute non-Workflow functions with Worker

To test a function in Workflow code that is not a Workflow, put the file it is exported from in WorkerOptions.workflowsPath. Then execute the function as if it were a Workflow using testEnv.client.workflow.execute().

Testing functions in Workflow context

For a function or method to run in the Workflow context (where it is possible to get current Workflow info or run inside the sandbox), it needs to be run by the Worker as if it were a Workflow. This is applicable in Python and TypeScript.

Example: Time skipping in mock Activity

it('sends reminder email if processOrder does not complete in time', async () => { let emailSent = false; const mockActivities: ReturnType<typeof createActivities> = { async processOrder() { await env.sleep('2 days'); }, async sendNotificationEmail() { emailSent = true; }, }; const worker = await Worker.create({ connection: env.nativeConnection, taskQueue: 'test', workflowsPath: require.resolve('../workflows'), activities: mockActivities, }); await worker.runUntil( env.client.workflow.execute(processOrderWorkflow, { workflowId: uuid(), taskQueue: 'test', args: [{ orderProcessingMS: ms('3 days'), sendDelayedEmailTimeoutMS: ms('1 day') }], }), ); assert.ok(emailSent); });

Reset Workflow Execution CLI example

temporal workflow reset --workflow-id my-background-check --event-id 4 --reason "Fixed non-deterministic code"

Reset Workflow Execution via Temporal CLI

Use the temporal workflow reset command to reset a Workflow Execution. The command requires --workflow-id, --event-id, and --reason parameters. By default, the command resets the latest Workflow Execution in the default Namespace. Use --run-id to reset a specific run and --namespace to specify a different Namespace. TLS options like --tls-cert-path and --tls-key-path can be used for secure connections. After resetting, monitor the new Workflow Execution to ensure it completes successfully.

Reset Workflow Execution via Web UI

To reset a Workflow Execution via Web UI: navigate to the Workflow Execution details page, click the Reset button in the top right dropdown menu, select the Event ID to reset to, provide a reason for the reset, and confirm the reset. The Web UI shows available reset points and creates a link to the new Workflow Execution after the reset completes.

Check historyLength against threshold in Continue-As-New test hook

Example of a test hook for Continue-As-New: ```typescript shouldContinueAsNew(): boolean { if (wf.workflowInfo().continueAsNewSuggested) { return true; } if (this.maxHistoryLength !== undefined && wf.workflowInfo().historyLength > this.maxHistoryLength) { return true; } return false; } ``` This helper method checks both the Temporal-recommended continueAsNewSuggested flag and a test-only maxHistoryLength value to determine if it is time to continue as new.

Test Continue-As-New with test hook to check behavior faster

Testing Workflows that naturally Continue-As-New may be time-consuming and resource-intensive. Instead, add a test hook to check your Workflow's Continue-As-New behavior faster in automated tests. For example, when a test flag is true, create a test-only variable with a small max history length and check it each time the Workflow considers using Continue-As-New.

Server performance debugging

Server performance can be debugged using Cloud metrics for Temporal Cloud deployments or self-hosted Server metrics for self-hosted deployments. Refer to the Cloud metrics documentation or self-hosted production checklist for scaling and metrics guidance.

Give your agent this brain