Debugging Workflows in production environment
You can debug production Workflows using the .NET SDK with the following tools: Web UI, Temporal CLI, Replay, Tracing, and Logging.
Temporal · Develop · all subjects
testing-and-debugging
164 notes in this subject, read out of this brain and free to use. This is page 1 of 3.
Testing Continue-As-New with test hooks in .NET
To test Continue-As-New behavior faster in automated tests without waiting for natural Event History limits, add a test hook parameter to your Workflow input. When the test hook is enabled, set a small maxHistoryLength value and use it alongside Workflow.ContinueAsNewSuggested to trigger Continue-As-New during testing.
.NET Continue-As-New test helper example
private bool ShouldContinueAsNew => // Don't continue as new while update running Workflow.AllHandlersFinished && // Continue if suggested or, for ease of testing, max history reached (Workflow.ContinueAsNewSuggested || Workflow.CurrentHistoryLength > maxHistoryLength);
Assert workflow test results
After executing a Workflow in tests, use s.env.IsWorkflowCompleted() to assert the Workflow ran through completion, s.env.GetWorkflowError() to check for errors, and s.env.GetWorkflowResult(&value) to retrieve the Workflow's return value for additional asserts.
Go SDK workflow unit test example querying workflow state
Example test that queries Workflow state during execution: ```go func (s *UnitTestSuite) Test_ProgressWorkflow() { value := 0 s.env.RegisterDelayedCallback(func() { res, err := s.env.QueryWorkflow("getProgress") s.NoError(err) err = res.Get(&value) s.NoError(err) s.Equal(10, value) }, time.Second*10+time.Millisecond*1) s.env.ExecuteWorkflow(ProgressWorkflow, 0) s.True(s.env.IsWorkflowCompleted()) res, err := s.env.QueryWorkflow("getProgress") s.NoError(err) err = res.Get(&value) s.NoError(err) s.Equal(value, 100) } ``` Use RegisterDelayedCallback to query the Workflow at specific points during execution. After ExecuteWorkflow() completes, you can also query the final state.
Go SDK workflow unit testing with TestWorkflowEnvironment
The Temporal Go SDK provides a test framework with TestWorkflowEnvironment for implementing unit tests and functional tests of Workflow logic. Create a test suite struct that embeds both suite.Suite from testify and testsuite.WorkflowTestSuite. Implement a SetupTest method to initialize a new test environment before each test. Implement an AfterTest function to assert that all mocks were called via s.env.AssertExpectations(s.T()). Use suite.Run to execute tests recognized by the go test command. Timeout for the entire test can be set using SetTestTimeout in the Workflow or Activity environment.
Go SDK workflow unit test example with mocked activity
Example test that mocks an Activity to simulate a failure: ```go func (s *UnitTestSuite) Test_SimpleWorkflow_ActivityFails() { s.env.OnActivity(SimpleActivity, mock.Anything, mock.Anything).Return( "", errors.New("SimpleActivityFailure")) s.env.ExecuteWorkflow(SimpleWorkflow, "test_failure") s.True(s.env.IsWorkflowCompleted()) err := s.env.GetWorkflowError() s.Error(err) var applicationErr *temporal.ApplicationError s.True(errors.As(err, &applicationErr)) s.Equal("SimpleActivityFailure", applicationErr.Error()) } ``` This test sets up a mock that returns an error, executes the Workflow, and asserts that the Workflow completed and returned the expected error.
Go SDK workflow unit test example with activity parameter validation
Example test that replaces an Activity with an alternate implementation to validate parameters: ```go func (s *UnitTestSuite) Test_SimpleWorkflow_ActivityParamCorrect() { s.env.OnActivity(SimpleActivity, mock.Anything, mock.Anything).Return( func(ctx context.Context, value string) (string, error) { s.Equal("test_success", value) return value, nil }) s.env.ExecuteWorkflow(SimpleWorkflow, "test_success") s.True(s.env.IsWorkflowCompleted()) s.NoError(s.env.GetWorkflowError()) } ``` Providing a function implementation as the Return parameter allows validation of Activity parameters. The framework will execute this function whenever the Activity is invoked.
TEMPORAL_DEBUG environment variable for debugging workflows
Set the TEMPORAL_DEBUG environment variable to true before debugging your Workflow Definition to alleviate Potential deadlock detected errors. The Temporal Go SDK includes deadlock detection which fails a Workflow Task if code blocks for over a second without relinquishing execution control. This can cause PanicError: Potential deadlock detected while stepping through Workflow Definitions during debugging. Make sure to set TEMPORAL_DEBUG to true only during debugging, not in production.
Query workflow state in tests
TestWorkflowEnvironment instances have a QueryWorkflow() method that lets you query the state of the currently running Workflow. Query the Workflow either after ExecuteWorkflow() is done or in a RegisterDelayedCallback() callback, otherwise you'll get a runtime error panic. RegisterDelayedCallback() doesn't actually make your test wait; Temporal's test framework advances time internally, so tests complete quickly.
Go SDK full workflow unit test setup example
Complete example of a workflow unit test suite setup: ```go package sample import ( "context" "errors" "testing" "github.com/stretchr/testify/mock" "github.com/stretchr/testify/suite" "go.temporal.io/sdk/activity" "go.temporal.io/sdk/testsuite" ) type UnitTestSuite struct { suite.Suite testsuite.WorkflowTestSuite env *testsuite.TestWorkflowEnvironment } func (s *UnitTestSuite) SetupTest() { s.env = s.NewTestWorkflowEnvironment() } func (s *UnitTestSuite) AfterTest(suiteName, testName string) { s.env.AssertExpectations(s.T()) } func (s *UnitTestSuite) Test_SimpleWorkflow_Success() { s.env.ExecuteWorkflow(SimpleWorkflow, "test_success") s.True(s.env.IsWorkflowCompleted()) s.NoError(s.env.GetWorkflowError()) } func TestUnitTestSuite(t *testing.T) { suite.Run(t, new(UnitTestSuite)) } ``` This demonstrates the basic structure: embed both suite.Suite and testsuite.WorkflowTestSuite, initialize the test environment in SetupTest, and assert expectations in AfterTest.
RegisterDelayedCallback for signals in workflow tests
RegisterDelayedCallback can be used to send Signals during workflow tests. When using Signal-With-Start, set the delay to 0.
Activity mocking in workflow tests
Mock Activity behavior using s.env.OnActivity(ActivityName, mock.Anything, mock.Anything).Return(...). You can return a simple value or error, or provide a function implementation as the Return parameter to replace the Activity with an alternate implementation. The framework will execute this function whenever the Activity is invoked and validate that the signature matches the original Activity function.
workflowcheck tool for static analysis of workflows
The Temporal Go SDK provides a command line tool called workflowcheck to statically analyze Workflow Definitions. This tool can help eliminate potential instances of non-determinism in workflow code.
Override Activity implementation in test
Provide a function implementation to the `Return()` method instead of a simple value: `s.env.OnActivity(ActivityFunc, mock.Anything, mock.Anything).Return(func(ctx context.Context, value string) (string, error) { /* custom logic */ })`. The framework validates that the mock function signature matches the original Activity signature and executes this function whenever the Activity is invoked.
Activity testing with mock environment
An Activity can be tested with a mock Activity environment created via `NewTestActivityEnvironment()` from `WorkflowTestSuite`. This allows testing the Activity in isolation by calling it directly without creating a Worker. The mock environment provides a way to mock the Activity context, listen to Heartbeats, and cancel the Activity.
Automatic time skipping in Go test environment
The `testsuite.TestWorkflowEnvironment` automatically skips time when possible. Time advances automatically whenever there are no Activities running. Workflow timers like `workflow.Sleep()` are fast-forwarded, but time doesn't skip while Activities are executing. This allows tests with long sleep periods or retry intervals to complete quickly.
Execute Activity in test
Create a test Activity environment with `env := s.NewTestActivityEnvironment()`, register the Activity with `env.RegisterActivity(ActivityFunc)`, then execute it with `result, err := env.ExecuteActivity(ActivityFunc, args...)`. Retrieve the result with `result.Get(&variable)`.
Test timeout configuration
Set the timeout for the entire test using `SetTestTimeout` in the Workflow or Activity environment.
Get Workflow Event History for replay
Use `client.GetWorkflowHistory(ctx, id, runID, false, enums.HISTORY_EVENT_FILTER_TYPE_ALL_EVENT)` to retrieve the Event History. Iterate through results with `iter.HasNext()` and `iter.Next()` to collect all events into a `*history.History` object.
Important: use workflow.Sleep not time.Sleep in tests
In Temporal test environments, use `workflow.Sleep()` instead of `time.Sleep()` because the test environment doesn't stub out `time.Sleep()`. Using `time.Sleep()` will cause the test to actually wait the full duration instead of skipping time.
Manual time skipping with RegisterDelayedCallback
By default the Go test suite uses a mock Workflow clock which automatically moves forward to fire the next Timer when blocked. For fine-grained control, use `env.RegisterDelayedCallback(callbackFunc, delayDuration)` to create a new Timer with the mock Workflow clock. When the Timer fires, the callback executes. This is useful for controlled time progression in tests.
Override Nexus asynchronous operation in test
Create a handler Workflow function, then create a Nexus service with `nexus.NewService(serviceName)` and register a workflow run operation using `nexus.NewWorkflowRunOperation(operationName, handlerWorkflow, func(ctx context.Context, input InputType, options nexus.StartOperationOptions) (client.StartWorkflowOptions, error) { /* custom logic */ })`. Register the service to the test environment with `env.RegisterNexusService(service)`.
Activity implementation without mocking is executed in Workflow tests
Unless Activity invocations are mocked or Activity implementation replaced, the test environment will execute the actual Activity code including any calls to outside services. This applies when testing Workflows without providing mock Activity implementations.
Cancel Activity in test
To test Activity cancellation, create a context with `ctx, cancel := context.WithCancel(context.Background())`, set it in worker options with `env.SetWorkerOptions(worker.Options{BackgroundActivityContext: ctx})`, then call `cancel()` when you want to cancel the Activity. The Activity should return a cancellation error.
Check Workflow test completion and errors
After executing a Workflow, check completion with `s.env.IsWorkflowCompleted()` which returns true if the Workflow ran through completion. Check for errors with `s.env.GetWorkflowError()`. Retrieve the result with `s.env.GetWorkflowResult(&value)` if the Workflow returned a value.
Mock Activity to return error in Workflow test
Use `s.env.OnActivity(ActivityFunc, mock.Anything, mock.Anything).Return("", errors.New("ErrorMessage"))` to mock an Activity that returns an error. This allows testing how the Workflow handles Activity failures without modifying Workflow code.
Execute Workflow in test
Call `s.env.ExecuteWorkflow(WorkflowFunc, args...)` to execute the Workflow logic and any invoked Activities inside the test process. The first parameter is the Workflow function, and subsequent parameters are values for the Workflow's custom input parameters.
Listen to Activity Heartbeats in tests
Use `env.SetOnActivityHeartbeatListener()` to set a callback function that fires when the Activity sends a Heartbeat. The callback receives `*activity.Info` and `converter.EncodedValues` parameters. Decode the heartbeat details by calling `details.Get(&variable)` on the EncodedValues.
Test suite structure for Workflow testing in Go
Define a test suite struct that embeds both `suite.Suite` from testify and `testsuite.WorkflowTestSuite` from the Temporal test framework. Add a property to hold an instance of `testsuite.TestWorkflowEnvironment`. Implement a `SetupTest()` method to initialize a new test environment before each test, ensuring each test runs in its own isolated sandbox. Implement an `AfterTest(suiteName, testName string)` function to call `s.env.AssertExpectations(s.T())` to assert that all mocks were called. Create a regular test function and pass the struct to `suite.Run()`.
Register delayed callback to query Workflow during test
Use `s.env.RegisterDelayedCallback(func() { ... }, time.Duration)` to query a Workflow at a specific point in time during execution. The test framework advances time internally so the test completes quickly despite the time duration specified. `RegisterDelayedCallback` can also be used to send Signals. When using Signal-With-Start, set the delay to 0.
Override Nexus synchronous operation in test
Create a Nexus service with `nexus.NewService(serviceName)`, register a synchronous operation using `nexus.NewSyncOperation(operationName, func(ctx context.Context, input InputType, options nexus.StartOperationOptions) (OutputType, error) { /* custom logic */ })`, then register the service to the test environment with `env.RegisterNexusService(service)`.
Mock asynchronous Nexus operation in test
Use `s.env.OnNexusOperation(...).Return(&nexus.HandlerStartOperationResultAsync{OperationToken: "token-string"}, nil)`, then call `env.RegisterNexusAsyncOperationCompletion(serviceName, operationName, operationToken, outputValue, errorOrNil, delay)` to register the operation completion. The operation token in the completion must match the one in the mock result.
Mock synchronous Nexus operation in test
Use `s.env.OnNexusOperation(serviceName, operationReference, inputExample, workflowNexusOperationOptions).Return(&nexus.HandlerStartOperationResultSync[OutputType]{Value: outputExample}, nil)`. Create operation reference with `nexus.NewOperationReference[InputType, OutputType](operationName)` if you don't have the operation implementation available. You can add a delay with `.After(duration)`.
Query Workflow in test with QueryWorkflow method
`TestWorkflowEnvironment` instances have a `QueryWorkflow()` method that lets you query the state of the currently running Workflow. Always query the Workflow either after `ExecuteWorkflow()` completes or in a `RegisterDelayedCallback()` callback; otherwise you will get a runtime error panic.
Example: Test hook for Continue-As-New in Go
func (cm *ClusterManager) shouldContinueAsNew(ctx workflow.Context) bool { if workflow.GetInfo(ctx).GetContinueAsNewSuggested() { return true } if cm.maxHistoryLength > 0 && workflow.GetInfo(ctx).GetCurrentHistoryLength() > cm.maxHistoryLength { return true } return false } This helper method checks both the framework's Continue-As-New suggestion and a custom maxHistoryLength threshold for testing purposes.
Java IDE debugger for Workflow debugging
You can debug Workflow code using a debugger provided by your favorite Java IDE, in addition to writing unit and integration tests.
PotentialDeadlockException during Workflow debugging
When stepping through Workflow code during IDE debugging, you may encounter PotentialDeadlockException if the debugger causes the code to block for over a second without relinquishing execution control. This is due to the Temporal Java SDK's deadlock detection mechanism.
TEMPORAL_DEBUG environment variable prevents deadlock detection in debugger
Set the TEMPORAL_DEBUG environment variable to true before debugging Workflow code in an IDE debugger to avoid PotentialDeadlockException errors. The Temporal Java SDK includes deadlock detection that fails a Workflow Task if code blocks for over a second without relinquishing execution control. Make sure to set TEMPORAL_DEBUG to true only during debugging, not in production.
Manual time skipping with TestWorkflowEnvironment.sleep(Duration)
Use TestWorkflowEnvironment.sleep(Duration) to manually advance virtual time and inspect intermediate Workflow state. Start the Workflow asynchronously with WorkflowClient.start(), then call testEnv.sleep() from the test to advance time and check Workflow state with query methods.
Mock Nexus handlers with JUnit5 using worker.registerWorkflowImplementationFactory
With JUnit5 TestWorkflowExtension, mock Nexus handler workflows by calling worker.registerWorkflowImplementationFactory(HelloHandlerWorkflow.class, () -> { HelloHandlerWorkflow mockHandler = mock(HelloHandlerWorkflow.class); when(mockHandler.hello(any())).thenReturn(...); return mockHandler; }) in the test method.
Use Workflow.sleep() not Thread.sleep() for testable timers
In Workflow code, use Workflow.sleep() to create Temporal timers that the test environment can skip, not Thread.sleep(). Thread.sleep() blocks a Java thread in real time and is not controlled by Temporal's time-skipping test server.
Example of manual time skipping with query method
Example of manually advancing Workflow time: ```java @Test void manuallyAdvanceWorkflowTime() { try (TestWorkflowEnvironment testEnv = TestWorkflowEnvironment.newInstance()) { Worker worker = testEnv.newWorker(TASK_QUEUE); worker.registerWorkflowImplementationTypes(ProgressWorkflowImpl.class); testEnv.start(); ProgressWorkflow workflow = testEnv.getWorkflowClient().newWorkflowStub( ProgressWorkflow.class, WorkflowOptions.newBuilder().setTaskQueue(TASK_QUEUE).build()); WorkflowClient.start(workflow::run); assertEquals(0, workflow.daysElapsed()); testEnv.sleep(Duration.ofHours(25)); assertEquals(1, workflow.daysElapsed()); } } ```
Mock Nexus Service with OperationHandler.sync for synchronous testing
An alternative to mocking Nexus handlers is to mock the Nexus Service itself using OperationHandler.sync. Create a test-only Nexus service implementation annotated with @ServiceImpl and @OperationImpl that delegates to a Mockito mock. OperationHandler.sync returns results inline without requiring backing workflows, allowing full control over return values and input verification.
Replay Event Histories from server using WorkflowReplayer
Example of replaying Event Histories downloaded from the server using WorkflowReplayer: ```java ListWorkflowExecutionsRequest listRequest = ListWorkflowExecutionsRequest.newBuilder() .setNamespace(client.getOptions().getNamespace()) .setQuery("TaskQueue = 'mytaskqueue'") .build(); ListWorkflowExecutionsResponse response = service.blockingStub().listWorkflowExecutions(listRequest); List<WorkflowExecutionHistory> histories = response.getExecutionsList().stream() .map(info -> { GetWorkflowExecutionHistoryResponse weh = service.blockingStub().getWorkflowExecutionHistory( GetWorkflowExecutionHistoryRequest.newBuilder() .setNamespace(namespace) .setExecution(info.getExecution()) .build()); return new WorkflowExecutionHistory(weh.getHistory(), info.getExecution().getWorkflowId()); }) .collect(Collectors.toList()); WorkflowReplayer.replayWorkflowExecutions(histories, true, WorkflowA.class, WorkflowB.class, WorkflowC.class); ```
Three types of automated tests in Temporal Java
End-to-end tests run a Temporal Server and Worker with all Workflows, Activities, and Nexus Operations, starting and interacting with Workflows from a Client. Integration tests run anything between end-to-end and unit testing, such as Activities with mocked Context, Workers with mock Activities and Nexus Operations using a Client, or Workflows with mocked SDK imports. Unit tests run a piece of Workflow, Activity, or Nexus Operation code and mock any code it calls. Integration tests are generally recommended as the majority of your test suite.
Mock Nexus operations in tests by registering service implementation
When integration testing Workflows with a Worker, mock Nexus operations by providing mock Nexus Service handlers to the Worker. Use setNexusServiceImplementation in the TestWorkflowRule or TestWorkflowExtension builder to set up Nexus endpoints and handler workflows needed for testing.
Automatic time skipping in tests with TestWorkflowEnvironment
When executing a Workflow and waiting for the result in TestWorkflowEnvironment, time is automatically skipped for Workflow timers such as Workflow.sleep(). This means Workflow timers are fast-forwarded and tests complete quickly. Time does not skip while Activities and Nexus operations are executing; Nexus operation handlers timeout after 10 seconds and time skipping resumes while waiting for retries.
Replay single Event History from JSON file with WorkflowReplayer
Example of replaying a single Event History loaded from a JSON file using WorkflowReplayer: ```java File file = new File("my_history.json"); WorkflowReplayer.replayWorkflowExecution(file, MyWorkflow.class); ```
WorkflowReplayer throws error on non-deterministic Event History
If Event History is non-deterministic, WorkflowReplayer throws an error. The failFast argument to replayWorkflowExecutions determines whether to fail immediately or wait until all histories have been replayed. Set failFast to false to continue replaying all histories before failing.
Mock Activities in integration tests with Mockito
When integration testing Workflows with a Worker, mock Activities by providing mock Activity implementations. Use Mockito to create mock Activity stubs when you do not wish to execute actual Activity or Nexus Operation implementations during unit testing. Example: GreetActivities activities = mock(GreetActivities.class, withSettings().withoutAnnotations()); when(activities.greet("Temporal")).thenReturn("Hello Temporal!");
Cancel Activity in tests using registerDelayedCallback
To test Activity cancellation, use TestWorkflowEnvironment with registerDelayedCallback to trigger a signal after a delay, simulating cancellation. The Activity should react correctly to the cancellation. Test must verify that WorkflowFailedException is thrown with ActivityFailure as the cause and CanceledFailure as the underlying cause.
TestWorkflowRule for JUnit4 testing
For JUnit4 tests, use TestWorkflowRule to simplify Temporal test environment setup and Workflow Worker creation and shutdown. Build with TestWorkflowRule.newBuilder().setWorkflowTypes(...).setActivityImplementations(...).build(). Access the workflow client with testWorkflowRule.getWorkflowClient() and task queue with testWorkflowRule.getTaskQueue().
Test Activity implementation example with TestActivityEnvironment
Example of a unit test for an Activity using TestActivityEnvironment: ```java public class GreetActivitiesTest { @Test public void testActivityImpl() { TestActivityEnvironment testEnv = TestActivityEnvironment.newInstance(); testEnv.registerActivitiesImplementations(new GreetActivitiesImpl()); GreetActivities activities = testEnv.newActivityStub(GreetActivities.class); String result = activities.greet("Temporal"); assertEquals("Hello Temporal!", result); } } ```
TestWorkflowEnvironment provides in-memory Temporal implementation with automatic time skipping
The TestWorkflowEnvironment class is an in-memory implementation of the Temporal service that supports automatic time skipping. It allows testing long-running Workflows in seconds without changing Workflow code. TestWorkflowEnvironment can be used with any Java unit testing framework such as JUnit.
Add temporal-testing dependency for Java SDK testing
Add io.temporal:temporal-testing as a test dependency to enable the Java SDK test framework. For Maven: use dependency with groupId io.temporal, artifactId temporal-testing, version 1.36.0, and scope test. For Gradle Groovy DSL: testImplementation("io.temporal:temporal-testing:1.36.0"). For JUnit4 or JUnit5 extensions, use testImplementation with capabilities requiring io.temporal:temporal-testing-junit4 or io.temporal:temporal-testing-junit5 respectively.
TestActivityEnvironment and TestActivityExtension for Activity testing
Temporal provides TestActivityEnvironment and TestActivityExtension classes for testing Activities outside the scope of a Workflow. These classes provide a way to mock the Activity context, listen to Heartbeats, and cancel the Activity, allowing Activities to be tested in isolation by calling them directly without needing to create a Worker.
Listen to Activity Heartbeats in tests with setActivityHeartbeatListener
Set up a heartbeat listener on TestActivityEnvironment using env.setActivityHeartbeatListener(Void.class, heartbeat -> heartbeatCount.incrementAndGet()) to verify that Activities emit the expected number of Heartbeats during testing.
TestWorkflowExtension for JUnit5 testing
For JUnit5 tests, use TestWorkflowExtension with @RegisterExtension annotation to simplify Temporal test environment setup and Workflow Worker startup and shutdown. Build with TestWorkflowExtension.newBuilder().setWorkflowTypes(...).setActivityImplementations(...).build(). The extension injects TestWorkflowEnvironment, Worker, and workflow stubs as test method parameters.
Spring Boot testing with test server
To test Temporal code in Spring Boot, enable the test server by setting spring.temporal.test-server.enabled: true in application.yml. This reconfigures the client to use io.temporal.testing.TestWorkflowEnvironment with an in-memory Java Test Server. When enabled, spring.temporal.connection configuration is ignored.