replay-suite

Still under development. Have not tried to clean up the code yet. All existing implementations details are subject to change.

TODOs:

• can't seem to reproduce NDEs by changing input types, even with the dev server. Need to look a bit more into this.
• allow running against an existing server: download just the executions generated by the tests, not all executions for a workflow type (should be pretty simple via run id)
• workflow with timer takes a shit ton of time to run even with server timer granularity set to 1ms. need some further investiagtions

See replaytests/ for runnable examples.

What this is

replaysuite is an experiment in generating Temporal replay-test histories from existing workflow unit tests. It aims extend coverage to nondeterminism errors without needing to pull histories from a live Temporal cluster. Tests still run through testsuite.WorkflowTestSuite as normal, but each workflow execution is also mirrored to a local (suite-managed) Temporal dev server to produce real event histories. Existing histories are replayed against the current workflow code before new histories are generated.

The idea behind this is, given full test coverage for a workflow, the unit tests themselves should be able to generate all possible history permutations or at least a reasonably good set. Due to the way history is constructed, the unit tests allow deriving fast implementations that receive the mocked inputs and return the mocked outputs to be run against the real server, making execution very fast.

The wrapper is intended to be close to a drop-in replacement for embedding testsuite.WorkflowTestSuite in a testify/suite test (in the future, it may end up as a full drop-in replacement).

How it differs from the standard test suite

There are two main API differences from a normal testsuite.WorkflowTestSuite setup:

• Embed replaysuite.WorkflowTestSuite instead of testsuite.WorkflowTestSuite. It is a drop-in for the SDK test suite, but also owns the dev server, shared worker, history replay step, and history dumping step.
• s.NewTestWorkflowEnvironment() in SetupTest now returns a wrapping *replaysuite.Env. The returned environment wraps a regular *testsuite.TestWorkflowEnvironment, so existing unit-test assertions and mocks still run there. When ExecuteWorkflow is called, the wrapper first runs the normal unit test and then mirrors the same workflow execution to the local dev server to generate a real history.

The replay suite uses WithReplayWorkflows (passed to Start) to:

• replay existing .histories/<workflow_type>/*.json files against the current workflow implementation at the start of the suite
• decide which mirrored workflow executions should be written back to .histories/ at the end of the suite

You may still need env.RegisterWorkflow(...) for normal unit-test behavior, for example when a workflow is invoked by name or when the inner SDK test environment needs to know about a child workflow. That does not opt the workflow into replay testing; WithReplayWorkflows does.

Configuration

Configure and launch the suite via Start with functional options; Stop dumps observed histories and tears it down:

type WorkflowTestSuite struct {
	suite.Suite
	replaysuite.WorkflowTestSuite
}

func (s *WorkflowTestSuite) SetupSuite() {
	s.Require().NoError(s.WorkflowTestSuite.Start(
		replaysuite.WithReplayWorkflows(Workflow),
		replaysuite.WithRedactWorkerIdentity(),
	))
}

func (s *WorkflowTestSuite) TearDownSuite() {
	s.Require().NoError(s.WorkflowTestSuite.Stop())
}

Available options: WithReplayWorkflows, WithHistoriesDir, WithRedactWorkerIdentity, WithSkipReplayTest. See suite.go for details.

Supported SDK test-suite APIs

The minimum version of the Go sdk required to use this suite is v1.35.0 since it needs RegisterDynamicWorkflow and RegisterDynamicActivity to overwrite the activity and child workflow implementation using the mocked inputs and outputs.

The replay wrapper currently supports the following SDK-style test functions:

• ExecuteWorkflow(...)
• OnActivity(...).Return(...)
• OnActivity(...).Return(...).Once()
• OnWorkflow(...).Return(...)
• OnWorkflow(...).Return(...).Once()
• RegisterActivity(...)
• RegisterActivityWithOptions(...)
• RegisterWorkflow(...)
• RegisterWorkflowWithOptions(...)
• IsWorkflowCompleted()
• GetWorkflowError()
• GetWorkflowResult(...)

OnActivity and OnWorkflow support testify argument matchers such as mock.Anything and mock.MatchedBy.

Some important limitations are intentional for now:

• only Return() and Once() are implemented on the custom mock wrapper; other SDK mock-chain helpers are not implemented yet
• anything that isn't mentioned, e.g. signals, queries, updates, Nexus, continue-as-new, heartbeats are not covered yet