NullReferenceException in CucumberMessageFactory.ToTestStepStarted during scenario cleanup causes scenario to be silently omitted from HTML report

[thogr]

Environment

Reqnroll.NUnit: 3.3.4 (also reproduced on 3.3.3)
NUnit: 4.5.1
NUnit3TestAdapter: 4.5.0
NUnitRetry.ReqnrollPlugin: 2.0.17
.NET: 9.0

Description

When a scenario step fails with an exception, Reqnroll's CucumberMessagePublisher crashes with its own NullReferenceException during scenario cleanup (OnAfterLastStepAsync). As a result, the scenario is silently omitted from the HTML report even though the failure is correctly printed to the console/standard output.

Steps to reproduce

Intermittent — triggered when a step throws an exception during execution. Occurs under parallel test runs. No minimal repro has been isolated yet.

Stack trace (3.3.4)

System.NullReferenceException : Object reference not set to an instance of an object.
   at Reqnroll.Formatters.PayloadProcessing.Cucumber.CucumberMessageFactory.ToTestStepStarted(TestStepExecutionTracker testStepExecutionTracker)
   at Reqnroll.Formatters.ExecutionTracking.TestStepExecutionTracker.ProcessEvent(StepStartedEvent stepStartedEvent)
   at Reqnroll.Formatters.ExecutionTracking.TestCaseExecutionTracker.ProcessEvent(StepStartedEvent stepStartedEvent)
   at Reqnroll.Formatters.ExecutionTracking.PickleExecutionTracker.ProcessEvent(StepStartedEvent stepStartedEvent)
   at Reqnroll.Formatters.ExecutionTracking.FeatureExecutionTracker.ProcessEvent(StepStartedEvent stepStartedEvent)
   at Reqnroll.Formatters.PubSub.CucumberMessagePublisher.StepStartedEventHandler(StepStartedEvent stepStartedEvent)
   at Reqnroll.Formatters.PubSub.CucumberMessagePublisher.OnEventAsync(IExecutionEvent executionEvent)
   at Reqnroll.Events.TestThreadExecutionEventPublisher.PublishEventAsync(IExecutionEvent executionEvent)
   at Reqnroll.Infrastructure.TestExecutionEngine.<ExecuteStepAsync>...MoveNext()
   at Reqnroll.Infrastructure.TestExecutionEngine.OnAfterLastStepAsync()
   at Reqnroll.TestRunner.CollectScenarioErrorsAsync()
   at <FeatureClass>.ScenarioCleanupAsync()

On 3.3.3 the top frame was TestStepExecutionTracker.ProcessEvent instead of CucumberMessageFactory.ToTestStepStarted, suggesting a refactor moved the null dereference one level deeper without fixing the root cause.

Expected behavior

When a step fails, the scenario should still appear in the HTML report as failed.

Actual behavior

The scenario is absent from the HTML report. The failure is only visible in console/standard output.

Additional context

The crash occurs during OnAfterLastStepAsync → CollectScenarioErrorsAsync, meaning Reqnroll is trying to publish a StepStartedEvent (likely for an AfterScenario hook) while the TestStepExecutionTracker is in a state that CucumberMessageFactory.ToTestStepStarted cannot handle — presumably because the previous step failure left the tracker partially initialized.

NOTE: This analysis was created by Claude Code.

[clrudolphi]

We will need more information to fully diagnose the issue.
Based upon what was described, I am having trouble figuring out how the (minor) change introduced in v3.3.4 would have caused the stack trace to be different in this failure situation.

1. The description indicates that a step has failed. Please describe that failure and the type of exception thrown.
2. Can you share the console output from the entire test run?
3. Would it be possible for you to share the content of the reqnroll_report.ndjson file when message output is configured for the Formatters? (See this page in the configuration documentation)
4. I note that this test is using the NUnit.Retry plugin - is the failure occurring during a retry?
5. Is the problem reproducible without the use of NUnit.Retry plugin?

[thogr]

Thank you for the questions. While investigating a related symptom — failing tests appearing as "Skipped" in the VSCode Test Explorer — we found behaviour that may be connected to this issue and that might help narrow down the root cause. Our findings are below, along with answers to your questions.

1–3. Exception, console output and cucumber_messages.ndjson

The original scenarios that triggered the HTML report omission have since been disabled (@ignore) as the underlying GUI behaviour changed, so we cannot provide that specific data.

During the VSCode investigation we deliberately provoked a failure (injected a wrong element name to force a WebDriverTimeoutException). Under those conditions:

Console output:

Then the "XXXDefine selection" window is expanded
-> error: Timed out after 3 seconds: No element for the ExpansionPanelButton XXXDefine selection was available (3.1s)
When I expand "Details"
-> skipped because of previous errors
Then the "Details" window is expanded
-> skipped because of previous errors
And the "Define selection" window is contracted
-> skipped because of previous errors
When I contract "Details"
-> skipped because of previous errors
Then the "Details" window is contracted
-> skipped because of previous errors
cucumber_messages.ndjson step statuses: PASSED ×5, FAILED ×1, PASSED ×1, SKIPPED ×5. testRunFinished: { success: false }. The ndjson correctly reflects the failure.

NUnit XML: result="Failed" label="Error". dotnet test CLI: Failed: 1, Skipped: 0. Both correct.

4. Is the failure occurring during a retry?

For the VSCode symptom: no. For the original HTML report issue: we cannot say — that was observed with NUnit.Retry enabled and we did not retest that specific symptom without it.

5. Reproducible without NUnit.Retry?

For the VSCode symptom: yes, confirmed. We removed NUnitRetry.ReqnrollPlugin from both and in the .csproj, rebuilt (generated .cs files no longer contain [NRetry(2)]), and the behaviour persists.

The VSCode symptom in detail

With the deliberately provoked failure above, we consistently observe:

First run after a rebuild or VSCode restart: Test Explorer correctly shows Failed
Second and subsequent runs within the same VSCode session: Test Explorer shows Skipped
dotnet test CLI: always shows Failed: 1, Skipped: 0 — correct regardless of run number
The VSCode workspace storage confirms the second-run result is a clean skip — state: 5 (Skipped) with messages: [] (no error details), not a misclassified failure.

This pattern only occurs in VSCode's design-time mode, where the test host process is kept alive between runs. dotnet test spawns a fresh process each time and is unaffected.

We have not identified the exact mechanism, but the behaviour is consistently reproducible and we believe it is related to this issue given the shared code path through OnScenarioEndAsync.

NOTE: The above analysis was provided by Claude Code.

[clrudolphi]

we believe it is related to this issue given the shared code path through OnScenarioEndAsync.

The correlation to OnScenarioEndAsync is a false one. During Reqnroll scenario execution all step exceptions are captured, subsequent steps are allowed to execute (usually resulting in a 'skipped' status for them) and then the exception is rethrown at the end (during OnScenarioEnd).

The fact that your test behaves properly when using dotnet test but not while executing through VSCode strongly implies that the problem is somewhere in the integration of NUnit and VSCode (not necessarily with Reqnroll). Reqnroll does not provide direct integration with the IDE's test explorer but relies on the various test frameworks for that integration.

I may have more time available later in the week to attempt to repro your situation (I'm not a regular user of VSCode).
In the meantime you may try upgrading NUnit3TestAdapter. v4.5.0 is 3 years old and 8 releases out of date.

[thogr]

Yes, I'm and was using NUnit3TestAdapter 6.2.0 when the VSCode issue was discovered.

[thogr]

I have created a minimal repro for the VSCode issue showing scenarios as "skipped" instead of "failed". Not sure if it is related or the same issue. Full description in the README.md file in the attached zip-file.
Note: My "buddy" Caude Code created most of the analysis, but I ran the tests in vscode.

ReqnrollVSCodeRepro.zip

Steps to reproduce

1. Build the project: dotnet build
2. Open the project folder in VSCode with C# Dev Kit installed

First run (after a clean state — either dotnet clean + Reload Window, or first time opening the project):

3. Run the full suite in either debug or normal mode
4. Observe: both scenarios show Failed ✓✓

Subsequent debug runs (suite):

5. Run the full suite again in debug mode
6. Observe: Assert Fail shows Failed ✓, Assert That False shows Skipped ✗
7. Repeat — this pattern is consistent on every subsequent debug run

Subsequent normal runs (suite):

5. Run the full suite again in normal mode
6. Observe: both scenarios show Skipped ✗
7. Repeat — both continue to show Skipped on every run

[thogr]

The VSCode issue is a VSCode C# Dev Kit extension issue. When I rollback to C# Dev Kit extension 3.10.14 it works again.
It is related to microsoft/vscode-dotnettools#3193

[clrudolphi]

The VSCode issue is a VSCode C# Dev Kit extension issue. When I rollback to C# Dev Kit extension 3.10.14 it works again. It is related to microsoft/vscode-dotnettools#3193

@thogr May we treat this bug report as a false positive and close it? Or do you believe there is a problem still remaining that requires further investigation on our side?

[thogr]

No, please don’t yet. The Vscode problem was not related to the original nullref problem, although we thought so first. The problem is that the problem so intermittent, so we will need some time to see when it hits again.

[thogr]

I got the nullref again today. So yes it is still an issue.

OpenWB0101WithGrid (8s 344ms): Error Message: System.NullReferenceException : Object reference not set to an instance of an object.
Stack Trace:
at Reqnroll.Formatters.PayloadProcessing.Cucumber.CucumberMessageFactory.ToTestStepStarted(TestStepExecutionTracker testStepExecut
ionTracker)
at Reqnroll.Formatters.ExecutionTracking.TestStepExecutionTracker.ProcessEvent(StepStartedEvent stepStartedEvent)
at Reqnroll.Formatters.ExecutionTracking.TestCaseExecutionTracker.ProcessEvent(StepStartedEvent stepStartedEvent)
at Reqnroll.Formatters.ExecutionTracking.PickleExecutionTracker.ProcessEvent(StepStartedEvent stepStartedEvent)
at Reqnroll.Formatters.ExecutionTracking.FeatureExecutionTracker.ProcessEvent(StepStartedEvent stepStartedEvent)
at Reqnroll.Formatters.PubSub.CucumberMessagePublisher.StepStartedEventHandler(StepStartedEvent stepStartedEvent)
at Reqnroll.Formatters.PubSub.CucumberMessagePublisher.OnEventAsync(IExecutionEvent executionEvent)
at Reqnroll.Events.TestThreadExecutionEventPublisher.PublishEventAsync(IExecutionEvent executionEvent)
at Reqnroll.Infrastructure.TestExecutionEngine.<>c__DisplayClass57_0.<b__6>d.MoveNext()
--- End of stack trace from previous location ---
at Reqnroll.Infrastructure.TestExecutionEngine.<>c__DisplayClass57_0.<g__HandleStepExecutionExceptions|0>d.MoveN
ext()
at Reqnroll.Infrastructure.TestExecutionEngine.OnAfterLastStepAsync()
at Reqnroll.TestRunner.CollectScenarioErrorsAsync()
at AcceptanceTests.Features.WB0101GridFeature.ScenarioCleanupAsync()
at AcceptanceTests.Features.WB0101GridFeature.OpenWB0101WithGrid() in /home/imess/git/acctests-dotnet/AcceptanceTests/Features/Tes
t_011_WB0101_grid.feature:line 13
at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter1.GetResult() at NUnit.Framework.Internal.AsyncToSyncAdapter.Await[TResult](TestExecutionContext context, Func1 invoke)
at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(TestExecutionContext context, Func1 invoke) at NUnit.Framework.Internal.Commands.TestMethodCommand.RunTestMethod(TestExecutionContext context) at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context) at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.<>c__DisplayClass1_0.<Execute>b__0() at NUnit.Framework.Internal.Commands.DelegatingTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action) 1) at Reqnroll.Formatters.PayloadProcessing.Cucumber.CucumberMessageFactory.ToTestStepStarted(TestStepExecutionTracker testStepExe cutionTracker) at Reqnroll.Formatters.ExecutionTracking.TestStepExecutionTracker.ProcessEvent(StepStartedEvent stepStartedEvent) at Reqnroll.Formatters.ExecutionTracking.TestCaseExecutionTracker.ProcessEvent(StepStartedEvent stepStartedEvent) at Reqnroll.Formatters.ExecutionTracking.PickleExecutionTracker.ProcessEvent(StepStartedEvent stepStartedEvent) at Reqnroll.Formatters.ExecutionTracking.FeatureExecutionTracker.ProcessEvent(StepStartedEvent stepStartedEvent) at Reqnroll.Formatters.PubSub.CucumberMessagePublisher.StepStartedEventHandler(StepStartedEvent stepStartedEvent) at Reqnroll.Formatters.PubSub.CucumberMessagePublisher.OnEventAsync(IExecutionEvent executionEvent) at Reqnroll.Events.TestThreadExecutionEventPublisher.PublishEventAsync(IExecutionEvent executionEvent) at Reqnroll.Infrastructure.TestExecutionEngine.<>c__DisplayClass57_0.<<ExecuteStepAsync>b__6>d.MoveNext() --- End of stack trace from previous location --- at Reqnroll.Infrastructure.TestExecutionEngine.<>c__DisplayClass57_0.<<ExecuteStepAsync>g__HandleStepExecutionExceptions|0>d.MoveN ext() at Reqnroll.Infrastructure.TestExecutionEngine.OnAfterLastStepAsync() at Reqnroll.TestRunner.CollectScenarioErrorsAsync() at AcceptanceTests.Features.WB0101GridFeature.ScenarioCleanupAsync() at AcceptanceTests.Features.WB0101GridFeature.OpenWB0101WithGrid() in /home/imess/git/acctests-dotnet/AcceptanceTests/Features/Tes t_011_WB0101_grid.feature:line 13 at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter1.GetResult()
at NUnit.Framework.Internal.AsyncToSyncAdapter.Await[TResult](TestExecutionContext context, Func1 invoke) at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(TestExecutionContext context, Func1 invoke)
at NUnit.Framework.Internal.Commands.TestMethodCommand.RunTestMethod(TestExecutionContext context)
at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context)
at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.<>c__DisplayClass1_0.b__0()
at NUnit.Framework.Internal.Commands.DelegatingTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action
action)

This time it is not intermittent. I'll try to create a small repro.

[thogr]

Finally we have a solid a repro for the nullref. It does involve NUnitRetry, as suggested. The analysis below is again produced by Claude Code.
And again, forget about the VSCode issue (and repro) showing scenarios as "skipped", that I posted above. That has nothing to do with this.
This is the real original problem.

NullReferenceException in CucumberMessageFactory.ToTestStepStarted — scenario missing from HTML report (consistent CLI reproduction)

NUnitRetry is required

This is a consistent, CLI-reproducible form of issue #1083. It requires NUnitRetry (or equivalent retry mechanism) — the bug is triggered by the combination of a [AfterStep] hook that throws on the first attempt and a retry that gets further than the original attempt.

Setting maxRetries: 1 (no retry) produces a correct HTML report, confirming the retry is the trigger.

Root cause

On attempt 0, the [AfterStep] hook throws a non-assertion exception (e.g. WebDriverTimeoutException) after the Background step. Reqnroll builds the testCase message with a single AfterStep slot (one testStepId), because only the Background step ran before the failure.

On attempt 1 (retry), the Background's AfterStep passes. Reqnroll continues and executes the first scenario step, which fires AfterStep again — reusing the same testStepId (the testCase has no separate slot for it).

The html formatter's TestStepExecutionTracker sees the same testStepId a second time and enters a corrupted state, causing NullReferenceException in CucumberMessageFactory.ToTestStepStarted. The scenario is then missing from the HTML report.

The message (ndjson) formatter is unaffected because it writes events directly without going through CucumberMessageFactory, so the .ndjson output is complete and correct.

Minimal reproduction

Feature file (Features/AfterStepInBackground.feature):

Feature: AfterStep non-assertion exception with background

  Background:
    Then pass

  Scenario: AfterStep throws non-assertion exception after background step
    Then pass
    Then pass

Step definitions (StepDefinitions/Steps.cs):

[Binding]
public class Steps(ScenarioContext scenarioContext, IReqnrollOutputHelper outputHelper)
{
    [Then("pass")]
    public void Pass() { }

    [AfterStep]
    public void AfterStepThrows()
    {
        // Throw only on the first attempt so the retry gets further,
        // exposing the duplicate testStepId bug.
        if (NUnit.Framework.TestContext.CurrentContext.CurrentRepeatCount == 0)
            throw new System.TimeoutException("simulating non-assertion exception");
    }

    [AfterScenario]
    public void AfterScenario()
    {
        if (scenarioContext.TestError != null)
            outputHelper.WriteLine($"Scenario failed: {scenarioContext.TestError.Message}");
    }
}

ReqnrollVSCodeRepro.csproj:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net9.0</TargetFramework>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.9.0" />
    <PackageReference Include="NUnit" Version="4.5.1" />
    <PackageReference Include="NUnit3TestAdapter" Version="6.2.0" />
    <PackageReference Include="NUnitRetry.ReqnrollPlugin" Version="2.0.17" />
    <PackageReference Include="Reqnroll.NUnit" Version="3.3.4" />
    <PackageReference Include="Reqnroll.Tools.MsBuild.Generation" Version="3.3.4" />
  </ItemGroup>
  <ItemGroup>
    <None Update="reqnroll.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
  </ItemGroup>
</Project>

reqnroll.json — both formatters enabled, NUnitRetry with maxRetries: 2:

{
  "generator": { "generatorPlugins": [{ "name": "NUnitRetry" }] },
  "formatters": {
    "html": { "outputFilePath": "TestResults/test-report.html" },
    "message": { "outputFilePath": "TestResults/cucumber_messages.ndjson" }
  },
  "NRetrySettings": { "maxRetries": 2, "applyGlobally": true }
}

To reproduce: dotnet test --filter "AfterStep"

Observed vs expected

Observed	NullReferenceException in CucumberMessageFactory.ToTestStepStarted — scenario missing from HTML report
Expected	Scenario visible in HTML report with TimeoutException as the failure cause

Environment

.NET	9.0
Reqnroll.NUnit	3.3.4
NUnitRetry.ReqnrollPlugin	2.0.17
NUnit	4.5.1
NUnit3TestAdapter	6.2.0

Stack trace

System.NullReferenceException: Object reference not set to an instance of an object.
   at Reqnroll.Formatters.PayloadProcessing.Cucumber.CucumberMessageFactory.ToTestStepStarted(TestStepExecutionTracker testStepExecutionTracker)
   at Reqnroll.Formatters.ExecutionTracking.TestStepExecutionTracker.ProcessEvent(StepStartedEvent stepStartedEvent)
   at Reqnroll.Formatters.ExecutionTracking.TestCaseExecutionTracker.ProcessEvent(StepStartedEvent stepStartedEvent)
   at Reqnroll.Formatters.ExecutionTracking.PickleExecutionTracker.ProcessEvent(StepStartedEvent stepStartedEvent)
   at Reqnroll.Formatters.ExecutionTracking.FeatureExecutionTracker.ProcessEvent(StepStartedEvent stepStartedEvent)
   at Reqnroll.Formatters.PubSub.CucumberMessagePublisher.StepStartedEventHandler(StepStartedEvent stepStartedEvent)
   at Reqnroll.Formatters.PubSub.CucumberMessagePublisher.OnEventAsync(IExecutionEvent executionEvent)
   at Reqnroll.Infrastructure.TestExecutionEngine.OnAfterLastStepAsync()
   at Reqnroll.TestRunner.CollectScenarioErrorsAsync()

[clrudolphi]

On attempt 1 (retry), the Background's AfterStep passes. Reqnroll continues and executes the first scenario step, which fires AfterStep again — reusing the same testStepId (the testCase has no separate slot for it).

Ugh. Yeah, I see it now. The implementation relies on the first attempt to confirm the steps and hooks that will fire for that pickle.

This will require some thought.

[clrudolphi]

From what I can tell, this example surfaces an inconsistency between the behavior of Reqnroll and its documentation. The description of the 'stopOnFirstError' setting described in the Runtime section of the documentation indicates that the default behavior is that if a failure occurs the remaining steps will be attempted (but skipped). But the behavior seems to be that if an error occurs, no attempt is made to execute the remaining steps (and their hooks).

The test suite for the TestExecutionEngine demonstrates that stopped execution is the current behavior:

   [Fact]
   public async Task Should_not_execute_step_when_there_was_an_error_earlier()
   {
       var testExecutionEngine = CreateTestExecutionEngine();
       var stepDefMock = RegisterStepDefinition();

       _scenarioContext.ScenarioExecutionStatus = ScenarioExecutionStatus.TestError;

       await testExecutionEngine.StepAsync(StepDefinitionKeyword.Given, null, "foo", null, null);

       _methodBindingInvokerMock.Verify(i => i.InvokeBindingAsync(stepDefMock.Object, It.IsAny<IContextManager>(), It.IsAny<object[]>(), It.IsAny<ITestTracer>(), It.IsAny<DurationHolder>()), Times.Never());
   }

   [Fact]
   public async Task Should_not_execute_step_hooks_when_there_was_an_error_earlier()
   {
       var testExecutionEngine = CreateTestExecutionEngine();
       RegisterStepDefinition();

       _scenarioContext.ScenarioExecutionStatus = ScenarioExecutionStatus.TestError;

       var beforeStepMock = CreateHookMock(_beforeStepEvents);
       var afterStepMock = CreateHookMock(_afterStepEvents);

       await testExecutionEngine.StepAsync(StepDefinitionKeyword.Given, null, "foo", null, null);

       _methodBindingInvokerMock.Verify(i => i.InvokeBindingAsync(beforeStepMock.Object, _contextManagerStub.Object, null, _testTracerStub.Object, It.IsAny<DurationHolder>()), Times.Never());
       _methodBindingInvokerMock.Verify(i => i.InvokeBindingAsync(afterStepMock.Object, _contextManagerStub.Object, null, _testTracerStub.Object, It.IsAny<DurationHolder>()), Times.Never());
   }

And I'll admit up front: I authored those tests last year as we refactored the TestExecutionEngine - but now I can't remember whether we discussed the configuration setting and whether that setting should affect the execution of skipped step hooks.

@gasparnagy, What is the proper behavior?

[clrudolphi]

If we leave this current behavior, or correct it and still honor the 'stopOnFirstError' configuration setting, then Messages will be broken when a Retry mechanism is involved and a scenario fails mid-way through execution. The cause is that we only create/track TestCase/TestStep structures on the first pass.

I can think of two options to address this, neither of which is great:

• Fix the tracking of TestCase/TestStep by deferring their creation/publication until we have a known full-pass through the pickle. This would be known when ever we have a successful run, or the FeatureFinished event fires (we would be waiting for a pickle re-try in the mean-time). This would defer publication of all messages related to the pickle until either of these events occur. The resulting Message stream would be odd (by Cucumber conventions) because the sequence of messages in the stream would not match their timestamps. Technically this is still compliant, but "odd" and might confuse downstream message reporting packages.
• At first pickle start, use the DryRunInvoker, to exercise the entire pickle execution path in order to identify the TestCase and TestStep content. This has the (ugly) side-effect that plugin code would also run during this dry run, possibly having serious side-effects. There would also be the (likely slight) performance penalty of cycling through the pickle twice (once with the DryRunInvoker and one, or more, time(s) with the regular invoker).

I am leaning toward option 1 but am open to feedback and other ideas.