foreach loop reports values of next iteration to debugger

[weltkante]

Description

I've got a scenario where the debugger consistently reports wrong values for the loop variable, accessing the foreach loop value of the next iteration. The runtime seems to lag behind one iteration, while the debugger shows value of the variable it would have in the next iteration. This makes debugging confusing and very hard if you happen to trigger it.

The problem happens every run, I've reduced the code triggering the issue below (some code comparing assemblies tripped into this bug, originally using Mono.Cecil, but moving to plain reflection observes the same behavior given the same structure of code)

Reproduction Steps

using System.Diagnostics;
using System.Reflection;

await BugRepro(typeof(System.Collections.Immutable.ImmutableDictionary<,>.Enumerator), null);

// method must be async for the bug to trigger
static async Task BugRepro(Type type, Type? comparingType)
{
    const BindingFlags f = BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance;

    foreach (var field in type.GetFields(f))
    {
        // repeating the assert from below is necessary to trigger the bug
        if (comparingType?.GetFields(f).SingleOrDefault(x => x.Name == field.Name) is null)
            continue;

        // original code did more stuff but could remove it
    }

    foreach (var field in type.GetFields(f))
    {
        // inspecting debugger variables, this should be false (can even set a breakpoint and step through)
        var fieldContainsGenericParameter = field.FieldType.ContainsGenericParameters;
        if (fieldContainsGenericParameter)
        {
            // inspecting debugger variables, this assert should have triggered
            Debug.Assert(field.Name != "_enumeratingBuilderVersion");

            // inspect debugger variables mentioned above
            Debugger.Break();

            // repeating the assert is necessary to trigger the bug
            Debug.Assert(comparingType?.GetFields(f).SingleOrDefault(x => x.Name == field.Name) is null);
        }
    }
}

Expected behavior

the foreach loop variable shown in the debugger should match its actual value the runtime is using

Actual behavior

the debugger shows the foreach loop variable of the next iteration and bases all its displayed information off of that

in particular the debugger shows us being in the if-block for the field _enumeratingBuilderVersion which has type Int32 for which ContainsGenericParameters is false, yet the runtime assigns fieldContainsGenericParameter true (which it is for the previous field in the iteration)

Regression?

no, repros in Desktop Framework and VS 2019 as well as .NET 7 and 8 under VS 2022

Known Workarounds

none known, once the code is structured to trigger the issue, the debugger seems to consistently report wrong values

Configuration

x64 Windows 10 22H2 19045.3448

.NET Framework 4.8.9181.0
.NET 7.0.11
.NET 8.0.100 rc.1.23455.8

VS 2019 16.11.30
VS 2022 17.8.0 Preview 2.0

Other information

No response

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I've got a scenario where the JIT generates bad code, accessing the foreach loop value of a previous iteration. The VS debugger "sees" the right value of the variable when inspecting but the runtime acts as if it had the value of the previous iteration (when accessing and storing referenced fields in variables or taking branches).

The problem happens every run and is deterministic, but I can't share the program as-is nor does it seem easy to reproduce in a blank project. Is there anything I can do to diagnose this further or is the only way to proceed trying to build a repro that I can share?

Reproduction Steps

not available publically, first want to clarify what the best way to proceed is

Expected behavior

the foreach loop variable should be updated on each iteration

Actual behavior

the foreach loop variable acts as if it had the value of a previous iteration

Regression?

No response

Known Workarounds

No response

Configuration

dotnet 7.0.11 x64 windows 10

Other information

No response

Author:	weltkante
Assignees:	-
Labels:	area-CodeGen-coreclr, untriaged
Milestone:	-

[AndyAyersMS]

fyi @dotnet/jit-contrib

[JulieLeeMSFT]

@BruceForstall, PTAL if we need to fix it in .NET 8.

[BruceForstall]

@weltkante Thank you for the bug report.

I'm having trouble reproducing this problem locally. On the command-line, I did dotnet new console, replaced the template Program.cs with your test case above, and ran dotnet run -c Debug and dotnet run -c Release for both .NET 7 and .NET 8 (RC1). Based on your report, I expected to see Debug.Assert(field.Name != "_enumeratingBuilderVersion"); assert.

Does the problem reproduce for you on the command-line? Or only while running under the debugger?

Have you tried .NET 8 to see if it continues to reproduce there?

[weltkante]

Had time to make more tests, this does repro under .NET 8 and FWIW also under Desktop Framework with VS 2019, so no regression but a longstanding bug. I'm not sure how you tested it under the command line since the repro is explicitly built to break into the debugger to inspect the variables, so yes, you need a debugger to observe the difference.

I tried to make it observable without a debugger (and failed) so I think I understand this a little better now. I also tried single stepping through the repro code and its not that the runtime is behind one iteration but the debugger somehow shows the loop variable for the next iteration very consistently, but only for this very specific structured code, if its structure is simplified further I see the "right" values in the debugger.

When I originally encountered the issue the program was in the middle of comparing assemblies so I couldn't see the whole picture and probably trusted the debugger too much.

The question is, I guess, is the runtime responsible to extract and report the loop iteration variable to the debugger? Then this particular structure of the program is still treated incorrectly, maybe optimized in a way that makes the variable impossible to recover? If its VS' responsibility to figure out the loop variable without runtime help I probably need to re-report the issue there.

Let me know how to proceed.

[BruceForstall]

Ok, I'm able to reproduce this in the debugger. What I see is that single-stepping through the first foreach (var field in type.GetFields(f)) loop, I see the value of field change every iteration (there are 4 iterations), when I have field in the Watch Window, and the last iteration the name is "_enumeratingBuilderVersion". Single-stepping through the second loop, the value of field never changes in the Watch Window (it is always "_enumeratingBuilderVersion").

However, if I change the second loop to be foreach (var field2 in type.GetFields(f)) (that is, change field to a new field2) then inspecting / viewing field2 works as expected.

This also seems to be specific to async, which generates a lot of complicated code.

@tommcdon This looks like a debugger issue. Can you investigate?

Tagging subscribers to this area: @tommcdon
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I've got a scenario where the debugger consistently reports wrong values for the loop variable, accessing the foreach loop value of the next iteration. The runtime seems to lag behind one iteration, while the debugger shows value of the variable it would have in the next iteration. This makes debugging confusing and very hard if you happen to trigger it.

The problem happens every run, I've reduced the code triggering the issue below (some code comparing assemblies tripped into this bug, originally using Mono.Cecil, but moving to plain reflection observes the same behavior given the same structure of code)

Reproduction Steps

using System.Diagnostics;
using System.Reflection;

await BugRepro(typeof(System.Collections.Immutable.ImmutableDictionary<,>.Enumerator), null);

// method must be async for the bug to trigger
static async Task BugRepro(Type type, Type? comparingType)
{
    const BindingFlags f = BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance;

    foreach (var field in type.GetFields(f))
    {
        // repeating the assert from below is necessary to trigger the bug
        if (comparingType?.GetFields(f).SingleOrDefault(x => x.Name == field.Name) is null)
            continue;

        // original code did more stuff but could remove it
    }

    foreach (var field in type.GetFields(f))
    {
        // inspecting debugger variables, this should be false (can even set a breakpoint and step through)
        var fieldContainsGenericParameter = field.FieldType.ContainsGenericParameters;
        if (fieldContainsGenericParameter)
        {
            // inspecting debugger variables, this assert should have triggered
            Debug.Assert(field.Name != "_enumeratingBuilderVersion");

            // inspect debugger variables mentioned above
            Debugger.Break();

            // repeating the assert is necessary to trigger the bug
            Debug.Assert(comparingType?.GetFields(f).SingleOrDefault(x => x.Name == field.Name) is null);
        }
    }
}

Expected behavior

the foreach loop variable shown in the debugger should match its actual value the runtime is using

Actual behavior

the debugger shows the foreach loop variable of the next iteration and bases all its displayed information off of that

in particular the debugger shows us being in the if-block for the field _enumeratingBuilderVersion which has type Int32 for which ContainsGenericParameters is false, yet the runtime assigns fieldContainsGenericParameter true (which it is for the previous field in the iteration)

Regression?

no, repros in Desktop Framework and VS 2019 as well as .NET 7 and 8 under VS 2022

Known Workarounds

none known, once the code is structured to trigger the issue, the debugger seems to consistently report wrong values

Configuration

x64 Windows 10 22H2 19045.3448

.NET Framework 4.8.9181.0
.NET 7.0.11
.NET 8.0.100 rc.1.23455.8

VS 2019 16.11.30
VS 2022 17.8.0 Preview 2.0

Other information

No response

Author:	weltkante
Assignees:	BruceForstall
Labels:	area-Diagnostics-coreclr
Milestone:	8.0.0