Restore backward-compatible reader positioning in JsonSerializerPrimitives

[Copilot]

Developer Notes:
Addresses issue #2943

There is a different branch with a test project proving this PR's solution. Since that other branch includes this solution, to see the tests fail copy the test project into the dev branch. Those tests aren't part of this PR as they use intentionally mismatched versions of our libraries the compatibility of which is not actually a guarantee we make and the tests won't hold much value for future development.

AI Notes:
PR #2491 changed all Read* methods in JsonSerializerPrimitives to unconditionally advance the Utf8JsonReader after reading a value. This silently broke older compiled packages (e.g. Microsoft.IdentityModel.Protocols.OpenIdConnect <7.4.0) when paired with newer Microsoft.IdentityModel.Tokens — every other JSON property was skipped because the reader was advanced twice per property.

Restore backward-compatible reader positioning in JsonSerializerPrimitives

Description

Root cause: Old callers use while(reader.Read()) and manually advance to the value before calling Read*(read=false), expecting the reader to remain AT the value on return. The post-read advance added in #2491 caused while(reader.Read()) to skip over the next token.

Changes to JsonSerializerPrimitives.cs:

• Removed unconditional post-read reader.Read() from: ReadBoolean, ReadLong, ReadString, ReadStringAsBool, ReadStringOrNumberAsString, ReadStringAsObject, ReadNumber, and scalar branches of ReadPropertyValueAsObject (True/False/Null/default)
• ReadJsonElement keeps its advance — necessary so nested EndObject/EndArray tokens don't prematurely terminate the outer while(true) loop
• Null-value advance in ReadString, ReadStringOrNumberAsString, ReadStringAsObject, and ReadStrings is now conditional on the read parameter
• Internal array loops (ReadStrings(IList), ReadStrings(ICollection), ReadStringsSkipNulls, ReadArrayOfObjects) now use explicit reader.Read() after each element; EndArray advance is conditional on read
• Fixed double-advance bug in ReadArrayOfObjects: for StartObject/StartArray elements, ReadPropertyValueAsObject calls ReadJsonElement which already advances past the complex value — the explicit reader.Read() is now only applied for scalar types (Null, Number, String) where ReadPropertyValueAsObject leaves the reader AT the value

Compatibility matrix:

Caller pattern	read=false (old)	read=true (new)
while(reader.Read())	reader AT value → caller's Read() advances ✓	reader AT value → caller's Read() advances ✓
while(true) + else if (!reader.Read())	fallback Read() advances ✓	fallback Read() advances ✓

Tests added:

• ReaderPositionAfterRead_BackwardCompatibility — asserts reader stays AT value after Read*(read=false), and AT EndArray after ReadStrings(read=false)
• ReaderPositionAfterRead_NewCallingPattern — asserts the while(true) loop correctly reads all properties with read=true

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[kllysng]

Can these changes be validated with one of our higher level libraries which leverage IdentityModel? Even running an E2E test locally to gut check that these behavioral un-breaking changes don't lead to unintended consequences?

[JoshLozensky]

Can these changes be validated with one of our higher level libraries which leverage IdentityModel? Even running an E2E test locally to gut check that these behavioral un-breaking changes don't lead to unintended consequences?

Did this offline

[kllysng]

Thank you @JoshLozensky for your due diligence on validating the changes