RFC 012 - Structured Assertion Messages#7750
Conversation
There was a problem hiding this comment.
Pull request overview
Adds RFC 011 to propose a standardized, structured multi-line format for MSTest assertion failure messages, aiming to improve readability and consistency across terminal/CI/IDE output.
Changes:
- Introduces an RFC defining a universal message skeleton (
Assertion failed.+ summary, optional user message, evidence block, optional call-site expression, stack trace). - Provides an assertion-by-assertion message catalog (including diff diagnostics, value rendering, truncation, and caller-expression handling).
- Documents open questions to resolve before implementation (user message placement, truncation thresholds, comparer display, etc.).
Show a summary per file
| File | Description |
|---|---|
| docs/RFCs/011-Structured-Assertion-Messages.md | New RFC proposing a structured assertion message format and a catalog of expected outputs across MSTest assertions. |
Copilot's findings
- Files reviewed: 1/1 changed files
- Comments generated: 3
- Replace ... placeholders in call-site expressions with <param> syntax
(e.g. <delta>, <expected>) for clarity
- Standardize all evidence labels to lowercase spaced format
(lowerBound → lower bound, ignoreCase → ignore case, etc.)
- Add actual value to IsExactInstanceOfType/IsNotExactInstanceOfType
- Make IsInstanceOfType and IsExactInstanceOfType wording parallel
('of type X (or derived)' vs 'exactly of type X')
- Fix IsInRange to use 5..10 range notation with expectations first
- Rename collection: to actual: in collection evidence blocks
- Add evidence blocks (collection values) to all CollectionAssert sections
that were missing them (Contains, DoesNotContain, IsSubsetOf, etc.)
- Add collection to HasCount and IsEmpty evidence blocks
- Fix Assert.That to use flat label format (no indented margin)
- Update backward compatibility: list Assert.Inconclusive as explicit
breaking change; note changes are part of MSTest v4
- Add 3 new unresolved questions from reviewer feedback:
#10 control character rendering strategy
#11 evidence block internal API
#12 collection multi-line rendering
- Expand unresolved question #3 (diff rendering) with inline highlight idea
There was a problem hiding this comment.
Copilot's findings
Comments suppressed due to low confidence (1)
docs/RFCs/011-Structured-Assertion-Messages.md:1110
- In the
StringAssert.EndsWithexample, the summary says "specified substring" but the evidence label usesexpected suffix:and earlierAssert.EndsWithexamples use "suffix" wording. For consistency/clarity, consider using "suffix" in the summary here as well.
Assertion failed. Expected string to end with the specified substring.
expected suffix: "world"
actual: "hello earth"
- Files reviewed: 1/1 changed files
- Comments generated: 2
- Remove CollectionAssert (legacy) and StringAssert (legacy) sections per scope decision - Add 'Scope: Legacy Assertion Classes' note explaining exclusion - Update Collection Diff Rules to use generic terms - Fix 'always starts' wording for Line 1 rule (Assert.Fail has no summary)
- User message placement: Option A (after summary, before values) - Truncation defaults: 1024 chars for strings, 32 elements for collections - Configuration via .runsettings and testconfig.json - Structured exception data: Expected/Actual string properties on AssertFailedException - Custom comparer display in evidence block when non-default - AreEqual<T> with T=string auto-detects and uses string-specific format - Control characters: C#-style escapes - Evidence block internal API: structured EvidenceBlock type (internal) - Collection multi-line rendering at 120-char threshold - Omit expected: line for implied-value assertions (IsTrue/IsFalse/IsNull/IsEmpty/etc.) - Omit IsNotNull evidence block entirely - Inline diff markers deferred to future enhancement - Substantially different threshold: 50% edit distance ratio - Replace Unresolved questions with Resolved Design Decisions table
Strings are always quoted with double quotes, so unquoted null is already unambiguous. The (null) convention was a legacy MSTest pattern not used by the runtime or Roslyn.
|
The RFC has matured significantly and I believe we've reached a solid foundation. I'd like to propose we move toward a final review pass. If there are remaining points of divergence, let's try to resolve them pragmatically — we don't need perfect consensus on every detail to ship a good v1. The structured format is designed to be additive: we can refine rendering rules, diff heuristics, and configuration options in future iterations based on real-world feedback without breaking the overall structure. Let's aim to get this approved and into implementation. Feedback from actual usage will be far more valuable than further theoretical debate at this point. cc @nohwnd |
- AreSame null cases: show expected/actual in evidence block instead of text-only summary, so no information is lost when one value is null - IsNotInstanceOfType/IsNotExactInstanceOfType: rename 'wrong type:' label to 'not expected type:' for consistency with the assertion semantics - Throws/ThrowsExactly wrong type: add 'actual exception:' line to show the caught exception details, helping diagnose initialization issues
Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
|
Thank you @nohwnd |
|
I know I'm late to the party, but wouldn't it have been better to put the user message after the failure detail (expected value, ...)? The only reason why I use a user message is to provide more details/context in case of an assert failure. For me, the failure itself should be upfront. This is no big deal though. I do not use user messages that frequently. Keep up the great work! |
|
@drieseng this one is tricky because I also use it for extra details but we have quite a few users using it at the main message. It can also push it to far away in case of big actual or assert (e.g. long string or long collection). We might end up changing that or making it configurable. I have seen some framework (I don't recall which language) where users can configure the shape of the assertion message and it felt interesting. Keep up with reporting suggestions and commenting, it's really important for me to have user feedback. |
Summary
Add RFC 012 proposing a standardized, structured multi-line format for assertion failure messages across all MSTest
Assert.*methods.CollectionAssertandStringAssertare considered legacy and are out of scope — their messages will be updated opportunistically but are not specified by this RFC.Motivation
The current single-line concatenated format is hard to scan in CI output, buries user messages, has no visual hierarchy, and breaks readability with long values. This RFC proposes a consistent multi-line structure that puts the most important information first.
What's in the RFC
Assertion failed.prefix, summary line, user message, labeled evidence block, call-site expression, stack traceRequesting review
Looking for team feedback on the overall structure and the open questions listed at the end of the RFC.