🚀 Feature: Give more control on how diffs are rendered

[forty]

Feature Request Checklist

• I have read and agree to Mocha's Code of Conduct and Contributing Guidelines
• I have searched for related issues and issues with the faq label, but none matched my issue.
• I want to provide a PR to resolve this

Overview

I use some custom comparators with chai which allows me to do things such as:

assert.deepEqual({a: 1}, {a: new Interval(0, 10)});

Unfortuantly, when the assertion fails, I don't have any control on how the diff should be rendered, and the diff will expose the internal structure of the Interval class with is not super helpful. I would like to have some control on how the diff is rendered.

Suggested Solution

I thought of a simple solution which works rather well for simple use case, which would be to call toJSON (if available) before object canonicalization (and this way my Interval class could decide a bit how it want to be rendered for the diff).
It feels a fair think to do, and it's already done in the specific case of Buffer class https://github.com/mochajs/mocha/blob/master/lib/utils.js#L218

Alternatives

• Defining a special function "toMochaDiff" (of something similar) to avoid colliding with other use of toJSON
• Give user full control over diff rendering by adding hooks to stringifyDiffObjs and generateDiff in base reporter

Additional Info

No response

[JoshuaKGoldberg]

This is a really interesting feature idea, thanks for filing 😄. It seems like it'd be ueful!

Though, per #5027, we're trying not to take on any new features that don't have broad user demand. We're more trying to keep Mocha afloat than anything else. So unless other test runners already do this we'd probably avoid it.

Requesting info: what do other test runners do around this, if anything?

[forty]

@JoshuaKGoldberg Very good question, thanks for asking :) I had not investigated this at all, so I did some research.

As expected, Jest does a lot, with full battery included style. They have an internal diff module jest-diff, and before diffing, they format object with their own formater lib pretty-format, which allow plenty of customization with plugins and all the bells and whistles. The default behavior involves calling .toJSON, then trying not to if that did not work out well https://github.com/jestjs/jest/blob/e267aff33d105399f2134bad7c8f82285104f3da/packages/jest-matcher-utils/src/index.ts#L108
As you can see, this logic is in their "matcher", which would be a bit akin to saying chai should render their own diff (which in a way also makes sense to me)

Node native test seem also to delegate the diff to the assertion lib. (node assert module produce Errors whose message contains the colored diff directly). It doesn't seem to offer any customisation and doesn't call toJSON or anything

Jasmine seem to also have the diff formating pushed to their matcher (also builtin). They allow customisation of the pretty pretty before diffing https://github.com/jasmine/jasmine/blob/d06dce46141017f9f444caf2b3fe2655c49ad24b/src/core/matchers/DiffBuilder.js#L4C12-L4C26 (and "customObjectFormatters")

My conclusion of all this is that it somewhat more logical for the test runner not to handle diffs, and to push it more to the matcher/assertion level (I imagine assert.deepInclude diff should be computed differently than assert.deepEqual for example and the assertion lib is the one who should know what they mean) though the caveat is that the final rendering of the diff itself is definitely more of the reporter business (for example if you need to render things in HTML rather than in ANSI code - it's not clear to me how the other reporter handle this, maybe with ANSI code to HTML conversion, or maybe by transmitting an intermediate diff representation to the reporter).

For mocha, whose philosophy seem to be more "bring you own xxx" (assertions, mock, etc), I think it would make sense to allow the user to handle their own diff the way they want to. If we want to allow maximum flexibility (and also lowest effort ^^), we could simply have a renderError(error) function in the config, which if not defined, does the current behavior, and if defined, is called an return a string representation of the error, including the diff. As I said, it's not clear to me how HTML should be dealt with. Maybe with a boolean color parameter to renderError to let the rendering know we don't want to have ANSI code.

[jedwards1211]

@JoshuaKGoldberg I've had two serious problems with diffs in projects that use Sequelize:

If expect(sequelizeModelInstance)... fails, the process prints unable to find module 'pg-native' and crashes before it can format the error diff
If expect(somethingContainingASequelizeTransaction)... fails, the process runs OOM while trying to format the error diff

I'd like to solve this in a more systematic way than just turning off all error diffs or changing code all over my test files.

[jedwards1211]

@JoshuaKGoldberg Okay, I was able to work around my problems by monkey-patching canonicalize. So it's worth thinking about how to make that whole system officially pluggable

[JoshuaKGoldberg]

@JoshuaKGoldberg I've had two serious problems with diffs in projects that use Sequelize:

If expect(sequelizeModelInstance)... fails, the process prints unable to find module 'pg-native' and crashes before it can format the error diff If expect(somethingContainingASequelizeTransaction)... fails, the process runs OOM while trying to format the error diff

I'd like to solve this in a more systematic way than just turning off all error diffs or changing code all over my test files.

🤔 this specific area seems like a bug, not a feature request. Both error diff formatting crashes and OOMs should not happen at all. If you're seeing that with just Mocha built-ins then I'd request please filing a bug report with a full reproduction, so we can fix that issue: https://github.com/mochajs/mocha/issues/new?template=01-bug.yml.

---

Still, this issue was a fun dive into how Mocha renders diffs. https://mochajs.org/#diffs is relevant:

Mocha supports the err.expected and err.actual properties of any thrown AssertionErrors from an assertion library.
Mocha will attempt to display the difference between what was expected, and what the assertion actually saw.

N.b. it can actually be any Error, not just any AssertionError the way libraries such as chai normally use.

class CustomError extends Error {
  constructor(message) {
    super(message);
    this.name = "CustomError";
  }

  actual = "abc";
  expected = "bbc";
}

describe("example", () => {
  it("2 + 2", () => {
    throw new CustomError("Oh no!");
  });
});

  1) example
       2 + 2:

      CustomError: Oh no!
      + expected - actual

      -abc
      +bbc

I can see why philosophically Mocha should expose ways to control diffs. 👍 that this project's philosophy is generally "bring your own X".

But, we'd still need to see multiple real-world use cases for wanting to override the default provided diff rendering. Features come at a development + maintenance + reading-the-docs cost and we need to justify that.

[forty]

Just to clarify my use case, here is a demo

import { assert } from 'chai';
import * as chai from 'chai';
import eql from 'deep-eql';

class Interval {
    constructor(value, interval){
      this.value = value;
      this.interval = interval
    }
}
const comparator = (left, right) => {
    if(right && right instanceof Interval){
        if(typeof left !== 'number'){
            return false;
        } else {
            return Math.abs(right.value - left) < right.interval
        }
    } else {
        return null; // use default comparison
    }
};

chai.config.deepEqual = (actual, expected) => {
    return eql(expected, actual, {
        comparator
    });
};

describe('demo', () => {
  it('should fail', () => {
    assert.deepEqual({ amount: 103}, { amount: new Interval(100, 2) });
  })
})

which results in

  1) demo
       should fail:

      AssertionError: expected { amount: 103 } to deeply equal { Object (amount) }
      + expected - actual

       {
      -  "amount": 103
      +  "amount": {
      +    "interval": 2
      +    "value": 100
      +  }
       }

And I would prefer it to be (for example)

  1) demo
       should fail:

      AssertionError: expected { amount: 103 } to deeply equal { Object (amount) }
      + expected - actual

       {
      -  "amount": 103
      +  "amount": 100±2 
       }

I'm already able to control how the Interval class is rendered (via toString & co) but the issue it that it will be rendered the same way regardless of wether the values are equal or not (when they are equal, they should render to empty)