PHPUnit's string comparison diff is not handled by ParaUnit

[Wirone]

As discussed in PHP-CS-Fixer#7187 it looks like ParaUnit does not handle how PHPUnit displays output for diff-based assertions.

In short: this is out of the scope of this repo. On our side we only assert that 2 strings are equal. This leads to PHPUnit\Framework\ExpectationFailedException which contains SebastianBergmann\Comparator\ComparisonFailure which has getDiff() and prints the diff between actual and expected code.

As a result, failures displayed in the ParaUnit's output miss important context.

It would be great if ParaUnit could catch whole PHPUnit's output and print it. Currently test output in PHP CS Fixer has this little flaw that you can't see what's the actual difference, you only see "Failed asserting that two strings are identical.", which requires running these tests locally and verifying what's the problem. This is not a big deal if there's the same failure for all PHP versions, but if it's only for specific version, it requires additional steps from developer (e.g. running through Docker). It would be just easier if there was diff in the CI output 🙂.

[Jean85]

This is probably due to the fact that PHPUnit splits the failure message in multiple pieces on the inside. Depending of which PHPUnit version is in use, Paraunit fetches the error output from different places:

• under PHPUnit 10, it uses the underlying event system:

  paraunit/src/Logs/TestHook/Failure.php

  Line 19 in 2d7e273

  $this->write(LogStatus::Failed, Test::fromPHPUnitTest($event->test()), $this->createMessageFromThrowable($event->throwable()));

• with PHPUnit 9, it uses the full message for the printer hook:

  paraunit/src/Parser/JSON/TestHook/Failure.php

  Line 14 in 31739b6

  $this->write(Log::STATUS_FAILURE, $test, $message);

Do you get the issue in both cases? I fear it's a bit hard to be fixed on PHPUnit 9, since we're already using the full message that is being passed, while on PHPUnit 10 we have the full event, which may contain what we need.

Could you dig a bit in PHPUnit's codebase? We need to understand where that diff is stored and how it's retrieved.

[Wirone]

We have PHP-CS-Fixer/PHP-CS-Fixer#6884 so we're yet to verify if problem remains on PHPUnit 10. On PHPUnit 9 I tried to utilise --stderr but diff is not visible too. @sebastianbergmann any hints?