Skip to content

[v2] Console summary truncates failure messages at ~200 chars, hiding the actual diff #2822

@almirgomes

Description

@almirgomes

Summary

In Karate v2 the end-of-run "failed features" block on the console truncates both karate.fail(...) messages and match failure diffs at roughly 200 characters, appending .... The truncated part is usually the most useful part — it contains the actual values being compared.

This is a regression vs v1, which printed the full diff inline on the console.

Environment

  • Karate 2.0.6 (also reproduces on 2.0.3, 2.0.4, 2.0.5)
  • Java 21
  • CLI: java -cp karate.jar io.karatelabs.Main run <feature>
  • OS: macOS 15.4 (also reported on Linux CI)

Repro 1 — karate.fail() message truncated

Feature: JS failure message truncation

Scenario: karate.fail message truncated in console summary
  * def check =
    """
    function() {
      var expected = 'value-A';
      var actual = 'value-B';
      if (actual != expected) {
        karate.fail('Expected [' + expected + '] but got [' + actual + '] — this is the actual diff the user needs to see, but it gets cut off');
      }
    }
    """
  * check()

Console output (truncated):

failed features:
  /tmp/repro.feature
    - karate.fail message truncated in console summary
      /tmp/repro.feature:14
      * check()
      js failed:
==========
  Line: 5, Col: 4
  Code: karate.fail('Expected [' + expected + '] but got [' + actual + '] — this is the actual diff the user needs to see, but it gets cut off');
  Error: Ex...

The error message Error: Expected [value-A] but got [value-B] … is cut at Error: Ex... — exactly the information the failure is trying to convey.

Repro 2 — match diff truncated

Feature: match failure truncation

Scenario: match shows partial diff only
  * def actual = { name: 'Alice', age: 30, city: 'Berlin', role: 'engineer' }
  * def expected = { name: 'Bob', age: 30, city: 'Munich', role: 'engineer' }
  * match actual == expected

Console output (truncated):

match failed: EQUALS
  $ | not equal | match failed for names: [name, city] (MAP:MAP)
  {"name":"Alice","age":30,"city":"Berlin","role":"engineer"}
  {"name":"Bob","age":30,"city":"Munich","role":"...

The expected JSON is cut at "role":".... For larger maps / JSON payloads the truncation hides the entire right-hand side.

Root cause (from decompiled jar)

io.karatelabs.core.SuiteResult.printSummary() uses ScenarioResult.getFailureMessageForDisplay() — which returns a truncated form. The full message is available via ScenarioResult.getFailureMessage() and does end up in the HTML report, but not on the console.

There's no public config toggle I could find for this (--log-console=debug doesn't change the summary block). karate.configure has no logModifier / displayLength / verbose equivalent for this path.

Impact

CI logs (Jenkins, GHA) are the primary failure-diagnosis surface for most teams. When a test fails at 03:00 and someone gets paged, "scroll up to the HTML report" isn't workable — the full error needs to be in the console output.

Proposed fix (suggestion)

Either:

  1. Print getFailureMessage() (full) in printSummary() instead of getFailureMessageForDisplay(), OR
  2. Add a CLI flag --verbose-errors / property karate.verbose.errors=true that opts in to the full message, OR
  3. Make the truncation threshold configurable.

Option (1) matches v1 behaviour and is what most users will expect after upgrading.

Happy to send a PR if one of those approaches is acceptable.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions