feat(image): pixelmatch engine with cluster verdict and ignoredBoxes

Jon Keys · claude · Jon Keys · commit 3b723f0a4545 · 2026-07-24T18:47:03.000Z
Adds 'pixelmatch' as a third comparison engine backed by
io.github.t12y:pixelmatch 1.0.0 (a Java port of mapbox/pixelmatch v7:
per-pixel OKLab/HyAB perceptual color difference with built-in
anti-aliasing detection), usable standalone or in engine combos
alongside resemble/ssim.

Options (per-call / &lt;name&gt;.json): matchingThreshold (the per-pixel
OKLab threshold, 0..1 where black-white = 1.0; named to avoid clashing
with the failure 'threshold'), includeAA, checkerboard, and the same
ignoredBoxes ([{left, right, top, bottom}], bounds inclusive) the
resemble/ssim engines accept - masked pixels are excluded from every
count, so dynamic regions (transaction ids, timestamps) can be ignored.

Cluster verdict ('clusters': true or a tuning map; off by default;
suite-wide via image.clusters, per-call wins): a significance layer
that classifies diff clusters as real changes (interior mass surviving
erosion, or vivid thin changes on a flat baseline) vs rendering noise
(anti-aliasing halos, sub-pixel font shifts between browser versions).
When enabled, mismatchPercentage becomes the significant-diff
percentage so the existing failure threshold ignores noise; the raw
percentage, a human-readable summary, and per-region bounding boxes
ride on the diff() result and embed meta
(pixelmatchRawMismatchPercentage, pixelmatchSummary,
pixelmatchRegions).

The engine is count-only: the diff image is always resemble's, which
is what the HTML lightbox and its client-side live re-diff render.
clusters and ssim's windowSize are unrelated options; pixelmatch
ignores windowSize.

Co-Authored-By: Claude Fable 5 &lt;noreply@anthropic.com&gt;
diff --git a/karate-image/README.md b/karate-image/README.md
@@ -16,7 +16,7 @@ image.baselineDir = 'baselines';
 image.optionsDir  = 'baselines';   // where <name>.json tuning files live (defaults to baselineDir)
 image.threshold   = 0.02;          // max % mismatch tolerated
 image.report      = 'mismatched';  // attach diff images: 'all' | 'mismatched' | null
-image.engine      = 'resemble';    // 'resemble' | 'ssim' | 'resemble,ssim' | 'resemble|ssim'
+image.engine      = 'resemble';    // 'resemble' | 'ssim' | 'pixelmatch' | combos: 'resemble,ssim' | 'resemble|pixelmatch'
 ```
 
 Multiple engines: the **smallest** mismatch wins (pass if any engine is within threshold). The
@@ -66,6 +66,67 @@ per-name options at `<optionsDir>/<name>.json` — so options can live locally w
 live in, say, S3. Options precedence (low→high): suite/scenario config → `<name>.json` →
 per-call inline.
 
+## The pixelmatch engine
+
+The third engine, `pixelmatch`, is a Java port of
+[mapbox/pixelmatch](https://github.com/mapbox/pixelmatch) v7 (`io.github.t12y:pixelmatch`):
+per-pixel OKLab/HyAB perceptual color difference with built-in anti-aliasing detection.
+Options (per-call, `<name>.json`, all optional):
+
+- `matchingThreshold` (default `0.1`) — per-pixel OKLab HyAB color threshold, 0..1 where
+  black↔white = 1.0; smaller is more sensitive. (Named to avoid clashing with `threshold`,
+  which is the *failure* threshold everywhere in karate-image.)
+- `includeAA` (default `false`) — count anti-aliased pixels as differences.
+- `checkerboard` (default `true`) — blend semi-transparent pixels against a checkerboard
+  pattern (vs plain white) when comparing.
+- `ignoredBoxes` — the same `[{left, right, top, bottom}, …]` boxes (all bounds inclusive)
+  as the resemble/ssim engines; pixels inside are excluded from the comparison entirely,
+  including the cluster verdict's raw and significant counts. Use for dynamic regions
+  (transaction ids, timestamps).
+
+The pixelmatch engine is **count-only**: the diff image in reports is always produced by
+resemble (that is what the HTML lightbox and its live re-diff render), so pixelmatch's own
+diff-rendering options are not exposed. The pixelmatch percentages ride on the `diff()`
+result and the embed `meta` (`pixelmatchMismatchPercentage`, plus the raw/summary/region
+diagnostics when clusters are enabled).
+
+### Cluster verdict (`clusters`)
+
+Pixelmatch-only, **off by default**: a significance layer that separates rendering noise
+(anti-aliasing halos, sub-pixel shifts, font hinting changes between browser versions) from
+real regressions. Enable with `clusters: true` (defaults) or a tuning map:
+
+```js
+image.engine   = 'pixelmatch';
+image.clusters = true;             // suite-wide; or per-call / per-name in <name>.json:
+// { clusters: { coreRadius: 1, minCoreArea: 16, hardDelta: 2.5,
+//               flatness: 0.5, flatFraction: 0.5, minThinArea: 8 } }
+```
+
+With clusters on, the reported `mismatchPercentage` becomes the **significant-diff
+percentage** — pixels in regions classified as real changes (interior mass surviving
+erosion, or thin-but-vivid changes on a flat baseline) — and your existing failure
+`threshold` applies to that. Pure rendering noise reports ~0.0% and passes; a missing
+button reports its true area share. Because noise no longer inflates the number, you can
+tighten `threshold` aggressively (0.01% becomes practical). The raw percentage stays
+visible as `pixelmatchRawMismatchPercentage`, a human-readable `pixelmatchSummary`, and
+per-region boxes in `pixelmatchRegions` (`{x, y, width, height, area, coreArea, meanDelta,
+reason: 'CORE'|'THIN_VIVID'}`) — in both the `diff()` result and the embed `meta`.
+
+| Param | Default | Controls | Raising it | Lowering it |
+|---|---|---|---|---|
+| `coreRadius` | 1 | Max thickness (2r+1 px) treated as potentially noise | Thicker real changes must rely on the safety net; risk of missing small solid changes rises. Use 2 for 2x/retina captures. | (min 1) Thinner noise survives erosion and is counted as significant; false positives return. |
+| `minCoreArea` | 16 | Smallest interior "core" that counts as significant | Small solid changes (icons, single glyphs) may be ignored → false negatives. | Residual noise that survives erosion gets counted → false positives. |
+| `hardDelta` | 2.5 | How vivid a thin change must be for the safety net (× `matchingThreshold`) | Real thin changes (underlines, dividers, recolored text) missed → false negatives. | Strong-ish rendering noise gets rescued as significant → false positives. |
+| `flatness` | 0.5 | How flat the baseline must be under a thin change (× `matchingThreshold`) | More of the baseline qualifies as "flat" → safety net fires more often. | Stricter flatness → thin changes near existing detail are ignored. |
+| `flatFraction` | 0.5 | Share of a thin cluster's pixels that must sit on flat baseline | Thin changes overlapping existing edges (e.g. text recolor) missed. | Noise straddling a flat area may be counted. |
+| `minThinArea` | 8 | Smallest thin cluster the safety net will consider | Small thin marks (a text caret is ~2x16 px) ignored → false negatives. | Isolated vivid speckles (dead pixels, dithering) counted → false positives. |
+
+Notes: `clusters` cannot be combined with ssim's `windowSize` option on the same call when
+both engines run (pixelmatch ignores `windowSize`; it belongs to ssim). The lightbox does
+not yet draw the significant-region boxes on the diff — the coordinates ride on the embed
+`meta` (`pixelmatchRegions`) for when it does.
+
 A runnable, readable walkthrough lives in
 [`src/test/resources/demo/visual-demo.feature`](src/test/resources/demo/visual-demo.feature)
 (establish → match → catch regression → rebase → per-name options); `VisualDemoTest` asserts
diff --git a/karate-image/pom.xml b/karate-image/pom.xml
@@ -39,6 +39,11 @@
             <artifactId>ssim</artifactId>
             <version>1.0.0</version>
         </dependency>
+        <dependency>
+            <groupId>io.github.t12y</groupId>
+            <artifactId>pixelmatch</artifactId>
+            <version>1.0.0</version>
+        </dependency>
     </dependencies>
 
     <profiles>
diff --git a/karate-image/src/main/java/io/karatelabs/ext/image/ImageApi.java b/karate-image/src/main/java/io/karatelabs/ext/image/ImageApi.java
@@ -179,6 +179,18 @@ private Map<String, Object> result(Map<String, Object> r, String name, Map<Strin
         if (r.containsKey(ImageComparison.SSIM_MISMATCH_PERCENT)) {
             out.put("ssimMismatchPercentage", num(r.get(ImageComparison.SSIM_MISMATCH_PERCENT)));
         }
+        if (r.containsKey(ImageComparison.PIXELMATCH_MISMATCH_PERCENT)) {
+            out.put("pixelmatchMismatchPercentage", num(r.get(ImageComparison.PIXELMATCH_MISMATCH_PERCENT)));
+        }
+        if (r.containsKey(ImageComparison.PIXELMATCH_RAW_MISMATCH_PERCENT)) {
+            out.put("pixelmatchRawMismatchPercentage", num(r.get(ImageComparison.PIXELMATCH_RAW_MISMATCH_PERCENT)));
+        }
+        if (r.containsKey(ImageComparison.PIXELMATCH_SUMMARY)) {
+            out.put("pixelmatchSummary", str(r.get(ImageComparison.PIXELMATCH_SUMMARY)));
+        }
+        if (r.containsKey(ImageComparison.PIXELMATCH_REGIONS)) {
+            out.put("pixelmatchRegions", r.get(ImageComparison.PIXELMATCH_REGIONS));
+        }
         out.put("mismatch", mismatch);
         out.put("scaleMismatch", scaleMismatch);
         out.put("threshold", threshold);
@@ -237,6 +249,19 @@ private Map<String, Object> embed(Map<String, Object> r, String name, Map<String
         if (r.containsKey(ImageComparison.SSIM_MISMATCH_PERCENT)) {
             meta.put("ssimMismatchPercentage", num(r.get(ImageComparison.SSIM_MISMATCH_PERCENT)));
         }
+        if (r.containsKey(ImageComparison.PIXELMATCH_MISMATCH_PERCENT)) {
+            meta.put("pixelmatchMismatchPercentage", num(r.get(ImageComparison.PIXELMATCH_MISMATCH_PERCENT)));
+        }
+        if (r.containsKey(ImageComparison.PIXELMATCH_RAW_MISMATCH_PERCENT)) {
+            meta.put("pixelmatchRawMismatchPercentage", num(r.get(ImageComparison.PIXELMATCH_RAW_MISMATCH_PERCENT)));
+        }
+        if (r.containsKey(ImageComparison.PIXELMATCH_SUMMARY)) {
+            meta.put("pixelmatchSummary", str(r.get(ImageComparison.PIXELMATCH_SUMMARY)));
+        }
+        // significant-region bounding boxes; TODO draw them on the diff in the lightbox
+        if (r.containsKey(ImageComparison.PIXELMATCH_REGIONS)) {
+            meta.put("pixelmatchRegions", r.get(ImageComparison.PIXELMATCH_REGIONS));
+        }
         meta.put("threshold", num(r.get("failureThreshold")));
         meta.put("defaultThreshold", num(r.get("defaultFailureThreshold")));
         if (r.get("engine") != null) {
@@ -376,6 +401,9 @@ private Map<String, Object> defaultOptions() {
         if (config.containsKey("allowScaling")) {
             d.put("allowScaling", config.get("allowScaling"));
         }
+        if (config.containsKey("clusters")) {
+            d.put("clusters", config.get("clusters"));
+        }
         return d;
     }
 
diff --git a/karate-image/src/main/java/io/karatelabs/ext/image/ImageComparison.java b/karate-image/src/main/java/io/karatelabs/ext/image/ImageComparison.java
@@ -23,6 +23,9 @@
  */
 package io.karatelabs.ext.image;
 
+import io.github.t12y.pixelmatch.ClusterAnalysis;
+import io.github.t12y.pixelmatch.ClusterOptions;
+import io.github.t12y.pixelmatch.Pixelmatch;
 import io.github.t12y.resemble.ErrorType;
 import io.github.t12y.resemble.Resemble;
 import io.github.t12y.resemble.Result;
@@ -43,9 +46,9 @@
 
 /**
  * Server-side pixel-diff engine for the image-comparison ext. Reads a baseline +
- * latest PNG/JPEG, runs the resemble and/or ssim engine over the unpacked RGBA
- * pixels, and returns a result map carrying the mismatch percentage and (when a
- * report is requested) the baseline / latest / diff {@link BufferedImage}s.
+ * latest PNG/JPEG, runs the resemble, ssim and/or pixelmatch engine over the
+ * unpacked RGBA pixels, and returns a result map carrying the mismatch percentage
+ * and (when a report is requested) the baseline / latest / diff {@link BufferedImage}s.
  * <p>
  * The pixel-comparison logic was contributed by jkeys089 (github.com/jkeys089),
  * who also authors the underlying {@code io.github.t12y:resemble} / {@code :ssim}
@@ -56,11 +59,18 @@ public class ImageComparison {
 
     public static final String RESEMBLE = "resemble";
     public static final String SSIM = "ssim";
+    public static final String PIXELMATCH = "pixelmatch";
     public static final String BASELINE_IMAGE = "baselineImage";
     public static final String LATEST_IMAGE = "latestImage";
     public static final String DIFF_IMAGE = "diffImage";
     public static final String RESEMBLE_MISMATCH_PERCENT = "resembleMismatchPercentage";
     public static final String SSIM_MISMATCH_PERCENT = "ssimMismatchPercentage";
+    public static final String PIXELMATCH_MISMATCH_PERCENT = "pixelmatchMismatchPercentage";
+    // present only when the cluster verdict is enabled: the raw (unfiltered) percentage
+    // stays visible as a diagnostic alongside the significant one that drives pass/fail
+    public static final String PIXELMATCH_RAW_MISMATCH_PERCENT = "pixelmatchRawMismatchPercentage";
+    public static final String PIXELMATCH_SUMMARY = "pixelmatchSummary";
+    public static final String PIXELMATCH_REGIONS = "pixelmatchRegions";
 
     private static final String[] IGNORED_BOX_KEYS = new String[]{"left", "right", "top", "bottom"};
     private static final String[] IGNORED_COLOR_KEYS = new String[]{"r", "g", "b"};
@@ -79,6 +89,7 @@ public class ImageComparison {
     private boolean includeDiffImageOnSuccess;
     private boolean includeDiffImageOnFailure;
     private String[] engines;
+    private Object clustersConfig;
     private final Map<String, Object> options;
     private final Map<String, Object> result;
 
@@ -172,6 +183,9 @@ private void configure(Map<String, Object> defaultOptions) {
             engines = engineConfig.split(",");
         }
 
+        // per-call/name.json options win over the suite-wide default (same precedence as engine)
+        clustersConfig = options.containsKey("clusters") ? options.get("clusters") : defaultOptions.get("clusters");
+
         String reportFormat = asString(defaultOptions.get("report"));
         if ("all".equalsIgnoreCase(reportFormat)) {
             includeDiffImageOnSuccess = true;
@@ -209,6 +223,9 @@ public static Map<String, Object> run(byte[] baselineImg, byte[] latestImg, Map<
                 case SSIM:
                     currentMismatchPercentage = imageComparison.execSSIM();
                     break;
+                case PIXELMATCH:
+                    currentMismatchPercentage = imageComparison.execPixelmatch();
+                    break;
                 default:
                     logger.error("skipping unsupported image comparison engine: {}", engine);
                     continue;
@@ -272,6 +289,78 @@ private double execSSIM() {
         return mismatchPercentage;
     }
 
+    private double execPixelmatch() {
+        io.github.t12y.pixelmatch.Options opts = pixelmatchOptions();
+
+        // pixelmatch is count-only: the diff image is always resemble's, which is what the
+        // HTML report lightbox (and its client-side live re-diff) renders. checkMismatch
+        // runs the resemble engine whenever a diff image is needed and none exists yet.
+        double mismatchPercentage;
+        if (opts.clusters != null) {
+            // cluster verdict: the significant-diff percentage drives pass/fail; rendering
+            // noise (AA halos, sub-pixel shifts) reports as ~0. Raw numbers stay visible.
+            ClusterAnalysis analysis = Pixelmatch.analyze(baselinePixels, latestPixels, null, width, height, opts);
+            mismatchPercentage = analysis.significantPercentage;
+            result.put(PIXELMATCH_RAW_MISMATCH_PERCENT, analysis.rawPercentage);
+            result.put(PIXELMATCH_SUMMARY, analysis.summary());
+            result.put(PIXELMATCH_REGIONS, regionMaps(analysis));
+        } else {
+            int mismatched = Pixelmatch.pixelmatch(baselinePixels, latestPixels, null, width, height, opts);
+            mismatchPercentage = 100.0 * mismatched / (width * height);
+        }
+
+        result.put(PIXELMATCH_MISMATCH_PERCENT, mismatchPercentage);
+        return mismatchPercentage;
+    }
+
+    private static List<Map<String, Object>> regionMaps(ClusterAnalysis analysis) {
+        List<Map<String, Object>> regions = new ArrayList<>();
+        for (ClusterAnalysis.Region region : analysis.significantRegions) {
+            Map<String, Object> m = new HashMap<>();
+            m.put("x", region.x);
+            m.put("y", region.y);
+            m.put("width", region.width);
+            m.put("height", region.height);
+            m.put("area", region.area);
+            m.put("coreArea", region.coreArea);
+            m.put("meanDelta", region.meanDelta);
+            m.put("reason", region.reason.name());
+            regions.add(m);
+        }
+        return regions;
+    }
+
+    private io.github.t12y.pixelmatch.Options pixelmatchOptions() {
+        io.github.t12y.pixelmatch.Options opts = io.github.t12y.pixelmatch.Options.defaults();
+
+        // 'threshold' already means the failure threshold throughout karate-image,
+        // so the per-pixel OKLab HyAB matching threshold gets its own key.
+        // (pixelmatch's diff-rendering options are not exposed: it never draws here)
+        opts.threshold = getDouble("matchingThreshold", opts.threshold);
+        opts.includeAA = getBool("includeAA", opts.includeAA);
+        opts.checkerboard = getBool("checkerboard", opts.checkerboard);
+        opts.ignoredBoxes = getIgnoredBoxes();
+        opts.clusters = clusterOptions();
+
+        return opts;
+    }
+
+    /** {@code clusters: true} enables the verdict with defaults; a map tunes it; absent/false/null disables. */
+    private ClusterOptions clusterOptions() {
+        if (clustersConfig instanceof Map) {
+            Map cfg = (Map) clustersConfig;
+            ClusterOptions clusters = ClusterOptions.defaults();
+            clusters.coreRadius = toInt(cfg.get("coreRadius"), clusters.coreRadius);
+            clusters.minCoreArea = toInt(cfg.get("minCoreArea"), clusters.minCoreArea);
+            clusters.hardDelta = toDouble(cfg.get("hardDelta"), clusters.hardDelta);
+            clusters.flatness = toDouble(cfg.get("flatness"), clusters.flatness);
+            clusters.flatFraction = toDouble(cfg.get("flatFraction"), clusters.flatFraction);
+            clusters.minThinArea = toInt(cfg.get("minThinArea"), clusters.minThinArea);
+            return clusters;
+        }
+        return toBool(clustersConfig) ? ClusterOptions.defaults() : null;
+    }
+
     private io.github.t12y.resemble.Options resembleOptions() {
         io.github.t12y.resemble.Options opts;
 
@@ -373,11 +462,14 @@ private static double toDouble(Object obj, double defaultValue) {
     }
 
     private int getInt(String name, int defaultValue) {
-        Object val = options.get(name);
-        if (!(val instanceof Number)) {
+        return toInt(options.get(name), defaultValue);
+    }
+
+    private static int toInt(Object obj, int defaultValue) {
+        if (!(obj instanceof Number)) {
             return defaultValue;
         }
-        return ((Number) val).intValue();
+        return ((Number) obj).intValue();
     }
 
     private String getString(String name, String defaultValue) {
diff --git a/karate-image/src/test/java/io/karatelabs/ext/image/ImageComparisonTest.java b/karate-image/src/test/java/io/karatelabs/ext/image/ImageComparisonTest.java
@@ -69,7 +69,7 @@ private static double round(double d) {
     }
 
     @ParameterizedTest
-    @ValueSource(strings = {"resemble", "ssim"})
+    @ValueSource(strings = {"resemble", "ssim", "pixelmatch"})
     void testIgnoredBoxes(String engine) {
         Map<String, Integer> box = new HashMap<>();
         box.put("left", 1);
diff --git a/karate-image/src/test/java/io/karatelabs/ext/image/ImageExtE2ETest.java b/karate-image/src/test/java/io/karatelabs/ext/image/ImageExtE2ETest.java
@@ -53,6 +53,55 @@ class ImageExtE2ETest {
     @TempDir
     Path tempDir;
 
+    @Test
+    void reportModeGatesEmbedsByVerdict() throws Exception {
+        Files.writeString(tempDir.resolve("karate-boot.js"), """
+            const image = boot.ext('image');
+            image.report = 'mismatched';
+            image.threshold = 0.1;
+            // no ssim: its default 11x11 window degenerates to ~0% difference on the
+            // tiny 3x3 fixtures here, short-circuiting the fallback chain
+            image.engine = 'resemble|pixelmatch';
+            image.clusters = true;
+            """);
+
+        Path feature = tempDir.resolve("report-mode.feature");
+        Files.writeString(feature, """
+            Feature: report mode gates embeds
+
+            Scenario: mismatched report omits passing embeds entirely
+            * def blue = new Uint8Array(java.util.Base64.getDecoder().decode('%s'))
+            * def green = new Uint8Array(java.util.Base64.getDecoder().decode('%s'))
+
+            # passing comparison: no embed, hence nothing for the HTML/PDF reports
+            * def passing = image.diff({ baseline: blue, latest: blue })
+            * match passing.pass == true
+            * match passing.embed == null
+
+            # failing comparison (clusters off per-call: a single changed pixel is below
+            # minThinArea, so the suite-wide verdict would classify it as noise and pass)
+            * def failing = image.diff({ baseline: blue, latest: green, clusters: false })
+            * match failing.pass == false
+            * match failing.embed == '#notnull'
+
+            # report=all embeds passing comparisons too, diff image included
+            * image.report = 'all'
+            * def passingAll = image.diff({ baseline: blue, latest: blue })
+            * match passingAll.pass == true
+            * match passingAll.embed == '#notnull'
+            * match passingAll.embed.parts == '#[3]'
+            """.formatted(BLUE, GREEN));
+
+        SuiteResult result = Runner.path(feature.toString())
+                .workingDir(tempDir)
+                .outputDir(tempDir.resolve("reports"))
+                .outputConsoleSummary(false)
+                .parallel(1);
+
+        assertEquals(1, result.getScenarioPassedCount(), "report-mode scenario should pass");
+        assertEquals(0, result.getScenarioFailedCount());
+    }
+
     @Test
     void establishMatchAndMismatch() throws Exception {
         Path baselineDir = tempDir.resolve("baselines");
diff --git a/karate-image/src/test/java/io/karatelabs/ext/image/PixelmatchEngineTest.java b/karate-image/src/test/java/io/karatelabs/ext/image/PixelmatchEngineTest.java

Original file line number	Diff line number	Diff line change
`@@ -69,7 +69,7 @@ private static double round(double d) {`
`69`	`69`	`}`
`70`	`70`
`71`	`71`	`@ParameterizedTest`
`72`		`- @ValueSource(strings = {"resemble", "ssim"})`
	`72`	`+ @ValueSource(strings = {"resemble", "ssim", "pixelmatch"})`
`73`	`73`	`void testIgnoredBoxes(String engine) {`
`74`	`74`	`Map<String, Integer> box = new HashMap<>();`
`75`	`75`	`box.put("left", 1);`