`--save-ace-state` produces a compressor that cannot compress

[froody]

Note this report was generated entirely by claude-opus5, but I am pretty confident in the analysis

Summary

zli train --save-ace-state with an SDDL profile emits a serialized compressor that
deserializes cleanly but fails on every input at compress time:

Code: Node parameter invalid
Message: Check `tokenSize.paramId == (-1)' failed
Node name: zl.convert_serial_to_struct#0
Transform ID: 5
  #0 EI_convert_serial_to_struct (src/openzl/codecs/conversion/encode_conversion_binding.c:175)

Training the same samples without the flag produces a working compressor.

The root cause is not the ACE state itself: attaching any graph-level local param —
a 4-byte dummy reproduces it exactly — to a graph that has just been base-overridden
demotes its parameterized nodes to their unparameterized base nodes, so
zl.convert_serial_to_struct#2 (carrying ZL_trlip_tokenSize) becomes #0 (carrying
nothing) and the mandatory parameter check rejects it.

This is easy to miss because zli compress silently falls back to generic compression
unless --strict is passed, so a broken model looks like a mild compression-ratio
regression rather than a hard failure. See "Secondary issue" below.

Environment

• OpenZL v0.2.3, commit 17308232ce42dcd7b53ecf2fe0db8ad5bc2b2223
• Built with CMake, -DCMAKE_BUILD_TYPE=Release, -DOPENZL_BUILD_CLI=ON
• macOS 26.x, arm64 M5 Max, Apple clang 21.0.0 (clang-2100.1.1.101)
• Reproduced with a clean build of that tag; no local patches

Reproduction

Fully self-contained — synthetic data, no external files.

1. SDDL (points.sddl) — a packed 3×Int32LE record array:

Point = {
  x: Int32LE
  y: Int32LE
  z: Int32LE
}

: Point[]

2. Samples — four 200k-point random walks (delta-compressible, like real point clouds):

import struct, random, os
random.seed(1)
os.makedirs("samples", exist_ok=True)
for s in range(4):
    x = y = z = 0
    buf = bytearray()
    for _ in range(200_000):
        x += random.randint(-40, 40)
        y += random.randint(-40, 40)
        z += random.randint(-8, 8)
        buf += struct.pack("<iii", x, y, z)
    open(f"samples/points_{s}.raw", "wb").write(bytes(buf))
open("input.raw", "wb").write(open("samples/points_0.raw", "rb").read())

3. Train and compress, with and without the flag:

# baseline — works
zli train --profile sddl --profile-arg points.sddl -t full-split \
    --use-all-samples -o no.zlc --force samples
zli compress --strict -c no.zlc input.raw -o out.zl -f

# with --save-ace-state — fails
zli train --profile sddl --profile-arg points.sddl -t full-split --save-ace-state \
    --use-all-samples -o yes.zlc --force samples
zli compress --strict -c yes.zlc input.raw -o out.zl -f

Observed

WITHOUT --save-ace-state:  model    4,594 B   compress OK (2,400,000 -> 423,721)
WITH    --save-ace-state:  model  192,567 B   compress FAILED
                                              zl.convert_serial_to_struct#0
                                              tokenSize.paramId == (-1)

Expected

--save-ace-state persists the ACE population for later training resumption. It should
not change the compressor's behaviour. Either the emitted compressor should compress
identically to the no-flag one, or — if -o output under this flag is intended to be
checkpoint-only and not a usable compressor — that should be documented and ideally
enforced, rather than silently emitting a compressor that fails on every input.

Scope

Only SDDL profiles are affected. Builtin profiles on the same samples are fine:

profile	model	compress
sddl (above)	192,567 B	FAILS
le-i32	83,139 B	OK
serial	69,940 B	OK

Consistent with the mechanism: the failure needs a trained graph containing a
parameterized node. The builtin graphs here apparently have none, so attaching graph
params is harmless for them.

How this was isolated

All variants below are the same samples, same input, v0.2.3. Rows 2–4 are diagnostic
patches to runReplacements, not proposed fixes.

variant	model	compress
1. no --save-ace-state	5,238 B	OK 393,666
2. --save-ace-state, stock	187,593 B	FAILS convert_serial_to_struct#0
3. + pass customGraphs/customNodes (from backendGraphId)	185,195 B	FAILS bitunpack#0 — error moves to the next node
4. + pass them from newGraphId instead	203,562 B	FAILS convert_serial_to_struct#0
5. + re-read local params post-override, then add state	191,940 B	FAILS convert_serial_to_struct#0
6. pure round-trip — read params back, re-apply, add nothing	4,941 B	OK 393,718
7. round-trip + 4-byte dummy copy-param, id 592	5,042 B	FAILS convert_serial_to_struct#0

Rows 6 and 7 are the result:

• Row 6 — ZL_Compressor_overrideGraphParams is not lossy. Reading the graph's
  parameters back after overrideBaseGraph and re-applying them verbatim yields a
  working compressor, and its output matches the no-flag baseline (393,718 vs 393,666).
• Row 7 — adding a single 4-byte copy-param to that same round-trip breaks it
  identically. So it is not the size of the ACE state (187 KB), not its contents, and
  not the custom graph/node lists. It is the presence of any graph-level local param.

Row 3 is also informative: supplying the custom node list fixes the first demoted node
and exposes the next one (zl.bitunpack#0, missing nbBits), which is what you would
expect if node parameterization is being shadowed rather than a single list being dropped.

Where it happens

tools/training/ace/ace_combination.cpp:43-56, in runReplacements:

for (const auto& [backendGraph, newGraphId] : newGraphIds) {
    auto backendGraphId = compressor.getGraph(backendGraph);
    auto localParams    = LocalParams(ZL_Compressor_Graph_getLocalParams(
            compressor.get(), backendGraphId.value()));
    compressor.unwrap(ZL_Compressor_overrideBaseGraph(
            compressor.get(), backendGraphId.value(), newGraphId));
    if (saveAceState) {
        auto gp = ZL_GraphParameters{ .localParams = localParams.get() };
        compressor.unwrap(
                ZL_Compressor_overrideGraphParams(
                        compressor.get(), backendGraphId.value(), &gp),
                "Graph replacement failed");
    }
}

tools/training/ace/ace.cpp:136-155 does the equivalent thing on the same graph (it
does pass customGraphs/customNodes through, but per row 3 that is not sufficient).

Supporting references:

• tools/training/ace/automated_compressor_explorer.h:150 — kAceStateParamId = 592
• tools/training/ace/ace_combination.cpp:315-330 — resume path; looks the checkpoint up
  by scanning zl.ace-prefixed graphs for copy-param 592 in their local params
• src/openzl/codecs/conversion/encode_conversion_binding.c:175 — the mandatory
  tokenSize check that rejects the demoted node
• src/openzl/compress/graphs/automated_compressor_explorer.c:33 —
  ZL_Compressor_buildACEGraphWithDefault2 parameterizes ZL_GRAPH_COMPRESS_GENERIC
  and names it zl.ace
• cpp/src/openzl/cpp/LocalParams.cpp:60-75 — addCopyParam correctly refreshes
  params_.copyParams.copyParams after push_back, so this is not a dangling-pointer
  bug in the wrapper

Suggested fixes

(a) Core. Graph-level localParams should not shadow node-level parameterization.
Either overrideGraphParams preserves the existing node parameterization, or graph params
merge with rather than replace what nodes already carry. This is the underlying defect and
touches parameter-resolution semantics, so it needs a maintainer's judgement.

(b) Trainer-side workaround. Don't store the checkpoint on the backend graph at all.
It is only ever read back at ace_combination.cpp:315-330; storing it as a global
parameter, or on a dedicated side graph that is not part of the compression path, and
updating that lookup to match would sidestep (a) entirely. This stays within
tools/training/ and requires no core changes.

Secondary issue: silent fallback hides this

zli compress falls back to generic compression when a trained compressor fails, unless
--strict is given:

without --strict:   no.zlc 423,721   yes.zlc 423,776     # 0.01% apart — looks fine
with    --strict:   no.zlc 423,721   yes.zlc FAILED

Because the fallback ratio happens to land close to the trained one, a completely broken
compressor is indistinguishable from a working one by output size alone. In our case this
masked the bug through a full training-and-benchmark cycle — the model failed on 703/703
real inputs while appearing to merely "compress slightly worse".

A warning on the fallback path (even at default verbosity, once per run) would make this
class of problem visible. Consider also whether a compressor that fails on every input
should be a hard error regardless of --strict.