spiral-input PHercParis4: `outer_shell/meta.json` stores the grid step (~20) in `scale` instead of its reciprocal, and `vc_flatten` OOMs on it trying to allocate 1.3 TB

[Nicodol]

Summary

On the published spiral-input pack for PHerc. Paris 4
(hf://buckets/scrollprize/datasets/spiral/PHercParis4, outer_shell/meta.json unchanged in
the bucket since 4 June 2026), the shell declares scale: [19.997318, 19.996687]. Every other
tifxyz in the pack declares 0.05 or 0.08, i.e. 1/step. The shell's grid is an ordinary
20-voxel grid, so the field should read [0.05, 0.05]; as written it is 400x off.

vc_flatten from the current release sizes its output grid as UV_range * scale, so on this
file it asks for a 361753 x 303196 grid and dies:

$ vc_flatten -i outer_shell -o shell_flat
ABF++: UV bounds: [-7914.3, -14979.8] to [10175.7, 182.471]
ABF++: Creating output grid 361753 x 303196 (input scale=19.9973x19.9967, UV range=18090x15162.2)
OpenCV(4.13.0) Error: Insufficient memory (Failed to allocate 1316184751056 bytes) in OutOfMemoryError

Copy the same three TIFFs, change only scale to [0.05, 0.05], and the same command
succeeds:

$ vc_flatten -i shell_fixed -o shell_flat_fixed
ABF++: UV bounds: [-7914.3, -14979.8] to [10175.7, 182.471]
ABF++: Creating output grid 906 x 760 (input scale=0.05x0.05, UV range=18090x15162.2)
ABF++: Rasterized 359175 / 688560 points (52.1632%)

Flattened surface:
  Grid size: 906 x 760
  Scale: 0.05, 0.05
  Valid points: 359175
Done!

Identical geometry, identical UV bounds, one field changed. Binary is
VC3D-07bb743-2026-08-08-win64 from the latest release, i.e. built from the same commit as
main at the time of writing.

To be precise about what is and is not affected: the spiral fitter is fine. fit_spiral
reads only shell_patch.valid_zyxs from the shell, so fits are unaffected. What breaks are
the tools that apply the documented meaning of scale to this directory.

Related, and found before writing this: #1319 reports the same field written wrong by
vc_obj2tifxyz, and the same vc_flatten sizing mechanism, in the opposite direction (its
output collapses to 6x4 instead of exploding). I have kept the overlapping background short
below and pointed at that issue rather than restating it.

Filing separately rather than commenting there, because the two close on different actions:
#1319 closes when vc_obj2tifxyz is fixed, and that day this file is still byte-for-byte
what it is today, since no code change reaches a file in the bucket. If you would rather
track both in one place, I am happy to fold this into #1319 instead.

Motivation

I was re-auditing this pack's meta.json files after #1272 (stale bboxes on 106 verified
patches, same pack), checking whether other recorded fields disagree with the grids they
describe. The shell was the only tifxyz of 45,994 that did, so I ran the published binaries
against it to see what actually consumes the field.

A second consumer, same 400x

vc_tifxyz2obj derives its UV multiplier as 1/scale and prints it. On the published shell,
against a conforming patch from the same pack:

$ vc_tifxyz2obj outer_shell outer_shell.obj
Point dims: [1012 x 906] cols: 1012 rows: 906
UVs: scaled by 1/scale from meta.json  (u*= 0.0500067, v*= 0.0500083 )
      (meta scale = [19.9973, 19.9967])

$ vc_tifxyz2obj verified_patches/0000_low_band_final patch.obj
Point dims: [3498 x 130] cols: 3498 rows: 130
UVs: scaled by 1/scale from meta.json  (u*= 20, v*= 20 )
      (meta scale = [0.05, 0.05])

The exported OBJ carries it through. The shell's furthest valid vertex is at grid column 800,
row 866, so its texture coordinates should span 16000 x 17320 (and the file's own bbox
spans 17,317 voxels in z, which agrees). What comes out is:

$ grep '^vt ' outer_shell.obj | awk '{if($2>mx)mx=$2; if($3>my)my=$3} END{print mx, my}'
40.005363 43.307175

400.0x too small, on both axes.

The repo's own tools disagree with this file

volume-cartographer/scripts/repair_tifxyz_spacing.py measures real spacing against
--target-spacing (default 20.0, --threshold 0.15). Via its measure_spacing, unmodified:

tifxyz                                     scale declared   measured (right, down)   factor   needs_repair
outer_shell                                    19.997318           (21.26, 20.49)   1.0437          False
verified_patches/0000_low_band_final            0.050000           (19.93, 19.95)   0.9972          False
unverified_patches/auto_grown_2026042011494...  0.050000           (19.99, 20.00)   0.9999          False

The geometry is fine: the shell measures as a 20-voxel grid, 4.4% off target and well inside
the script's 15% threshold. It is the metadata that disagrees. Note that this script reads
meta["scale"] (load_scale) and measures the spacing, but never compares the two.

volume-cartographer/scripts/spiral/spiral_service.py:840 is the repo's existing check on
exactly this relationship, and it rejects the published file:

>>> from spiral_service import _validate_tifxyz_output_step
>>> _validate_tifxyz_output_step(shell_meta, 20.0)
RuntimeError: Lasagna output scale does not match requested preview step 20
>>> _validate_tifxyz_output_step(patch_meta_0p05, 20.0)     # same pack
[0.05000000074505806, 0.05000000074505806]
>>> _validate_tifxyz_output_step(patch_meta_0p08, 12.5)     # same pack
[0.07999999821186066, 0.07999999821186066]

It raises for the shell at every plausible step (20.0, 20.49, 21.26).

Numbers

Exhaustive sweep, every meta.json under the pack, both grid axes, no sampling:

directory	scale	n	1/scale	measured step, rows	measured step, cols
unverified_patches	0.05	40,782	20.000	19.999	20.004
verified_patches	0.05	4,213	20.000	20.833	19.850
verified_patches	0.08	709	12.500	12.544	12.558
verified_patches/backups	0.05	265	20.000	20.063	19.972
verified_patches/backups	0.08	24	12.500	12.512	12.535
outer_shell	19.997	1	0.050	20.493	21.261

45,994 tifxyz, 0 read errors. Steps are per-directory medians over grid edges with both
endpoints valid ((x,y,z) != -1 and z > 0). Across all 45,260 tifxyz at scale 0.05 the
ratio of measured step to 1/scale has median 1.0001 (p1 0.947, p99 1.109); at 0.08, median
1.0038. The shell's grid sits in that same band against a step of 20 (ratios 1.025 and
1.063), not against the 0.05 its scale implies.

The shell has 468,046 valid vertices of 916,872 (51.0%) on a 906 x 1012 grid; that count is
identical under the (-1,-1,-1) sentinel, under (xyz != -1).all()
(lasagna/tifxyz_io.py:85), and under the z <= 0 invalidation the C++ reader applies
(volume-cartographer/core/src/QuadSurface.cpp:2386-2390).

Internal contradiction, using only fields of this file

area_vx2 is 293,645,839.8, in voxels² by its own name. The grid has at most 905 x 1011 =
914,955 quads, so the file asserts quads of at least 321 vx², i.e. edges of at least 17.9
voxels. Read scale literally and a quad is (1/19.997)² = 0.0025 vx². The two fields are
252,000x apart.

The convention, and where the repo disagrees with itself about it

Consumers divide by scale to get voxels:

• volume-cartographer/scripts/spiral/merge_concat_runs.py:95-101:
  """Recover grid step size (voxels/pixel) from a source meta's scale.""" … (L97-100
  elided) … return 1.0 / float(scale[0])
• volume-cartographer/scripts/spiral/render_ink.py:97: step_size = 1.0 / meta['scale'][0]
• volume-cartographer/scripts/spiral/dt_targets.py:297:
  # patch.scale is grid cells per voxel, so convert the shared physical …
• volume-cartographer/scripts/spiral/tifxyz.py:42:
  self.area = (self.valid_quad_mask).sum() * (1 / self.scale).prod()
• lasagna/fit.py:1111-1115, lasagna/init_shell_index.py:99-102: same 1.0 / scale[0]
• volume-cartographer/core/src/QuadSurface.cpp:902-909: gridToSurface divides, and
  size() (L871-875) returns cols / _scale[0], i.e. 50 x 45 voxels for this shell.
• volume-cartographer/apps/VC3D/SegmentationCommandHandler.cpp:1418:
  Requested: "scale": [0.05, 0.05] for a 20-voxel flatten.

Two places say the opposite, which is presumably how this happens:

• lasagna/tifxyz_format.md §5.1 says scale "describes the grid spacing in
  surface-parameter space" and that tools read (u = col * sx, v = row * sy), multiplying
  rather than dividing. That document presents itself as sufficient to implement compatible
  readers, and its own §8 points at QuadSurface.cpp as the reference, which divides. §5.1
  looks simply wrong and is worth fixing in the same pass.
• volume-cartographer/apps/src/vc_obj2tifxyz.cpp:241,260 writes a distance into scale
  ("OBJ units per pixel", "the distance between adjacent samples"). That is vc_obj2tifxyz writes scale from the stretch factor, not the actual grid density - breaks vc_flatten downstream #1319.

I do not know which tool wrote this particular file, and I have not tried to guess.

Repro

Two commands with the published binary and the published data:

vc_flatten -i outer_shell -o shell_flat            # OOM, asks OpenCV for 1.3 TB

cp -r outer_shell shell_fixed                      # same grids, one field changed
python -c "import json;p='shell_fixed/meta.json';m=json.load(open(p));m['scale']=[0.05,0.05];json.dump(m,open(p,'w'),indent=4)"
vc_flatten -i shell_fixed -o shell_flat_fixed      # Done!

Or without any binary, from the 420-byte meta.json alone (grid is 906 x 1012):

import json, urllib.request
u = ("https://huggingface.co/buckets/scrollprize/datasets/resolve"
     "/spiral/PHercParis4/outer_shell/meta.json")
m = json.load(urllib.request.urlopen(u))
lo, hi = m["bbox"]
print("scale               ", m["scale"])
print("step = 1/scale      ", 1 / m["scale"][0], "voxel")
print("bbox span (voxels)  ", [round(hi[i] - lo[i]) for i in range(3)])
print("area_vx2 per quad, upper bound on quads:", m["area_vx2"] / (905 * 1011))

scale                [19.997318267822266, 19.996686935424805]
step = 1/scale       0.05000670522952583 voxel
bbox span (voxels)   [6716, 6407, 17317]
area_vx2 per quad, upper bound on quads: 320.94019903302734

The measured step, if you want the grids (11 MB):

import numpy as np, tifffile
d = "outer_shell"
P = np.stack([tifffile.imread(f"{d}/{c}.tif") for c in "xyz"], -1).astype(np.float64)
ok = (P != -1).all(-1) & (P[..., 2] > 0)
for axis, name in ((1, "cols"), (0, "rows")):
    a, b = (P[:, 1:], P[:, :-1]) if axis else (P[1:], P[:-1])
    m = (ok[:, 1:] & ok[:, :-1]) if axis else (ok[1:] & ok[:-1])
    e = np.linalg.norm(a - b, axis=-1)[m]
    print(name, "median step:", round(float(np.median(e[e > 0])), 2), "voxels")
# cols median step: 21.26 voxels
# rows median step: 20.49 voxels

Possible fixes

1. Republish outer_shell/meta.json with scale: [0.05, 0.05], matching the 45,260 other
   tifxyz built at the same step. One field, 420-byte file, and vc_flatten stops dying on
   it.
2. Fix lasagna/tifxyz_format.md §5.1, which currently documents the multiply direction and
   is the likeliest source of new instances.
3. Optionally, guard the sizing: vc_flatten computing a 361753 x 303196 output from a
   906 x 1012 input is worth refusing with a message rather than handing 1.3 TB to OpenCV.
   Happy to send that, or the metadata check (repair_tifxyz_spacing.py already loads
   scale and already measures the spacing without comparing them), if either is useful to
   you.