The Slurm pipeline added in #9 has one sbatch script per step, and Parquet is not one of them — 04-export.sbatch runs --format json only. So a Parquet export has no first-class way to run on the cluster: #29 (measure Parquet at corpus scale) currently suggests an sbatch --wrap one-liner, which works but has no #SBATCH headers of its own and no home for the reasoning behind them.
This is deliberately not urgent: Parquet has no consumer today. It exists for downloadable queries that nobody is running, so this is a someday-when-it-matters item rather than a gap. Filing it so the --wrap workaround in #29 is not mistaken for the intended shape.
What it would be
A 06-parquet.sbatch alongside the others, sourcing slurm/config.sh like the rest, and a parquet entry in submit.sh's step list. Deliberately not part of submit.sh all, since a routine rebuild does not need it.
What blocks doing it well
The #SBATCH headers are supposed to be sized from measurements (see slurm/README.md), and Parquet has never run at corpus scale — that is #29. Written today the headers would be a guess copied from the JSON export, which is exactly the pattern that produced the two retractions in #9 (#36, #38).
So the honest order is #29 first, this second: measure once with the --wrap, then write the script around the number. If Parquet ever gains a real consumer, that ordering is still right, just more urgent.
The Slurm pipeline added in #9 has one
sbatchscript per step, and Parquet is not one of them —04-export.sbatchruns--format jsononly. So a Parquet export has no first-class way to run on the cluster: #29 (measure Parquet at corpus scale) currently suggests ansbatch --wrapone-liner, which works but has no#SBATCHheaders of its own and no home for the reasoning behind them.This is deliberately not urgent: Parquet has no consumer today. It exists for downloadable queries that nobody is running, so this is a someday-when-it-matters item rather than a gap. Filing it so the
--wrapworkaround in #29 is not mistaken for the intended shape.What it would be
A
06-parquet.sbatchalongside the others, sourcingslurm/config.shlike the rest, and aparquetentry insubmit.sh's step list. Deliberately not part ofsubmit.sh all, since a routine rebuild does not need it.What blocks doing it well
The
#SBATCHheaders are supposed to be sized from measurements (seeslurm/README.md), and Parquet has never run at corpus scale — that is #29. Written today the headers would be a guess copied from the JSON export, which is exactly the pattern that produced the two retractions in #9 (#36, #38).So the honest order is #29 first, this second: measure once with the
--wrap, then write the script around the number. If Parquet ever gains a real consumer, that ordering is still right, just more urgent.