Raised by #9 (Phase 6, stage 5).
One --db_dir per experiment set means several separately submitted Slurm jobs append to one overall_results.csv. Those processes share no process pool, so results.write_row now holds an flock across reading the header and appending the row. That is what makes the header write safe between jobs — the previous lock was a multiprocessing.Manager lock created per sweep, which serialised workers inside one job and nothing between jobs.
flock needs the filesystem mounted for it. Lustre grants it only with the flock mount option. On a refusal the code reports it once per file and appends anyway, since refusing to record a finished experiment is worse than a small risk of an interleaved row — but that means a silent degradation if nobody looks.
slurm/smoke_test.sh probes it and prints one of:
flock: supported on <filesystem>
flock: REFUSED - jobs writing to one results file may interleave
To close: run the smoke script on Isambard and record which. If it is refused, the fallback is one file per job plus a concatenation step, rather than a lock.
Raised by #9 (Phase 6, stage 5).
One
--db_dirper experiment set means several separately submitted Slurm jobs append to oneoverall_results.csv. Those processes share no process pool, soresults.write_rownow holds anflockacross reading the header and appending the row. That is what makes the header write safe between jobs — the previous lock was amultiprocessing.Managerlock created per sweep, which serialised workers inside one job and nothing between jobs.flockneeds the filesystem mounted for it. Lustre grants it only with theflockmount option. On a refusal the code reports it once per file and appends anyway, since refusing to record a finished experiment is worse than a small risk of an interleaved row — but that means a silent degradation if nobody looks.slurm/smoke_test.shprobes it and prints one of:To close: run the smoke script on Isambard and record which. If it is refused, the fallback is one file per job plus a concatenation step, rather than a lock.