gmat-sweep v0.4 — four more backends, six DataFrame helpers, Sobol sensitivity, archive bundles

[djankov]

gmat-sweep v0.4 is on PyPI. This one is the quality-of-life and ecosystem release: three more execution backends join KubernetesJobPool (MPIPool, ProcessPoolExecutorPool, DebugPool), six DataFrame helpers land on top of the aggregator (sweep_summary, sweep_diff, mc_convergence, lazy_fused_reports, plus monte_carlo_extend / latin_hypercube_extend), Sobol sensitivity and matplotlib plot helpers ship behind opt-in extras, an opt-in Polars output engine runs alongside the pandas default, Sweep.archive packs a finished sweep into a deterministic .zip for Zenodo / JOSS deposits, and notebook-friendly _repr_html_ rendering lands on Sweep / RunOutcome / ManifestEntry. CI gains a smoke-canary cell against the canonical ghcr.io/astro-tools/gmat image; per-PR cells trim from 18 to 10 with the heavy integration work moving to a scheduled integration.yml. Coverage gate raises from ≥ 85 % to ≥ 90 %.

from gmat_sweep import monte_carlo, sweep, sweep_summary
from gmat_sweep.plotting import sweep_corner
from gmat_sweep.sensitivity import sobol_sample, sobol_analyze

# Sobol sensitivity workflow — Saltelli design → sweep → first/total indices.
perturb = {"Sat.SMA": ("normal", 7100.0, 50.0),
           "Sat.DryMass": ("normal", 100.0, 10.0)}
samples = sobol_sample(perturb, n=256, seed=42)
df = sweep("mission.script", samples=samples)
indices = sobol_analyze(df[df["__status"] == "ok"], perturb, metric="MissDistance")

# Or pair Monte Carlo with the new summary + corner-plot helpers.
mc = monte_carlo("mission.script", n=1000,
                 perturb={"Sat.SMA": ("normal", 7100.0, 50.0)}, seed=42)
bands = sweep_summary(mc, by="time", q=(0.05, 0.5, 0.95))
sweep_corner(mc, params=["Sat.SMA"], metric="MissDistance")

What's new in v0.4

• Four more execution backends. KubernetesJobPool (pip install gmat-sweep[k8s]) makes every run one batch/v1 Job and every Pod a fresh interpreter, draining completions via a kubernetes.watch.Watch loop. New pod-per-run recipe walks the wiring (#101). MPIPool ([mpi]) wraps mpi4py.futures.MPIPoolExecutor and works under both dynamic-spawn and pre-allocated-rank launches without any launcher detection on this side (#102). ProcessPoolExecutorPool (stdlib, Python ≥ 3.11) wraps concurrent.futures.ProcessPoolExecutor(max_tasks_per_child=1) — no joblib / loky dependency, coexists with LocalJoblibPool (#104). DebugPool dispatches in-process so a breakpoint() reached during a run drops into the driver's debugger — two opt-in flags gate the deliberate isolation violation (#105).
• Six DataFrame helpers on top of the aggregator. sweep_summary collapses a (run_id, time)-MultiIndexed frame into per-by statistics with a 2-level (statistic, field) column MultiIndex — default 5/50/95 quantiles plus mean/std feed directly into the plot helpers (#114). sweep_diff pairwise-compares two same-shape sweep DataFrames, emitting <col>__diff and/or <col>__rel per shared numeric column (#119). mc_convergence returns per-prefix running mean / std / SE-of-mean — answers "did my Monte Carlo converge?" for a chosen metric (#117). lazy_fused_reports fuses N ReportFile outputs per run into one DataFrame with column-level MultiIndex keyed by (report_name, column) (#107). monte_carlo_extend and latin_hypercube_extend (sibling typed-refusal sentinel) run only the new n samples on top of an existing sweep — the original run_ids' draws are bit-equal to the same indices of a fresh monte_carlo(n=old_n + n, seed=...) (#115).
• Sobol sensitivity indices via SALib. New gmat_sweep.sensitivity module behind gmat-sweep[sensitivity]. sobol_sample(perturb, n, ...) builds the Saltelli/Sobol design as an explicit-row DataFrame; sobol_analyze(df, perturb, metric, ...) returns first / total / second-order indices with bootstrap confidence intervals. New docs/sensitivity.md walks the design and the metric-callable contract (#121).
• Matplotlib plot helpers. New gmat_sweep.plotting module behind gmat-sweep[plot]. sweep_corner is a pair plot of perturbed dotted-paths coloured by a per-run scalar metric — kind="auto" flips to hexbin past 2000 runs to avoid silent overplot saturation. sweep_heatmap is a contour-grade heatmap on a 2-axis grid sweep. matplotlib is imported lazily inside each helper, so import gmat_sweep.plotting succeeds without the extra (#109).
• Sweep.archive for Zenodo / JOSS deposits. Sweep.archive(out, *, include_logs=False) and a matching gmat-sweep archive CLI subcommand pack a finished sweep — script, manifest, per-run Parquet outputs, a generated reproduce-recipe README, and a sha256sum-compatible MANIFEST.hash — into one deterministic .zip ready for an archival deposit. Bundled manifest output_paths / log_path are rewritten to bundle-relative form (#111).
• Opt-in Polars output engine. engine="polars" across sweep / monte_carlo / latin_hypercube / monte_carlo_extend, Sweep.to_dataframe / to_ephemerides / to_contacts (plus a Sweep.to_polars() shortcut), and the standalone lazy_multiindex / lazy_ephemerides / lazy_contacts / sweep_diff / mc_convergence. Pandas remains the default; behind gmat-sweep[polars]. Marked experimental for v0.4 (#120).
• Notebook-friendly _repr_html_. Sweep / RunOutcome / ManifestEntry render as compact HTML key/value tables in Jupyter instead of the default <gmat_sweep.sweep.Sweep object at 0x…> placeholder; __repr__ is unchanged (#108).
• Three new example notebooks. Sobol sensitivity — Saltelli design + first/total-order indices with 95 % CIs. Archive bundle — pack a sweep with Sweep.archive(), inspect + re-aggregate from the unzipped bundle. Extending Monte Carlo — 100-run base + monte_carlo_extend(n=200) + bit-for-bit determinism assertion on the original 100. sweep_summary, sweep_corner, and sweep_heatmap cells appear inline across the existing notebooks (#124).
• Smoke-canary cell against the canonical GMAT image. smoke-canary-image under the new integration.yml runs the four-line vision-snippet sweep and a 4-run Monte Carlo against ghcr.io/astro-tools/gmat:R2025a and :R2026a on every push to main, nightly, and on demand (#112).
• Per-PR CI trimmed from 18 to 10 cells. R2026a × 3 OS × 3 Python = 9, plus a single R2025a Linux/py3.12 sentinel. The full 3 × 3 × 2 = 18 matrix expands on main push; heavy backend and canary cells move to the new integration.yml (push:main + nightly + on-demand). No coverage loss post-merge (#126).
• CITATION.cff at the repo root so GitHub's "Cite this repository" UI and academic tooling resolve a canonical citation for the project (#116).

Hardening pass

A late-cycle round closed the v0.4-review punch list across the manifest, aggregator, worker / orchestration, every backend pool, the CLI, and the plot helpers:

• Streaming throughout. Pool.imap(specs, *, in_flight=None) ABC with a chunked default; grids.iter_grid_run_specs + Sweep.__init__ accepting an Iterable[RunSpec] mean a 10⁵-row factorial no longer pins 10⁵ specs + 10⁵ futures in driver memory. Manifest.iter_entries(path) and classmethod find_failed / find_missing mean gmat-sweep resume on a 10k-run manifest no longer pays 10k JSON parses against an entry list it never reads (#137, #140).
• Aggregator memory + correctness. _aggregate peak heap goes from ~67× to ~8× the final-frame size on a 1000-run fixture via pa.concat_tables(...).to_pandas(). Welford's online recurrence replaces the cumulative sum-of-squares variance identity — fixes catastrophic cancellation on km-magnitude metrics where np.clip(var, 0, None) was reporting zero std (#136).
• Uniform failure folding across pools. Every backend now folds worker-side transport failures (loky / ProcessPoolExecutor / Dask / MPI rank / Ray worker death, RayTaskError, pickling fault, …) into a synthetic RunOutcome.failed at the drain site instead of letting .result() raise and abort the sweep (#138).
• KubernetesJobPool deadline + close cleanup. New job_deadline_seconds: int = 3600 hang-protects stuck Jobs; close() background-deletes every in-flight Job on shutdown; spec JSON written to the PVC is unlinked on every Job-create failure path (#135).
• Manifest robustness. Optional batched fsync (fsync_each: bool = True / fsync_batch: int = 50) amortises durability across batches when you want it. canonical_script_sha256 strips a leading UTF-8 BOM before normalisation so a .script saved from a BOM-emitting Windows editor hashes equal to the same script without a BOM. ManifestCorruptError.line_number and a path:line CLI top-level handler make corruption locatable. A Manifest._migrate_header shim is a no-op for v1 → v1 but gives a future v2 bump a single hook (#137).
• CLI cold start. import gmat_sweep.cli is now lazy — pandas / pyarrow / tqdm / joblib are not pulled until first use. gmat-sweep --help drops from ~300–800 ms to ~120 ms on a warm cache, pinned by test_cold_start_does_not_load_heavy_dependencies (#139).
• Monotonic duration + spec validation. duration_s is now sourced from time.monotonic() — an NTP step mid-run can't drive duration negative. RunOutcome.from_dict / RunSpec.from_dict validate status and field types explicitly; bad data surfaces as ManifestCorruptError or exit code _EXIT_BAD_SPEC=3 (#140).

Behaviour changes worth knowing about

• LocalJoblibPool(workers=...) deprecated. max_workers= is the new spelling; workers= still works but emits DeprecationWarning. CLI --workers is unaffected (#138).
• Manifest.find_failed() / find_missing(...) are classmethods now. Pre-1.0; pass a path argument. The only public doc snippet that called the old form has been updated (#137).
• Polars engine is marked experimental in v0.4. Same APIs, but the contract may shift before v1.0; pandas remains the unflagged default (#136).
• Coverage gate raised from ≥ 85 % to ≥ 90 % on the Ubuntu / Python 3.12 / R2026a cell. The four per-file 95 % gates on grids.py, distributions.py, manifest.py, and aggregate.py are unchanged (#113).

Full notes: https://github.com/astro-tools/gmat-sweep/blob/main/CHANGELOG.md#040--2026-05-10

Install

pip install -U gmat-sweep                     # core (LocalJoblibPool, ProcessPoolExecutorPool, DebugPool)
pip install -U "gmat-sweep[dask]"             # adds DaskPool
pip install -U "gmat-sweep[ray]"              # adds RayPool
pip install -U "gmat-sweep[k8s]"              # adds KubernetesJobPool
pip install -U "gmat-sweep[mpi]"              # adds MPIPool
pip install -U "gmat-sweep[plot]"             # adds sweep_corner / sweep_heatmap (matplotlib)
pip install -U "gmat-sweep[sensitivity]"      # adds Sobol via SALib
pip install -U "gmat-sweep[polars]"           # adds the experimental polars output engine
pip install -U "gmat-sweep[examples]"         # matplotlib + every backend extra for the example notebooks

Same baseline: Python 3.10–3.12 and a local GMAT install. R2025a and R2026a are exercised on every PR (Ubuntu / Windows / macOS × Py 3.10 / 3.11 / 3.12 — the per-PR matrix is 10 cells now, with the full 18-cell matrix and the heavy backend / canary cells running on every push to main and nightly).

Links

• PyPI: https://pypi.org/project/gmat-sweep/
• Docs: https://astro-tools.github.io/gmat-sweep/
• Source: https://github.com/astro-tools/gmat-sweep
• Release notes: https://github.com/astro-tools/gmat-sweep/releases/tag/v0.4.0

Feedback wanted

• Tried the new pod-per-run k8s backend or the MPI backend on a real cluster? KubernetesJobPool and MPIPool are the v0.4 cluster surface that does not exist in v0.3 — comment or open an issue with the diff and any gotcha that didn't survive the recipes / docstrings. Job-deadline tuning, PVC layout, and mpirun launcher detection are the three places this is most likely to surprise.
• Using Sweep.archive for a deposit? The bundle is meant to be self-describing — script + manifest + per-run Parquets + a reproduce-recipe README + a sha256sum-compatible hash — but actual archival workflows are the test. If the bundle layout reads cleanly to a reviewer who has never seen gmat-sweep before, great; if there's a missing affordance (DOI placeholder, dataset card, citation block), that's exactly the kind of feedback that earns a v0.5 entry.
• Running the polars output engine? Marked experimental for v0.4 deliberately — the (run_id, time) index → column-pair translation is the conversation we want to have before v1.0. If you have a pandas → polars migration where the contract bent in an unexpected place, please flag the column shape on an issue.