Paper: Adapt: Prototyping a Real-Time, Reproducible Data Analysis Framework for Adaptive Radar Scanning

[RBhupi]

If you are creating this PR in order to submit a draft of your paper, please name your PR with Paper: <title>. An editor will then add a draft label; this will trigger GitHub Actions to run automated checks on your paper and build a preview. You may then work to resolve failed checks and ensure the preview build looks correct. If you have any questions, please tag the proceedings team in a comment on your PR with @scipy-conference/2026-proceedings.

See the project readme for more information.

[github-actions]

Curvenote Preview

Directory	Preview	Checks	Updated (UTC)
papers/bhupendra_raut	🔍 Inspect	✅ 75 checks passed (17 optional)	Aug 14, 2026, 2:44 AM

[github-actions]

Curvenote Preview

📭 No submissions available to inspect.

[QiZhao-NJU]

Hi, this is Qi Zhao. I am confirming that I have been assigned as a reviewer for this paper and I am still available to complete the review. Thank you.

[nyetsche]

I am confirming that I am assigned as a reviewer and still capable of doing so.

[chrisholloman]

Hello, all. I'm Chris Holloman (@chrisholloman), from the Scipy Proceedings committee. I will serve as the editor on this paper. Please let me know if I can help in anyway. Thanks @QiZhao-NJU and @nyetsche for agreeing to review this paper.

[nyetsche]

Initial review completed @RBhupi . Overall a good paper and links to extant, "ready to run" github repo. Most of my comments are for more details or examples of the code/configuration proposed, particularly if done in a way that the scientific expert would find useful ("pipeline override for weather phenomenon A and D, but not B, C, or D"). 👍

[QiZhao-NJU]

Summary:
Thank you for the submission. The paper presents Adapt, a modular framework for real-time radar data processing and adaptive radar-scanning workflows. I find the topic relevant to the SciPy Proceedings, and the architecture is promising, especially the modular design, DAG-based execution, and contract-based validation. However, the current manuscript still reads more like a design proposal or roadmap than a fully verifiable software proceedings paper.

Comments:

1. Please clarify the implementation status. Some parts appear to describe implemented features, while others seem to be planned work. A short table separating implemented, experimental, and future components would improve clarity.

2. Please add a direct code and documentation link, preferably with a tagged version or commit hash. The current placeholder should be replaced.

3. Please include a minimal reproducible example, such as a command or configuration file showing how to run the NEXRAD pipeline on a small historical case and what outputs are expected.

4. Please provide evidence for the “real-time” claim. A small benchmark table with per-module runtime, end-to-end latency, hardware setting, and processing time per scan would be helpful.

5. Please add one concrete NEXRAD case study, including radar site, date/time range, number of scans, detected/tracked cells, and example output figures.

6. Please describe the scientific modules in more detail, including inputs, outputs, contracts, main algorithmic choices, and possible failure modes.

7. Please better distinguish Adapt from existing tools such as PyART, tobac, TINT, CoCo-MET, and TITAN. A short comparison table would make the contribution clearer.

Overall, I am supportive of the direction, but I think the paper needs revision before it is ready for inclusion. Addressing the reproducibility, implementation-status, benchmarking, and validation issues above would make the contribution much clearer and more useful to the SciPy community.

[RBhupi]

Initial review completed @RBhupi . Overall a good paper and links to extant, "ready to run" github repo. Most of my comments are for more details or examples of the code/configuration proposed, particularly if done in a way that the scientific expert would find useful ("pipeline override for weather phenomenon A and D, but not B, C, or D"). 👍

Thank you @nyetsche for the detailed review.

• Abstract "ARM computing infrastructure": Done. Changed to "DOE-ARM" to avoid confusion with the ARM CPU architecture.
• Why PyART and not tobac/TINT/CoCoMET: Added a sentence to the Scientific Python Stack. PyART is the core radar library (I/O, gridding, classification, and more — beyond tracking); tobac/TINT/CoCoMET are focused segmentation/tracking toolkits. Our detection uses a tobac-like segmentation and our tracking is an improved TINT adapted for real-time operation. Citation order (Raut = TINT) also corrected.
• "scientific modules" used before defined: Done. First usages now read "scientific algorithms"; "modules" is reserved for Adapt modules after they are defined.
• NEXRAD Level-II / AWS S3: Done. Added an inline definition of Level-II (base reflectivity, velocity, spectrum width), the NOAA dataset citation (doi:10.7289/V5W9574V), and the public bucket s3://noaa-nexrad-level2 with the AWS Open Data registry URL.
• Describe DuckDB/Parquet/PyArrow/Zarr/Icechunk: Done. Added descriptions and citations for the Scientific Python Stack and Repository sections.
• importlinter code style + "regardless of test outcomes": Done. Now import-linter in code style, reworded to "fails the build even when every unit test passes."
• DataRepository clarification: Done. Clarified that the repository is a structured on-disk directory (SQLite tables + per-scan NetCDF/Parquet); DataRepository owns the layout so modules never open files or build paths, which is what lets us swap the backend (e.g. Zarr/Icechunk) without touching the science code.
• "repository structure will change" — breaking changes?: Done. Reworded to state the layout and APIs are stabilising ahead of v1.0 and consumers should expect breaking changes until then.
• "campaign operator" jargon: Done. Replaced with "A shared configuration file captures pipeline settings and algorithm parameters for reproducibility."
• Code example for input_contracts/output_contracts: Done. Added a declaration example on the detection module (output_contracts = {"segmented_ds": (check_segmentation, "v1")}) in the Contract-Based Validation section, using the segmenter-swap case you pointed to.
• GraphBuilder figure: GraphBuilder is a small component that just derives the DAG from module input/output declarations, so we clarified this in text rather than adding a separate figure.
• Example config + CLI run (Multiple Pipelines): Done. Added a config.yaml excerpt and CLI commands (adapt config, adapt run-nexrad config.yaml --radar KHTX --mode historical ...), including --only/--not for subsetting modules.
• Config hierarchy example / what-overrides-what: Done. The same config + CLI block shows a user config overriding only selected defaults, with CLI flags overriding the config file.

[RBhupi]

Thank you @QiZhao-NJU for the review. Changes below are in the latest commit.

1. Implementation status: Done. Added an implementation-status table separating Implemented / Experimental / In-progress / Planned components.
2. Code/documentation link with version: The code and docs link is included; for this draft we have kept it to the repository URL (the framework is pre-v1.0 and evolving), and will pin a tagged release for the camera-ready.
3. Minimal reproducible example: Done. Added a config.yaml excerpt and the CLI to run the NEXRAD pipeline on a small historical window for a single radar.
4. Evidence for "real-time": Done. Added a per-module benchmark table from a sample KHTX run (Apple M3, 36 GB RAM): ~19 s average per scan versus the 4–6 min NEXRAD volume interval, showing real-time headroom.
5. Concrete NEXRAD case: The benchmark run doubles as a worked case (KHTX, six scans, 343 detected cells); we plan to add example output figures for the camera-ready.
6. Module detail: Done. Expanded the modules table with inputs → outputs and function per module, corrected the count to the full six (added cell_volume_stats), and ordered them by data flow. Contract enforcement is described with an example in the Contract-Based Validation section.
7. Distinguish Adapt from PyART/tobac/TINT/CoCoMET/TITAN: Adapt is a framework/orchestrator rather than a competitor to these libraries — it composes them (e.g. PyART for I/O and gridding) and adds real-time streaming execution, a modular DAG, configuration-defined pipelines, and boundary contracts. We clarified this framing in the text (Introduction and Scientific Python Stack) rather than adding a comparison table.

[RBhupi]

@chrisholloman Thank you for your work. The replies to the reviewers comments are posted and the changes are in the new commit Changes as per reviewers comments commit id 67b04ff.

[RBhupi]

@chrisholloman This build is failing because of the github outage. Please check!

[chrisholloman]

@chrisholloman This build is failing because of the github outage. Please check!

Hi @RBhupi, were you able to try this again after github's outage ended? Maybe just push an empty commit and let it rebuild?

[QiZhao-NJU]

Thank you for the revisions. The added implementation-status clarification, code/documentation link, minimal runnable example, runtime benchmark, KHTX worked case, expanded module description, and clearer positioning relative to existing radar tools address my main concerns. I am satisfied with the revision overall.