docs: organise data campaigns by code and task

[junwen94]

Problem

goldilocks-data currently has reusable Python modules, campaign scripts, notebooks, and processed results split across top-level directories. A visitor cannot yet enter the repository through a DFT code and task, then follow one self-contained path from plugin installation to submission, monitoring, collection, analysis, and data access.

The repository needs a simple campaign-first structure suitable for both contributors and a GitHub Pages data portal.

Goals

• Organize user-facing campaign material as code → task.
• Make every task directory independently reproducible through its own README.md.
• Keep submission, monitoring, collection, cleanup, and analysis responsibilities explicit.
• Document the AiiDA plugin required by each DFT code.
• Migrate the existing Quantum ESPRESSO k-point campaign as the first real implementation.
• Use the same code/task structure to drive the GitHub Pages navigation and dataset catalog.

Non-goals

• Do not create empty VASP, CASTEP, or other code/task directories before real campaigns exist.
• Do not introduce a speculative cross-code adapter framework in this phase.
• Do not commit large calculation archives directly to Git without an explicit storage decision.
• Do not redesign the scientific k-index schedule or convergence thresholds during the directory migration.

Architecture

goldilocks-data/
├── README.md
├── codes/
│   └── qe/
│       ├── README.md
│       └── kpoints/
│           ├── README.md
│           ├── campaign.yaml
│           ├── scripts/
│           │   ├── submit.py
│           │   ├── monitor.py
│           │   ├── collect.py
│           │   └── cleanup.py
│           ├── notebooks/
│           │   └── analysis.ipynb
│           └── results/
│               ├── README.md
│               ├── manifest.json
│               ├── snapshot-metadata.json
│               └── source-summary.csv
├── src/goldilocks_data/
├── tests/
└── docs/
    ├── index.html
    ├── codes/
    └── tasks/

src/goldilocks_data/ continues to hold genuinely reusable mechanics. codes/<code>/<task>/ owns the concrete campaign configuration, operational scripts, reproducible notebook, and result manifest.

Required task README content

Every codes/<code>/<task>/README.md must explain:

1. Scientific objective and calculation scope.
2. Required DFT executable and version.
3. Required AiiDA plugin and installation command.
4. AiiDA profile, computer, code, and pseudopotential setup.
5. Input structure source and selection rules.
6. Dry-run and real submission commands.
7. Monitoring and safe stopping instructions.
8. Result collection and snapshot generation.
9. Analysis/notebook execution.
10. Data locations, versions, provenance, and known limitations.

For QE k-points, the README will document aiida-quantumespresso and aiida-pseudo explicitly.

QE k-points campaign contract

The first campaign is:

code: qe
task: kpoints
calculation: scf
pseudopotential family: PseudoDojo/0.4/PBEsol/SR/standard/upf
k-point convention: Gamma-inclusive, unshifted
extension size: 3 consecutive k-index values per structure
active WorkChain limit: 50
monitor interval: 15 minutes
convergence window: at least 3 remaining points
medium: 5 meV/atom
well: 3 meV/atom
ultra: 1 meV/atom

The operational loop is:

submit next 3 k-index points
→ wait for completion
→ collect fresh AiiDA results
→ recompute convergence labels
→ stop at ultra or submit the next 3 points

Failed, missing-energy, or partially submitted sources must be separated from the normal extension queue and reported explicitly.

Campaign metadata

campaign.yaml is the machine-readable source for code/task identity, plugin versions, AiiDA process, pseudopotential family, k-point convention, convergence thresholds, resource defaults, batching limits, and output schema/version.

The GitHub Pages catalog should read or mirror this metadata rather than maintaining an unrelated hand-written set of scientific parameters.

Phases

Phase 1: establish the convention

• Add codes/README.md describing the code/task convention.
• Add codes/qe/README.md describing QE and its AiiDA plugin requirements.
• Add codes/qe/kpoints/campaign.yaml with the current campaign contract.
• Add a complete codes/qe/kpoints/README.md reproduction guide.

Verification: a new contributor can identify the correct task directory and understand all prerequisites without inspecting unrelated notebooks.

Phase 2: migrate the current QE k-points workflow

• Move/adapt the one-cycle extension controller to codes/qe/kpoints/scripts/submit.py.
• Move/adapt the 15-minute wrapper to codes/qe/kpoints/scripts/monitor.py.
• Add collect.py for AiiDA → Parquet/CSV snapshot generation.
• Add or adapt cleanup.py for finished remote folders.
• Move the current batch analysis notebook to codes/qe/kpoints/notebooks/analysis.ipynb.
• Preserve tests for de-duplication, active limits, three-point extensions, and convergence thresholds.

Verification: dry-run, unit tests, Ruff, and one read-only AiiDA collection check pass from the documented commands.

Phase 3: define result publication

• Add a result manifest.json with dataset identity, version, row counts, generation date, checksums, and download locations.
• Keep the small source summary and metadata close to the campaign where practical.
• Choose external storage for large Parquet/calculation archives before publication.
• Document live AiiDA provenance separately from immutable released snapshots.

Verification: every downloadable artifact has a version, provenance description, and integrity metadata.

Phase 4: connect GitHub Pages

• Present implemented codes and tasks only.
• Add a QE/k-points page covering method, plugin setup, data access, and analysis.
• Render current PseudoDojo statistics and SSSP comparison from verified snapshot metadata.
• Link each public page back to its campaign README, scripts, notebook, and result manifest.
• Configure and verify GitHub Pages deployment for this private repository or document the required visibility/plan constraint.

Verification: the deployed site supports the path “code → task → how it was generated → where to find it → what the analysis shows.”

Acceptance criteria

• The repository has one documented code/task convention.
• The current QE k-points campaign is self-contained under codes/qe/kpoints/.
• Installation instructions name and verify the relevant AiiDA plugins.
• The documented monitor never exceeds 50 active WorkChains and submits three consecutive points per source.
• Unconverged sources can move through repeated collect/analyse/extend rounds.
• The analysis uses 5/3/1 meV/atom for medium/well/ultra with at least three tail points.
• The data manifest separates live provenance, small in-repo summaries, and externally stored large artifacts.
• GitHub Pages is navigable by code and task and does not advertise unimplemented campaigns as available.

---

Written by an agent on behalf of Junwen Yin.

[junwen94]

Done

• Created the local issue-3-code-task-architecture branch.
• Organised the first campaign as codes/qe/kpoints/, containing task documentation, campaign settings, submission/extension/monitoring scripts, the curated analysis notebook, and a portable result snapshot.
• Kept historical notebooks locally under an ignored archive/ directory.
• Removed personal absolute paths from the active scripts and notebook; local CIF and SSSP inputs are now explicit configuration.
• Updated the root README and GitHub Pages links to the new code/task paths.
• Added repository ignore rules for local caches, agent files, backups, and intermediate pickle data.

Decisions

• Use code → task as the public hierarchy; do not add speculative plugin adapters or empty code directories.
• Keep reusable mechanics in src/goldilocks_data/, while each task owns its operational scripts, notebook, settings, and results manifest.
• Preserve AiiDA as the authoritative provenance record and describe exported files as dated analysis snapshots.
• The continuation loop keeps the agreed limits: three kindex points per material, at most 50 new/active WorkChains, checked every 15 minutes.

Verification

• uv run pytest: 18 passed.
• Ruff passed for src, tests, and the campaign scripts.
• All three campaign CLIs passed --help smoke checks.
• The curated notebook and JSON metadata/manifest parse successfully.

Blockers / Open Questions

• None for the local architecture milestone.
• The work is not committed or pushed yet; the official upstream repository is currently empty.

Git State

• Branch: issue-3-code-task-architecture
• Pushed: no
• PR: none
• Ready for next step: yes — review the content boundary, then commit and open a PR when requested.

---

Written by an agent on behalf of Junwen Yin.