docs(skills): add MatMaster DPA4 workflows

Signed-off-by: weiqichen77 <weiqchen@hku.hk>

Author: weiqichen77

doc/agent-skills.md

@@ -29,6 +29,9 @@ in the DeePMD-kit repository under `skills/`.
 - `lammps-deepmd`: Prepare, explain, and run LAMMPS simulations with DeePMD-kit
   potentials, including DPA4/SeZM `.pt2` deployment and common NVE, NVT, and
   NPT setups.
+- `matmaster-dpa4-workflows`: Run prepared DPA4 training, fine-tuning,
+  inference, and LAMMPS cases on MatMaster/Bohrium while delegating scientific
+  setup and acceptance to the sibling DeePMD-kit skills.
 
 ## Related reference
 
@@ -85,3 +88,5 @@ without launching an expensive calculation. For example:
   a LoRA fine-tuning input, but do not start training.”
 - “Use the `lammps-deepmd` skill to prepare an NVT LAMMPS input file for a
   DeePMD-kit model, and explain each command.”
+- “Use the `matmaster-dpa4-workflows` skill to prepare and validate a Bohrium
+  job specification for an existing DPA4 case, but do not submit it.”

skills/matmaster-dpa4-workflows/SKILL.md

@@ -0,0 +1,111 @@
+---
+name: matmaster-dpa4-workflows
+description: Run prepared DPA4 training, fine-tuning, inference, and LAMMPS cases on MatMaster/Bohrium. Use when an agent must stage persistent inputs, resolve Bohrium projects/images/GPU resources, generate and validate a job specification, submit or monitor jobs, collect outputs, recover checkpoints or restarts, and record platform provenance. Delegate scientific setup and acceptance to the installed DeePMD-kit skills.
+compatibility: Install with deepmd-train, deepmd-finetune-dpa4, deepmd-python-inference, and lammps-deepmd. Requires Python 3; production submission requires a MatMaster platform tool or a Bohrium CLI compatible with 2.5.17.
+license: LGPL-3.0-or-later
+metadata:
+  author: MatMaster
+  version: '1.0'
+  repository: https://github.com/deepmodeling/deepmd-kit
+---
+
+# Run DPA4 workflows on MatMaster
+
+Act only as the MatMaster/Bohrium platform layer. Own storage, resource
+resolution, job packaging, submission, monitoring, collection, and retry
+lineage. Do not own model architecture, training parameters, checkpoint/head
+selection, inference semantics, LAMMPS input design, or scientific acceptance.
+
+## Route scientific work
+
+- Use `deepmd-train` for a new DPA4 training case.
+- Use `deepmd-finetune-dpa4` for standard or LoRA fine-tuning.
+- Use `deepmd-python-inference` for Python inference or `dp test`.
+- Use `lammps-deepmd` for deployment, minimization, and MD.
+
+Require the owning skill to return a prepared case contract. If that skill is
+unavailable, install or reload the complete DeePMD-kit skill set instead of
+reconstructing its instructions here.
+
+Read references only when needed:
+
+- [workflow-handoffs.md](references/workflow-handoffs.md): required inputs,
+  outputs, and acceptance boundary for each workflow;
+- [matmaster-operations.md](references/matmaster-operations.md): storage,
+  resources, Bohrium CLI fallback, lifecycle, and recovery.
+
+## Execute
+
+### 1. Accept a prepared case
+
+Require a relative entry command, normally `bash run.sh`, plus all referenced
+files or durable paths, checksums, runtime/resource requirements, declared
+outputs, smoke-test criteria, and checkpoint/restart policy. Do not change the
+scientific command to fit an available resource.
+
+### 2. Inventory and stage
+
+Run:
+
+```bash
+python scripts/check_environment.py --case-dir CASE_DIR
+```
+
+Use `--require-file`, `--require-deepmd`, and `--require-lammps` as required by
+the handoff. Keep durable inputs and results under `/personal` or `/share`.
+Use relative paths inside job packages; the Bohrium working directory is not a
+stable absolute path.
+
+### 3. Resolve execution resources
+
+Discover a real project ID, full image address, current machine type, wall
+time, and output destination. Never reuse a historical project, image tag, GPU
+SKU, or case count as a default. Prove image/data/output compatibility with the
+bounded smoke test supplied by the owning skill.
+
+### 4. Build and validate the job
+
+Generate a concrete job specification:
+
+```bash
+python scripts/make_job_spec.py CASE_DIR \
+    --output CASE_DIR/job.json \
+    --project-id PROJECT_ID \
+    --image FULL_IMAGE_ADDRESS \
+    --machine MACHINE_TYPE \
+    --name JOB_NAME
+```
+
+Preview submission, then use the installed platform tool or the native Bohrium
+CLI dry run. Submit only after validation and explicit authorization. Stop if
+neither a submission tool nor a compatible `bohr` executable exists.
+
+### 5. Monitor and collect
+
+Reconcile every job/group ID with `assets/bohrium-job/manifest.json`. Download
+finished and failed cases, retain partial checkpoints/restarts, and run a
+filesystem/log first pass:
+
+```bash
+python scripts/audit_cases.py ROOT --json REPORT.json
+```
+
+Add `--mode` and repeat `--require` for declared outputs. A scheduler success
+or file-audit pass is not scientific acceptance; return artifacts to the owning
+DeePMD skill.
+
+### 6. Recover the smallest subset
+
+Classify packaging, path, image, quota, queue, resource, interruption, and
+wall-time failures here. Route training divergence/configuration, checkpoint
+validity, inference correctness, model export, type mapping, LAMMPS input, and
+numerical failures back to the owning skill. Retry only invalid cases and set
+`retry_of`; never overwrite evidence from an earlier attempt.
+
+## Return the platform ledger
+
+Return persistent paths, project/image/machine selections, case/job/group and
+attempt mappings, expected/submitted/downloaded/platform-valid counts, retry
+lineage, logs, declared outputs, audit report, and unresolved platform risks.
+Keep the platform verdict separate from training, inference, MD, and scientific
+verdicts.

skills/matmaster-dpa4-workflows/assets/bohrium-job/job.json

@@ -0,0 +1,19 @@
+{
+  "job_name": "__JOB_NAME__",
+  "command": "bash run.sh",
+  "log_file": "run.log",
+  "backward_files": [
+    "run.log",
+    "runtime.txt",
+    "logs/",
+    "results/",
+    "provenance.json"
+  ],
+  "project_id": "__PROJECT_ID__",
+  "machine_type": "__MACHINE_TYPE__",
+  "image_address": "__FULL_IMAGE_ADDRESS__",
+  "job_type": "container",
+  "max_reschedule_times": 0,
+  "max_run_time": "__MINUTES__",
+  "nnode": 1
+}

skills/matmaster-dpa4-workflows/assets/bohrium-job/manifest.json

@@ -0,0 +1,30 @@
+{
+  "run_id": "__RUN_ID__",
+  "workflow": "__TRAIN_FINETUNE_INFERENCE_OR_LAMMPS__",
+  "prepared_by": "__DEEPMD_SKILL__",
+  "project_id": "__PROJECT_ID__",
+  "image_address": "__FULL_IMAGE_ADDRESS__",
+  "machine_type": "__MACHINE_TYPE__",
+  "cases": [
+    {
+      "case_id": "__CASE_ID__",
+      "case_dir": "__RELATIVE_CASE_DIR__",
+      "entry_command": "bash run.sh",
+      "input_checksums": {},
+      "declared_outputs": [],
+      "attempts": [
+        {
+          "attempt": 0,
+          "job_id": null,
+          "group_id": null,
+          "retry_of": null,
+          "platform_status": "prepared",
+          "platform_verdict": null,
+          "reason": null,
+          "changed_platform_fields": [],
+          "result_dir": null
+        }
+      ]
+    }
+  ]
+}

skills/matmaster-dpa4-workflows/references/matmaster-operations.md

@@ -0,0 +1,86 @@
+# MatMaster and Bohrium operations
+
+## Capability order
+
+Use the first available capability:
+
+1. installed MatMaster/Bohrium skill or platform tool;
+1. an installed Bohrium CLI whose help matches the required operation;
+1. explicit prerequisite installation or user/platform intervention.
+
+This skill bundles no executable, credential, project, image, or GPU default.
+Do not invent an OpenAPI route when a platform tool or compatible CLI is
+missing. Use `python scripts/check_environment.py --probe --require-bohr` to
+inventory the fallback without exposing secret values.
+
+## Storage and job boundaries
+
+| Layer                  | Use                                | Persistence                       |
+| ---------------------- | ---------------------------------- | --------------------------------- |
+| MatMaster session node | prepare, inspect, and post-process | system disk is not durable        |
+| `/personal`            | user-scoped inputs and results     | durable                           |
+| `/share`               | project-shared inputs and results  | durable, permission scoped        |
+| Bohrium job workspace  | production execution               | temporary except declared outputs |
+
+Do not assume that a session-node package exists in the job image. Do not save
+mounted `/personal` or `/share` data into an image. Confirm project permission
+before writing shared storage, and stop unused billable nodes according to the
+accepted lifecycle policy.
+
+## Images and resources
+
+- Query current projects, full image addresses, and machine availability.
+- Pin the image address and version/digest; a display name is insufficient.
+- Select resources that satisfy the prepared handoff, then run its smoke test.
+- Treat image-pull/cache startup as distinct from a hung calculation.
+- Confirm project, quota, budget, wall time, and reschedule policy before submit.
+
+## Package and submit
+
+Keep the command in a short `run.sh`, use relative paths, write an incremental
+log, and declare every result needed after the temporary workspace disappears.
+Start from `assets/bohrium-job/job.json` or generate the resolved specification
+with `scripts/make_job_spec.py`. Reject every remaining `__PLACEHOLDER__`.
+
+For Bohrium CLI 2.5.17, the validation shape is:
+
+```bash
+bohr version
+bohr job submit -i job.json \
+    --input_directory ./case_dir/ \
+    --dry-run --output json
+```
+
+Inspect `bohr job submit --help` when the installed interface differs. Remove
+`--dry-run` only for the authorized production submission. Prefer a MatMaster
+built-in submit tool when present.
+
+Record the case ID, attempt, job/group IDs, project, image, machine, input
+checksums, declared outputs, and `retry_of` in the manifest. A reschedule may
+restart `run.sh` from the beginning; enable rescheduling only when the wrapper
+is safely restart-aware.
+
+## Monitor, collect, and recover
+
+For each job group:
+
+1. reconcile its members with the manifest;
+1. inspect representative running and failed logs;
+1. distinguish queue/image startup from execution progress;
+1. download finished and failed terminal jobs;
+1. run the file/log audit and return results to the owning skill;
+1. retry only invalid cases in a new attempt directory.
+
+`Finished` is a platform state, not proof of a valid model or simulation.
+Preserve partial logs, checkpoints, and restarts before any retry.
+
+Use terminate when recoverable outputs should normally remain. Use kill only
+for an unrecoverable job after resolving its exact ID. Delete jobs or persistent
+files only when explicitly requested and after verifying the target.
+
+## Authentication
+
+Let platform tools handle credentials when possible. For a CLI fallback, read
+the access key from the injected environment (`BOHR_ACCESS_KEY`, or
+`ACCESS_KEY` when required by that CLI version). Never print, hard-code, copy
+into a job package, or return credentials, storage tokens, or node passwords.

skills/matmaster-dpa4-workflows/references/workflow-handoffs.md

@@ -0,0 +1,52 @@
+# DeePMD workflow handoffs
+
+## Common contract
+
+Before staging, require the owning DeePMD skill to provide:
+
+- workflow kind, owning skill, case ID, and relative entry command;
+- every input file or durable path plus checksums/manifests;
+- image/runtime, accelerator, memory, node/rank, and wall-time requirements;
+- incremental logs, checkpoints/restarts, final outputs, and output cadence;
+- smoke-test contract, retry/resume constraints, and acceptance owner.
+
+Wrap the exact accepted command in `run.sh`. Do not reinterpret scientific
+configuration during packaging.
+
+## Training and fine-tuning
+
+`deepmd-train` or `deepmd-finetune-dpa4` owns the training input, data split,
+starting checkpoint/head, checkpoint cadence, command, metrics, and scientific
+acceptance.
+
+- Reference large durable datasets/checkpoints rather than copying partial data.
+- Prove data access, representative batch size, accelerator compatibility, and
+  checkpoint writing with the supplied bounded smoke test.
+- Preserve resolved inputs, logs, metrics, periodic checkpoints, runtime record,
+  and all attempts.
+- On interruption, return every checkpoint. The owning skill selects the valid
+  resume checkpoint and command.
+
+The platform layer may report progress and file growth, but must not choose the
+best checkpoint or interpret convergence.
+
+## Inference
+
+`deepmd-python-inference` owns the model artifact, input ordering, type mapping,
+command, batch size, declared predictions, and accuracy acceptance. Preserve
+ordering and case identity; return predictions, logs, runtime metadata, and
+failures without interpreting accuracy.
+
+## LAMMPS
+
+`lammps-deepmd` owns deployment, type mapping, LAMMPS inputs, ensemble,
+timestep, ranks, output cadence, restart policy, and simulation acceptance.
+Preserve trajectories, logs, final structures, and partial restarts without
+rewriting those scientific choices.
+
+## Return boundary
+
+Return case/attempt/job/group identity, platform selections and versions,
+input/output paths, scheduler/exit evidence, logs, retry lineage, and a platform
+verdict. The owning skill determines training, inference, MD, and scientific
+validity.