Clean up dpa_tools property workflow

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: claude

deepmd/dpa_tools/README.md

@@ -1,8 +1,10 @@
 # dpa_tools
 
-Fine-tuning, descriptor extraction, cross-validation, and data utilities for
-DPA-3 pretrained models.  Lives as a self-contained subpackage of `deepmd-kit`
-at `deepmd.dpa_tools`.
+Property-prediction tools built on top of DPA-3 pretrained models. `dpa_tools`
+turns molecular or atomistic structure data into `deepmd/npy` datasets, extracts
+DPA descriptors, and trains lightweight or fine-tuned property predictors for
+small- to medium-sized datasets. It lives as a self-contained subpackage of
+`deepmd-kit` at `deepmd.dpa_tools`.
 
 ## Relationship with deepmd-kit
 
@@ -16,7 +18,7 @@ at `deepmd.dpa_tools`.
   `dp --pt test`, auto-generating `input.json` config files.
 - **Inference**: deepmd-kit's built-in `DeepProperty` handles neural-network
   models; dpa_tools adds a lightweight frozen-descriptor + sklearn-head path.
-- **SMILES pipeline**: `data/smiles.py` converts CSV (SMILES or MOL files) +
+- **SMILES pipeline**: `data/smiles.py` converts CSV with SMILES columns +
   property labels into `deepmd/npy` format via RDKit 3D conformer generation.
 - **CLI**: registered as `dp dpa` subcommand group via `deepmd/main.py`.
   Torch and all DPA dependencies are loaded lazily — only when a `dp dpa ...`
@@ -30,15 +32,16 @@ at `deepmd.dpa_tools`.
 pip install deepmd-kit[dpa-tools]
 ```
 
-The `dpa-tools` extra brings in `scikit-learn`.  `torch` and `dpdata` are
-already provided by deepmd-kit's core dependencies.  For SMILES→3D conversion
-install RDKit (`conda install -c conda-forge rdkit`).
+The `dpa-tools` extra installs the Python dependencies used by this package,
+including `scikit-learn`, `dpdata`, `torch`, `rdkit`, and `e3nn`. For
+CUDA/GPU-specific PyTorch builds, install the desired PyTorch variant first or
+follow the PyTorch installation instructions for your platform.
 
 ## Python API
 
 ```python
 from deepmd.dpa_tools import (
-    DPAFineTuner,          # train (all strategies: frozen_sklearn, linear_probe, finetune, mft, scratch)
+    DPAFineTuner,          # train (strategies: frozen_sklearn, linear_probe, finetune, mft)
     DPAPredictor,          # read-only inference from frozen bundles
     extract_descriptors,   # standalone descriptor extraction
     cross_validate,        # leak-proof cross-validation
@@ -56,15 +59,14 @@ from deepmd.dpa_tools import (
 
 ### DPAFineTuner
 
-Four training strategies:
+Training strategies:
 
 | Strategy | Description | Best for |
 |----------|------------|----------|
 | `frozen_sklearn` | Freeze descriptor, extract once, fit sklearn head (RF/Ridge/MLP) | Small data (<1k samples), CPU inference |
 | `linear_probe` | Freeze backbone, train property fitting net only | Medium data, GPU |
 | `finetune` | Full-network fine-tuning | Larger data, GPU |
 | `mft` | Multi-task: property head + force-field head | Prevents representation collapse |
-| `scratch` | Train from random init (experimental) | Large-scale data only |
 
 ```python
 model = DPAFineTuner(
@@ -120,7 +122,14 @@ from deepmd.dpa_tools import auto_convert
 
 # CSV with SMILES → auto-detected, RDKit generates 3D coords
 result = auto_convert("data.csv", "./npy", property_name="homo", property_col="HOMO")
-# → {"method": "smiles", "train_systems": [...], "valid_systems": [...], ...}
+# prints: RDKit converted samples: ... / RDKit failed rows     : ...
+# → {"method": "smiles", "train_systems": [...], "valid_systems": [...],
+#    "samples_used": ..., "failed_rows": [...], "skipped_zero": ...,
+#    "skipped_overlap": ...}
+
+# To force the SMILES pipeline, pass fmt="smiles"; the value is case-insensitive
+# ("SMILES" and "Smiles" also work).
+result = auto_convert("data.csv", "./npy", fmt="SMILES", property_name="homo", property_col="HOMO")
 
 # Structure file → auto-detected by dpdata
 result = auto_convert("POSCAR", "./npy")
@@ -130,8 +139,6 @@ result = auto_convert("POSCAR", "./npy")
 Supports `.csv`, `.xlsx`, `.xls` for SMILES inputs and any format dpdata
 recognises for structure files (POSCAR, extxyz, cif, OUTCAR, …).
 
-A demo CSV and MOL files are included in `demo/`.
-
 ### Cross-validation
 
 Formula-grouped to prevent same-molecule leakage:
@@ -163,7 +170,7 @@ All commands live under `dp dpa` with two-level nesting:
 dp dpa
   extract-descriptors   extract pooled DPA descriptors to .npy
   fit                   train a model (any strategy)
-                        --strategy {frozen-sklearn|linear-probe|finetune|mft|scratch}
+                        --strategy {frozen_sklearn|linear_probe|finetune|mft}
   cv                    cross-validate (metric estimation, no model output)
   predict               predict with a frozen .pth bundle
   evaluate              evaluate a frozen .pth against stored labels
@@ -214,7 +221,6 @@ deepmd/dpa_tools/
 ├── trainer.py              # DPATrainer (dp --pt train subprocess wrapper)
 ├── cv.py                   # cross-validation + data splitting
 ├── conditions.py           # scalar condition manager (T, P)
-├── demo/                   # demo CSV + MOL files for the SMILES pipeline
 ├── config/
 │   └── manager.py          # MFT input.json generation
 ├── data/

deepmd/dpa_tools/cli.py

@@ -191,14 +191,16 @@ def _cmd_data_convert(args: argparse.Namespace) -> int:
         overwrite=args.overwrite,
         validate=args.validate,
         strict=args.strict,
+        verbose=False,
     )
     if result["method"] == "smiles":
         print(f"Train systems: {len(result['train_systems'])}")
         print(f"Valid systems: {len(result['valid_systems'])}")
         print(f"Type map     : {result['type_map']}")
         print(f"Samples used : {result['samples_used']}")
-        if result["failed_rows"]:
-            print(f"Failed rows  : {len(result['failed_rows'])}")
+        print(f"Failed rows  : {len(result['failed_rows'])}")
+        print(f"Skipped zero : {result['skipped_zero']}")
+        print(f"Skipped overlap: {result['skipped_overlap']}")
     else:
         _LOG.info("Wrote deepmd/npy → %s", result["output_dir"])
     return 0

deepmd/dpa_tools/cv.py

@@ -287,7 +287,7 @@ def cross_validate(
     extracted **once** and a cheap sklearn head is trained per fold — even
     ``cv=5`` completes in seconds.
 
-    Training paradigms (``linear_probe`` / ``finetune`` / ``scratch`` / ``mft``)
+    Training paradigms (``linear_probe`` / ``finetune`` / ``mft``)
     are expensive: each fold re-trains a full DeepMD model.  To prevent
     accidental hour-long runs, *allow_expensive_cv* must be explicitly set
     to ``True`` for those strategies when *cv* is an integer >= 2.  Otherwise
@@ -500,7 +500,7 @@ def cross_validate(
             # Phase 2 will wire this to DPATrainer / MFTFineTuner.
             raise NotImplementedError(
                 "cross_validate for training paradigms "
-                "(linear_probe / finetune / scratch / mft) is not yet "
+                "(linear_probe / finetune / mft) is not yet "
                 "implemented. Use frozen_sklearn for now."
             )
 
@@ -548,7 +548,6 @@ def _estimate_runtime(strategy: str, n_splits: int) -> str:
     per_run = {
         "linear_probe": "~5-15 min/run",
         "finetune": "~10-30 min/run",
-        "scratch": "~20-60 min/run",
         "mft": "~20-60 min/run",
     }.get(strategy, "unknown")
     return f"{n_splits} × {per_run}"

deepmd/dpa_tools/data/convert.py

@@ -26,15 +26,28 @@
 _SMILES_COLUMNS = frozenset({"smiles", "smi", "mol"})
 
 
-def _sniff_csv(path: str) -> set[str]:
+def _sniff_csv(path: str) -> set[str] | None:
     """Return the set of column names from a CSV file, or ``None`` if
     the file does not look like a table."""
     try:
         with open(path, newline="", encoding="utf-8") as fh:
             reader = csv.DictReader(fh)
             if reader.fieldnames is None:
                 return None
-            return {h.lower() for h in reader.fieldnames}
+
+            columns = []
+            for header in reader.fieldnames:
+                if header is None:
+                    return None
+                header = header.strip()
+                if not header:
+                    return None
+                # Reject binary/malformed files that csv.DictReader otherwise
+                # treats as a one-column header, e.g. b"\x00\x01\x02".
+                if any(ord(ch) < 32 for ch in header):
+                    return None
+                columns.append(header.lower())
+            return set(columns)
     except Exception:
         return None
 
@@ -87,6 +100,7 @@ def auto_convert(
     overwrite: bool = False,
     validate: bool = True,
     strict: bool = False,
+    verbose: bool = True,
 ) -> dict:
     """Convert any supported input to ``deepmd/npy``, auto-detecting the format.
 
@@ -103,7 +117,8 @@ def auto_convert(
     any additional metadata the chosen backend provides.
     """
     # --- explicit SMILES hint, or auto-sniff ---
-    if fmt == "smiles" or (fmt is None and _is_smiles_input(input_path)):
+    is_smiles_fmt = isinstance(fmt, str) and fmt.lower() == "smiles"
+    if is_smiles_fmt or (fmt is None and _is_smiles_input(input_path)):
         from deepmd.dpa_tools.data.smiles import smiles_to_npy
 
         result = smiles_to_npy(
@@ -116,14 +131,20 @@ def auto_convert(
             seed=seed,
             overwrite=overwrite,
         )
-        return {
+        converted = {
             "method": "smiles",
             "train_systems": result.train_systems,
             "valid_systems": result.valid_systems,
             "type_map": result.type_map,
             "samples_used": result.samples_used,
             "failed_rows": result.failed_rows,
+            "skipped_zero": result.skipped_zero,
+            "skipped_overlap": result.skipped_overlap,
         }
+        if verbose:
+            print(f"RDKit converted samples: {converted['samples_used']}")
+            print(f"RDKit failed rows     : {len(converted['failed_rows'])}")
+        return converted
 
     # --- structure file → dpdata ---
     out = convert(

deepmd/dpa_tools/finetuner.py

@@ -490,22 +490,8 @@ class DPAFineTuner:
                         fitting net via ``dp --pt train --finetune``.
     ``finetune``        Load the pretrained backbone and fine-tune the full
                         network (descriptor + fitting net).
-    ``scratch``         (known limitation) Random-initialize and train from
-                        scratch — type_map is auto-inferred correctly but
-                        ``dp --pt train`` exits before writing train.log;
-                        descriptor config likely missing required fields.
-                        Not recommended for small-data regimes.
     ==================  ======================================================
 
-    .. note::
-
-       ``strategy="scratch"`` is a known limitation as of Phase 2 closeout.
-       The entry point and auto-type_map logic are retained, but the emitted
-       ``input.json`` does not yet produce a successful ``dp --pt train`` run
-       (exit 1 before train.log).  Scratch training on 19-formula small data
-       has negligible practical value; completing it is deferred to a future
-       phase when larger datasets make random-init training meaningful.
-
     Refactored: descriptor-loading, feature-extraction, and sklearn-fitting
     logic extracted into ``_FrozenSklearnPipeline``.  DPAFineTuner is now a
     thin dispatcher that delegates to the pipeline for ``frozen_sklearn``
@@ -514,8 +500,7 @@ class DPAFineTuner:
     Parameters
     ----------
     pretrained : str
-        Path to the pretrained DPA checkpoint (.pt).  Set to ``None`` for
-        ``scratch`` strategy.
+        Path to the pretrained DPA checkpoint (.pt).
     model_branch : str, optional
         Branch name for multi-task checkpoints (e.g. ``"Omat24"``).  Used
         by ``frozen_sklearn`` for descriptor extraction.
@@ -529,7 +514,7 @@ class DPAFineTuner:
         Random seed for the sklearn predictor or training.
     strategy : str
         ``"frozen_sklearn"`` (default), ``"linear_probe"``, ``"finetune"``,
-        or ``"scratch"``.
+        or ``"mft"``.
     property_name : str
         Property label filename under ``set.*/`` (training paradigms).
     task_dim : int
@@ -554,7 +539,7 @@ class DPAFineTuner:
 
     _VALID_POOLING = {"mean", "sum", "mean+std", "mean+std+max+min"}
     _VALID_STRATEGIES = {
-        "frozen_sklearn", "linear_probe", "finetune", "mft", "scratch",
+        "frozen_sklearn", "linear_probe", "finetune", "mft",
     }
 
     def __init__(
@@ -600,9 +585,6 @@ def __init__(
             )
 
         self.strategy = strategy
-        # Scratch forces pretrained=None (random init, no ckpt).
-        if strategy == "scratch":
-            pretrained = None
 
         self.pretrained      = pretrained
         self.model_branch    = model_branch
@@ -737,8 +719,7 @@ def _resolve_type_maps(self, train_data) -> list[str]:
         *train_data* element set is a subset.
 
         Returns the checkpoint's type_map (e.g. 118-element full periodic
-        table for DPA-3.1-3M).  For scratch (``pretrained=None``) there is no
-        checkpoint — the type_map is the union of data ``atom_names``.
+        table for DPA-3.1-3M).
         """
         from deepmd.dpa_tools.data.type_map import (
             read_checkpoint_type_map,
@@ -750,27 +731,10 @@ def _resolve_type_maps(self, train_data) -> list[str]:
             systems = load_data(train_data)
         except DPADataError:
             # Data paths may not exist during testing; fall back gracefully.
-            if self.pretrained is None:
-                raise ValueError(
-                    "strategy='scratch' requires valid data paths or "
-                    "pass type_map=[...] explicitly."
-                )
             return read_checkpoint_type_map(
                 self.pretrained, branch=self.init_branch,
             )
 
-        if self.pretrained is None:
-            try:
-                tm = read_data_type_map_union(systems)
-            except ValueError:
-                raise ValueError(
-                    "strategy='scratch' requires atom_names in data "
-                    "systems, or pass type_map=[...] explicitly. "
-                    "Without a checkpoint, the global type_map cannot be "
-                    "auto-inferred."
-                )
-            return tm
-
         tm = read_checkpoint_type_map(
             self.pretrained, branch=self.init_branch,
         )
@@ -784,7 +748,7 @@ def _resolve_type_maps(self, train_data) -> list[str]:
         return tm
 
     # -------------------------------------------------------------------
-    # Training-paradigm fit (linear_probe / finetune / scratch)
+    # Training-paradigm fit (linear_probe / finetune)
     # -------------------------------------------------------------------
 
     def _fit_training(self, train_data, valid_data, type_map):
@@ -834,7 +798,7 @@ def fit(
         """Train the model.
 
         *frozen_sklearn* (default): extract descriptors, fit sklearn head.
-        *linear_probe* / *finetune* / *scratch*: run ``dp --pt train``.
+        *linear_probe* / *finetune*: run ``dp --pt train``.
         *mft*: multi-task fine-tuning (property head + force-field head).
 
         Parameters

deepmd/dpa_tools/trainer.py

@@ -210,8 +210,6 @@ def __init__(
     # ----- mode label (debugging convenience) -----
     @property
     def mode(self) -> str:
-        if self.pretrained is None:
-            return "Scratch"
         return "LP" if self.freeze_backbone else "FT"
 
     # ----- descriptor sourcing -----
@@ -238,9 +236,7 @@ def _get_descriptor(self) -> dict:
         else:
             descriptor = copy.deepcopy(DPA3_DESCRIPTOR_DEFAULT)
         # Paper alignment (qm9_gap input.json): silut:3.0 activation (alias of
-        # the ckpt's custom_silu:3.0) + explicit fix_stat_std=0.3. Enforced on
-        # both the ckpt-read and scratch paths so the emitted JSON matches the
-        # paper repo verbatim.
+        # the ckpt's custom_silu:3.0) + explicit fix_stat_std=0.3.
         descriptor["activation_function"] = "silut:3.0"
         descriptor["repflow"]["fix_stat_std"] = 0.3
         # LP: freeze the descriptor by setting trainable=False on the descriptor
@@ -410,8 +406,8 @@ def fit(self) -> str:
         Idempotency: training is skipped if a checkpoint at step
         ``>= max_steps`` exists in ``output_dir``. If ``max_steps`` is
         increased between runs (i.e. only a shorter checkpoint exists),
-        training is restarted from scratch (or from ``pretrained``) —
-        checkpoint resumption is not supported.
+        training is restarted from ``pretrained`` — checkpoint resumption is
+        not supported.
         """
         os.makedirs(self.output_dir, exist_ok=True)
 

deepmd/main.py

@@ -1027,7 +1027,7 @@ def main_parser() -> argparse.ArgumentParser:
                                 help="Path to DPA checkpoint (.pt).")
     parser_dpa_fit.add_argument("--model-branch", default=None)
     parser_dpa_fit.add_argument("--strategy", default="frozen_sklearn",
-                                choices=["frozen_sklearn", "linear_probe", "finetune", "mft", "scratch"])
+                                choices=["frozen_sklearn", "linear_probe", "finetune", "mft"])
     parser_dpa_fit.add_argument("--predictor", default="rf",
                                 choices=["rf", "linear", "ridge", "mlp"])
     parser_dpa_fit.add_argument("--pooling", default="mean",

pyproject.toml

@@ -94,6 +94,13 @@ test = [
     # to support Array API 2024.12
     'array-api-strict>=2.2;python_version>="3.9"',
 ]
+dpa-tools = [
+    "scikit-learn",
+    "dpdata",
+    "torch",
+    "rdkit",
+    "e3nn",
+]
 docs = [
     "sphinx>=3.1.1",
     "sphinx-book-theme",