Merge pull request #53 from microbiomedata/mode-and-closure-cli

Add `--mode` / `--closure` CLI; deprecate `output_directory` + `generate_reports` kwargs (closes #49)

Author: sierra-moxon

CHANGELOG.md

@@ -0,0 +1,59 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Added
+- `OntologyLoaderController(mode=...)` kwarg accepting `'meticulous'` (default) or `'fast-initial'`.
+  - `meticulous` preserves 0.2.x behavior exactly: pure linkml-store, per-item upsert, force-refresh of the pystow cache, TSV reports written to `report_directory`.
+  - `fast-initial` is the new maximum-throughput first-time-install path: raw pymongo `insert_many(ordered=False)`, no pre-read, no upsert, no report tracking, no TSV writes. Reuses the pystow cache if present.
+- `OntologyLoaderController(closure=...)` kwarg accepting a string or list. Values: `combined` (default), `isa`, `partof`, `all` (exclusive shorthand for combined+isa+partof), `none` (exclusive — emit no ancestry closure).
+- `OntologyLoaderController(report_directory=...)` kwarg — renamed from `output_directory`.
+- `OntologyLoaderController(source_ontology=...)` now accepts a list of strings as well as a single string. Multiple ontologies are processed sequentially in the given order; failure on one halts the run.
+- New CLI surface — four flags total: `--source-ontology` (repeatable, required), `--report-directory`, `--mode`, `--closure` (repeatable).
+- `MongoDBLoader.insert_ontology_data_fast_initial(...)` — the raw-pymongo write method used by `mode='fast-initial'`.
+- `OntologyProcessor(force_refresh=...)` constructor kwarg — `True` (default) preserves 0.2.x cache-wiping behavior; `False` reuses the cached pystow artifact when present.
+
+### Changed
+- `OntologyProcessor.get_relations_closure()` signature: `predicates=` removed in favor of `closure=` (string or list of strings). The old hardcoded ancestry-relation name (`entailed_isa_partof_closure`) is now selected per-closure: `entailed_isa_partof_closure`, `entailed_isa_closure`, or `entailed_partof_closure`.
+- CLI no longer takes `--generate-reports`. Equivalents under the new design:
+  - Old `--generate-reports true` (the implicit default) → new `--mode meticulous` (the default).
+  - Old `--generate-reports false` → new `--mode fast-initial`.
+
+### Deprecated
+- `OntologyLoaderController(output_directory=...)` — use `report_directory=` instead. The old kwarg is an alias and emits `DeprecationWarning`. Passing both raises `ValueError`.
+- `OntologyLoaderController(generate_reports=True)` — no-op with `DeprecationWarning`. (True was always the default.)
+- `OntologyLoaderController(generate_reports=False)` — maps to `mode='fast-initial'` with `DeprecationWarning`. If `mode` was also passed and isn't `'meticulous'`, raises `ValueError`.
+
+### Removed
+- (nothing removed in this release — all 0.2.x kwargs continue to work as deprecated aliases. Removal slated for the next major release after downstream callers — notably the `nmdc-runtime` Dagster job — migrate.)
+
+### Migration
+
+The exact 0.2.x call site in `nmdc-runtime`'s Dagster op (`nmdc_runtime/site/ops.py`, `load_ontology`) — `OntologyLoaderController(source_ontology=str, output_directory=str, generate_reports=True, mongo_client=..., db_name=...)` — runs **unchanged** under this release. Two `DeprecationWarning` lines appear in the Dagster logs as a nudge to update.
+
+To migrate when convenient:
+
+```python
+# 0.2.x
+OntologyLoaderController(
+    source_ontology="envo",
+    output_directory="/tmp/ontology_reports",
+    generate_reports=True,
+    mongo_client=client,
+    db_name="nmdc",
+)
+
+# 0.3.0+
+OntologyLoaderController(
+    source_ontology="envo",        # or ["envo", "po", "uberon"]
+    report_directory="/tmp/ontology_reports",
+    mode="meticulous",             # default; or "fast-initial"
+    closure="combined",            # default; or "isa", "partof", "all", "none", or a list
+    mongo_client=client,
+    db_name="nmdc",
+)
+```

README.md

@@ -75,13 +75,36 @@ The Docker container networking uses container names (like 'mongo') for internal
 ``` 
 
 #### Command line
+
 ```bash
 % poetry install
 % poetry run ontology_loader --help
-% poetry run ontology_loader --source-ontology "envo"
-% poetry run ontology_loader --source-ontology "uberon"
+% poetry run ontology_loader --source-ontology envo
+% poetry run ontology_loader --source-ontology envo --source-ontology po --source-ontology uberon
 ```
 
+Four flags:
+
+- `--source-ontology <name>` — required, repeatable. Lowercase prefix (envo, po, uberon, ncbitaxon, …). Multiple ontologies are processed sequentially in the given order.
+- `--report-directory <dir>` — TSV report destination (only used in `meticulous` mode). Defaults to a fresh temp directory.
+- `--mode {meticulous|fast-initial}` — default `meticulous`. See "Modes" below.
+- `--closure {combined|isa|partof|all|none}` — default `combined`. Repeatable; values combine. `all` and `none` are exclusive.
+
+##### Modes
+
+- **`meticulous`** (default): Preserves 0.2.x behavior — pure linkml-store, per-item upsert, force-refresh of the pystow cache on every run, TSV reports (`ontology_updates.tsv`, `ontology_inserts.tsv`, `ontology_relation_inserts.tsv`) written to `--report-directory`. Use this for incremental updates of an already-loaded ontology.
+- **`fast-initial`**: Maximum-throughput first-time install. Raw pymongo `insert_many(ordered=False)`, no upsert, no pre-read, no report tracking, no TSV writes. Reuses the pystow cache if present (downloads only when missing). Use this when the target collections are empty or duplicate-key errors are acceptable. Expected ~3-5x faster than `meticulous` on large ontologies (e.g. NCBITaxon's 2.7M classes + 54.7M relations).
+
+##### Closure shorthands
+
+- `--closure combined` (default): emits `entailed_isa_partof_closure` (rdfs:subClassOf ∪ BFO:0000050).
+- `--closure isa`: emits `entailed_isa_closure` (rdfs:subClassOf only).
+- `--closure partof`: emits `entailed_partof_closure` (BFO:0000050 only).
+- `--closure all`: shorthand for `--closure combined --closure isa --closure partof`. Exclusive.
+- `--closure none`: emit no ancestry closure, only direct relationships. Exclusive.
+
+Repeat the flag to combine specific closures: `--closure isa --closure partof` emits both `entailed_isa_closure` and `entailed_partof_closure`.
+
 #### Running the tests
 ```bash
 % make test
@@ -92,7 +115,8 @@ The Docker container networking uses container names (like 'mongo') for internal
 % make lint
 ```
 
-#### Python example usage
+#### Python API
+
 ```bash
 pip install nmdc-ontology-loader
 ```
@@ -101,48 +125,68 @@ pip install nmdc-ontology-loader
 from ontology_loader.ontology_load_controller import OntologyLoaderController
 import tempfile
 
-def load_ontology():
-    """Load an ontology using the default MongoDB connection."""
-    loader = OntologyLoaderController(
-        source_ontology="envo",
-        output_directory=tempfile.gettempdir(),
-        generate_reports=True,
-    )
-    loader.run_ontology_loader()
+# Default: pure linkml-store + TSV reports (preserves 0.2.x behavior)
+OntologyLoaderController(
+    source_ontology="envo",                          # str or list[str]
+    report_directory=tempfile.gettempdir(),          # only used in 'meticulous' mode
+    mode="meticulous",                               # or 'fast-initial'
+    closure="combined",                              # str or list[str]
+).run_ontology_loader()
 ```
 
-#### Using with an existing MongoDB connection
+##### Fast first-time install of a large ontology
 
-If you already have a MongoDB connection established (e.g., in a Dagster/Dagit job), you can pass it directly to the OntologyLoaderController:
+```python
+OntologyLoaderController(
+    source_ontology="ncbitaxon",
+    mode="fast-initial",        # raw pymongo insert_many, no upsert, no reports
+    closure="isa",              # is_a only; combined closure is too large for NCBITaxon
+).run_ontology_loader()
+```
+
+##### Multiple ontologies in one invocation
+
+```python
+OntologyLoaderController(
+    source_ontology=["envo", "po", "uberon"],   # processed sequentially in given order
+    mode="meticulous",
+).run_ontology_loader()
+```
+
+##### Using with an existing MongoDB connection
+
+If you already have a MongoDB connection (e.g., in a Dagster/Dagit job), pass it directly:
 
 ```python
 from pymongo import MongoClient
 from ontology_loader.ontology_load_controller import OntologyLoaderController
-import tempfile
 
-# Use an existing MongoDB client
 mongo_client = MongoClient("mongodb://admin:password@localhost:27018/nmdc?authSource=admin")
 
-# Pass the client and database name to OntologyLoaderController
-loader = OntologyLoaderController(
+OntologyLoaderController(
     source_ontology="envo",
-    output_directory=tempfile.gettempdir(),
-    generate_reports=True,
-    mongo_client=mongo_client,  # Pass the existing client
-    db_name="nmdc",  # Required when passing an existing client
-)
-
-# The loader will use the provided client instead of creating a new connection
-loader.run_ontology_loader()
+    mode="meticulous",
+    mongo_client=mongo_client,   # Pass the existing client
+    db_name="nmdc",              # Required when passing an existing client
+).run_ontology_loader()
 ```
 
-This approach is particularly useful when:
-- You're running in a job scheduler like Dagster/Dagit
-- You want to reuse an existing connection pool
-- You have custom MongoDB connection settings that are managed externally
-- You need to use a connection with specific authentication or configuration
+> **Note**: When passing an existing MongoDB client, you must also provide `db_name`. The database name cannot be auto-determined from a MongoClient instance.
+
+#### Migrating from 0.2.x
+
+The 0.2.x constructor signature (`source_ontology`, `output_directory`, `generate_reports`, `mongo_client`, `db_name`) continues to work as deprecated aliases. The exact call site in nmdc-runtime's Dagster `load_ontology` op runs unchanged under 0.3.0; two `DeprecationWarning` lines appear in the logs as a nudge.
+
+| old kwarg | new kwarg | behavior |
+|---|---|---|
+| `source_ontology=<str>` | `source_ontology=<str \| list[str]>` | unchanged; now also accepts a list |
+| `output_directory=<str>` | `report_directory=<str>` | renamed; old kwarg works as alias with `DeprecationWarning`. Passing both raises. |
+| `generate_reports=True` | (gone — implicit) | no-op with `DeprecationWarning` (True was always the default) |
+| `generate_reports=False` | `mode='fast-initial'` | mapped with `DeprecationWarning`. If `mode` was also passed and isn't `'meticulous'`, raises. |
+| (none) | `mode='meticulous'` (default) | new; default preserves 0.2.x write path |
+| (none) | `closure='combined'` (default) | new; default preserves 0.2.x ancestry behavior |
 
-> **Note**: When passing an existing MongoDB client, you must also provide the `db_name` parameter to specify which database to use. This is required as the database name cannot be automatically determined from a MongoDB client instance.
+See `CHANGELOG.md` for the full release note and a side-by-side migration code sample.
 
 ### Testing CRUD operations in a live MongoDB
 

src/ontology_loader/cli.py

@@ -1,37 +1,84 @@
-"""Cli methods for ontology loading from the command line."""
+"""Command-line interface for the ontology loader."""
 
 import logging
 
 import click
 
-from ontology_loader.ontology_load_controller import OntologyLoaderController
+from ontology_loader.ontology_load_controller import VALID_MODES, OntologyLoaderController
+from ontology_loader.ontology_processor import VALID_CLOSURES
 
 logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s")
 logger = logging.getLogger(__name__)
 
 
 @click.command()
-@click.option("--source-ontology", default="envo", help="Lowercase ontology prefix, e.g., envo, go, uberon, etc.")
-@click.option("--output-directory", default=None, help="Output directory for reporting, default is /tmp")
-@click.option("--generate-reports", default=True, help="Generate reports")
-def cli(source_ontology, output_directory, generate_reports):
-    """
-    CLI entry point for the ontology loader.
+@click.option(
+    "--source-ontology",
+    "source_ontologies",
+    multiple=True,
+    required=True,
+    help=(
+        "Lowercase ontology prefix (e.g., envo, po, uberon, ncbitaxon). "
+        "Pass repeatedly to load multiple ontologies in one invocation: "
+        "--source-ontology envo --source-ontology po --source-ontology uberon. "
+        "Processed sequentially in the given order; fail-fast on per-ontology error."
+    ),
+)
+@click.option(
+    "--report-directory",
+    default=None,
+    help=("Directory for TSV reports (only used when --mode=meticulous). " "Defaults to the system temp directory."),
+)
+@click.option(
+    "--mode",
+    type=click.Choice(VALID_MODES, case_sensitive=False),
+    default="meticulous",
+    show_default=True,
+    help=(
+        "meticulous: pure linkml-store, per-item upsert, TSV reports; matches 0.2.x behavior. "
+        "fast-initial: raw pymongo insert_many, no upsert, no reports; for first-time installs "
+        "of large ontologies."
+    ),
+)
+@click.option(
+    "--closure",
+    "closures",
+    type=click.Choice(VALID_CLOSURES, case_sensitive=False),
+    multiple=True,
+    default=("combined",),
+    show_default=True,
+    help=(
+        "Which ancestry closures to emit. Repeatable; values combine. "
+        "combined = rdfs:subClassOf + BFO:0000050 (entailed_isa_partof_closure); "
+        "isa = rdfs:subClassOf only; partof = BFO:0000050 only; "
+        "all = shorthand for combined + isa + partof (exclusive); "
+        "none = no ancestry closure, direct relationships only (exclusive). "
+        "Example: --closure isa --closure partof emits both."
+    ),
+)
+def cli(source_ontologies, report_directory, mode, closures):
+    r"""
+    Load one or more ontologies into MongoDB.
+
+    Set MongoDB connection details via MONGO_HOST, MONGO_PORT, MONGO_USERNAME, MONGO_PASSWORD, MONGO_DB.
 
-    :param source_ontology: Lowercase ontology prefix, e.g., envo, go, uberon, etc.
-    :param output_directory: Output directory for reporting, default is /tmp
-    :param generate_reports: Generate reports or not, default is True
+    Behavior contract:
 
-    Set the parameters for the connection to mongodb in the environment variables MONGO_HOST, MONGO_PORT,
-    MONGO_USER, MONGO_PASSWORD, MONGO_DB.
+    \b
+    - --mode meticulous (default): preserves Sierra/0.2.x behavior — pure linkml-store
+      per-item upsert, force-refresh of the pystow cache, TSV reports written to
+      --report-directory.
+    - --mode fast-initial: maximum-throughput first-time install — raw pymongo
+      insert_many, no upsert, no reporting, reuses the pystow cache if present.
     """
-    logger.info(f"Processing ontology: {source_ontology}")
+    closure_arg = list(closures)
+    logger.info(f"Processing ontologies: {list(source_ontologies)} (mode={mode}, closure={closure_arg})")
 
-    # Initialize the MongoDB Loader
     loader = OntologyLoaderController(
-        source_ontology=source_ontology,
-        output_directory=output_directory,
-        generate_reports=generate_reports,
+        source_ontology=list(source_ontologies),
+        report_directory=report_directory,
+        mode=mode,
+        closure=closure_arg,
     )
     loader.run_ontology_loader()
 

src/ontology_loader/mongo_db_config.py

@@ -60,6 +60,7 @@ def set_existing_client(self, client):
         When set, this client will be used instead of creating a new connection.
 
         Args:
+        ----
             client: An existing pymongo.MongoClient instance
 
         """
@@ -69,7 +70,8 @@ def has_existing_client(self):
         """
         Check if an existing MongoDB client has been set.
 
-        Returns:
+        Returns
+        -------
             bool: True if an existing client is available, False otherwise
 
         """