Stratigraphy ingestion system

Makefile

@@ -19,7 +19,7 @@ format:
 test:
 	# These tests may fail due to an older GDAL version in use.
 	# We need to figure out how to bundle GDAL or run in a Docker context
-	uv run macrostrat test all --skip-env -x -s
+	uv run macrostrat test all --skip-env -s -v
 
 test-ci:
 	# We need a fairly recent version of GDAL (3.10) for map integration tests to pass.

README.md

@@ -59,6 +59,59 @@ presents a broad list of management functionality for Macrostrat's system. **Sub
 This CLI is rapidly evolving so expect breakage! When in doubt run `make`, or ~equivalently
 `uv sync` to update your installation.
 
+## Testing
+
+The `macrostrat test` command is a thin wrapper around `pytest`.
+The `all` subcommand forwards extra arguments directly to `pytest`, so you can
+mix standard pytest flags (`-x`, `-k`, `-m`, etc.) with Macrostrat-specific options
+defined in `conftest.py`.
+
+### Test modes
+
+- **Environment/conformance tests** use the `env_config`, `env_db`, and `db` fixtures.
+  They target a configured Macrostrat environment from your `macrostrat.toml`.
+- **Clean-room database tests** use `empty_db` and `test_db`.
+  They create a temporary PostgreSQL cluster, apply schema, and run tests against it.
+- **Unit-style tests** do not require either database fixture.
+
+### Macrostrat-specific pytest options
+
+- `--skip-env`: skip environment-backed tests that require `env_config`/`env_db`/`db`. This is the default
+  for CI runs.
+- `--env ENV`: override the active Macrostrat environment for the test run.
+- `--skip-database`: skip creation of the temporary clean-room database.
+- `--skip-slow`: skip tests marked `@pytest.mark.slow`.
+- `--optimize-database`: (on by default) enable faster schema setup for clean-room tests by skipping
+  non-essential statements (indexes, grants, and ownership changes).
+
+### Fixture behaviors
+
+- `env_db` connects to the database for the active environment, then sets `ROLE web_anon` to ensure that tests are read-only.
+- `db` wraps each test class in a transaction and rolls it back. Environment-backed tests should
+  not change the database so this is purely a precautionary measure.
+- `test_db` applies schema for the current environment. with transactional rollback as well.
+
+### Common commands
+
+```bash
+# Full suite (environment + clean-room + unit tests)
+macrostrat test all
+
+# Local/CI-friendly run without environment-backed tests (same intent as `make test`)
+macrostrat test all --skip-env -x -s
+
+# Focus on fast tests only
+macrostrat test all --skip-env --skip-database --skip-slow
+
+# Target a specific environment for conformance tests
+macrostrat test all --env development
+```
+
+For marker-based filtering, the repository also defines `docker` and `requires_gdal`
+pytest markers in `pyproject.toml`.
+
+
+
 ## Documentation
 
 Documentation is a work in progress. We have starting points for:

conftest.py

@@ -1,14 +1,14 @@
 """Basic tests that the CLI runs without crashing."""
 
 import importlib
+from os import environ
 from pathlib import Path
 
-import docker
 from pytest import fixture, mark, skip
 from typer.testing import CliRunner
 
 from macrostrat.database import Database
-from macrostrat.dinosaur.upgrade_cluster import database_cluster
+from macrostrat.schema_management.defs import test_database_cluster
 from macrostrat.utils import get_logger, override_environment
 
 runner = CliRunner()
@@ -20,10 +20,10 @@
 
 def pytest_addoption(parser):
     parser.addoption(
-        "--skip-test-database",
+        "--skip-database",
         action="store_true",
         default=False,
-        help="skip test database creation",
+        help="skip local database creation",
     )
 
     parser.addoption(
@@ -44,6 +44,13 @@ def pytest_addoption(parser):
         help="skip slow tests",
     )
 
+    parser.addoption(
+        "--optimize-database",
+        action="store_true",
+        default=True,
+        help="optimize database for fast testing",
+    )
+
 
 def pytest_collection_modifyitems(config, items):
     if config.getoption("--skip-slow"):
@@ -80,6 +87,9 @@ def env_config(request):
         # Print the current environment to the PyTest output
         log.info("Current env: %s", mod_instance.settings.env)
 
+        if mod_instance.settings.env is None:
+            skip("No environment configured")
+
         yield mod_instance.settings
 
 
@@ -91,7 +101,7 @@ def env_config(request):
 
 # TODO: ensure that tests on "live" environments are read-only by connecting to a read-only user.
 @fixture(scope="session")
-def db(env_config):
+def env_db(env_config):
     """The actually operational database for the current environment."""
 
     if env_config is None:
@@ -101,55 +111,73 @@ def db(env_config):
         skip("No database configured for this environment")
 
     db = Database(env_config.pg_database)
-    # Change the user on the connection
-    # db.run_sql("SET ROLE macrostrat_reader;")
+
+    # Change the user on the connection to a read-only user
+    # TODO: verify read-only
+    db.run_sql("SET ROLE web_anon;")
 
     yield db
 
 
+@fixture(scope="class")
+def db(env_db):
+    with env_db.transaction(rollback=True):
+        yield env_db
+
+
 def load_config_module():
     mod_instance = importlib.util.module_from_spec(module_spec)
     module_spec.loader.exec_module(mod_instance)
     return mod_instance
 
 
-@fixture()
-def cfg():
-    cfg_file = __here__ / "macrostrat" / "cli" / "tests" / "macrostrat.test.toml"
-    with override_environment(MACROSTRAT_CONFIG=str(cfg_file), NO_COLOR="1"):
-        mod_instance = load_config_module()
-
-        assert cfg_file == mod_instance.settings.config_file
-        yield mod_instance.settings
-
-
 @fixture(scope="session")
-def test_db(request):
+def empty_db(request):
     """A temporary, initially empty database for Macorstrat testing."""
     # Get the current settings without an override
-    cfg = load_config_module().settings
-    if request.config.getoption("--skip-test-database"):
-        import pytest
-
-        pytest.skip("skipping Docker test database")
-
-    # Spin up a docker container with a temporary database using the
-    # pg_database_container image
-
-    image = cfg.get("pg_database_container", "postgres:15")
+    if request.config.getoption("--skip-database"):
+        skip("skipping Docker test database")
 
-    client = docker.from_env()
+    optimize = request.config.getoption("--optimize-database")
 
-    img_root = cfg.srcroot / "base-images" / "database"
-
-    # Build postgres pgaudit image
-    img_tag = "macrostrat-local-database:latest"
+    with test_database_cluster(username="macrostrat_admin", optimize=optimize) as db:
+        yield db
 
-    client.images.build(path=str(img_root), tag=img_tag)
 
-    # Spin up an image with this container
-    port = 54884
-    with database_cluster(client, img_tag, port=port) as container:
-        url = f"postgresql://postgres@localhost:{port}/postgres"
-        db = Database(url)
-        yield db
+@fixture(scope="session")
+def test_db(request, empty_db: Database):
+    """The database used for testing."""
+    from macrostrat.core.config import settings
+    from macrostrat.schema_management import apply_schema_for_environment
+
+    _filter = lambda s, p: True
+
+    if request.config.getoption("--optimize-database"):
+        # If we're optimizing the database, we want to skip any statements that are not necessary for testing.
+        # This is a bit hacky, but it allows us to significantly speed up the tests by skipping things like
+        # indexes, constraints, and permissions that are not necessary for mist tests.
+        def _filter(statement: str, path: Path):
+            stmt = statement.strip().lower()
+            if (
+                stmt.startswith("create index")
+                or stmt.startswith("create unique index")
+                or statement.startswith("alter index")
+            ):
+                return False
+
+            # Modify ownership of tables
+            if stmt.startswith("alter table") and "owner to" in stmt:
+                return False
+
+            if stmt.startswith("grant"):
+                return False
+
+            return True
+
+    apply_schema_for_environment(
+        empty_db,
+        env=settings.env or "development",
+        statement_filter=_filter,
+        suppress_logging=True,
+    )
+    return empty_db

py-modules/cli/macrostrat/cli/entrypoint.py

@@ -460,7 +460,10 @@ def show_app_dir():
 
 
 @self_app.command()
-def state():
+def state(clear: bool = False):
+    if clear:
+        app.state.clear()
+
     """Show the current state of the application"""
     app.console.print(app.state.get())
 

py-modules/cli/macrostrat/cli/subsystems/rebuild/__init__.py

@@ -3,60 +3,90 @@
 
 from macrostrat.core.database import get_database
 
-from .scripts import grant_permissions
+from .utils import grant_permissions
+
+cli = Typer(help="Rebuild scripts tools", no_args_is_help=True)
 
-cli = Typer(help="Rebuild database tools")
 console = Console()
 
 
 def _get_scripts() -> dict:
     from .scripts import (
-        Autocomplete,
-        LookupStratNames,
-        LookupUnitAttrsApi,
-        LookupUnitIntervals,
-        LookupUnits,
-        Stats,
-        StratNameFootprints,
-        UnitBoundaries,
+        autocomplete,
+        lookup_strat_names,
+        lookup_unit_attrs_api,
+        lookup_unit_intervals,
+        lookup_units,
+        stats,
+        strat_name_footprints,
+        unit_boundaries,
     )
 
-    return {
-        "autocomplete": Autocomplete,
-        "lookup-strat-names": LookupStratNames,
-        "lookup-unit-attrs-api": LookupUnitAttrsApi,
-        "lookup-unit-intervals": LookupUnitIntervals,
-        "lookup-units": LookupUnits,
-        "stats": Stats,
-        "strat-name-footprints": StratNameFootprints,
-        "unit-boundaries": UnitBoundaries,
+    commands = {
+        "autocomplete": autocomplete,
+        "lookup-strat-names": lookup_strat_names,
+        "lookup-unit-attrs-api": lookup_unit_attrs_api,
+        "lookup-unit-intervals": lookup_unit_intervals,
+        "lookup-units": lookup_units,
+        "stats": stats,
+        "strat-name-footprints": strat_name_footprints,
+        "unit-boundaries": unit_boundaries,
     }
 
+    return {k: wrap_command(v) for k, v in commands.items()}
+
+
+mark_slow = ("strat-name-footprints",)
+
+
+def is_column_script(name):
+    return "unit" in name
 
-@cli.command()
-def scripts(
-    name: str | None = Option(
-        None, "--name", "-n", help="Run a specific script by name"
-    ),
-    list_: bool = Option(False, "--list", "-l", help="List available scripts"),
-):
-    """Run rebuild scripts."""
-    all_scripts = _get_scripts()
-
-    if list_:
-        for n in all_scripts:
-            console.print(f"  [cyan]{n}[/]")
-        return
-
-    if name and name not in all_scripts:
-        raise RuntimeError(
-            f"No script named '{name}'. Available: {', '.join(all_scripts)}"
-        )
-    to_run = {name: all_scripts[name]} if name else all_scripts
-
-    for script_name, cls in to_run.items():
-        console.print(f"[bold cyan]→ {script_name}[/]")
-        cls().run()
+
+def wrap_command(command_func):
+    """Decorator to wrap command functions with logging and permission granting"""
+
+    def wrapper():
+        command_func()
         console.print(f"[dim]  granting permissions...[/]")
         grant_permissions(get_database())
-        console.print(f"[green]✓ done[/]")
+        console.print(f"[green]✓ done[/]\n")
+
+    wrapper.__doc__ = command_func.__doc__
+
+    return wrapper
+
+
+_scripts = _get_scripts()
+
+for name, cls in _scripts.items():
+    short_help = cls.__doc__.splitlines()[0]
+    if name in mark_slow:
+        short_help += " [dim red](slow)[/]"
+    cli.command(name=name, help=short_help)(cls)
+
+
+column_help = "Run column-related rebuild scripts:\n" + "\n".join(
+    [f"    [dim]- {name}[/]" for name in _scripts.keys() if is_column_script(name)]
+)
+
+
+@cli.command(
+    rich_help_panel="Meta",
+    name="columns",
+    help=column_help,
+)
+def columns():
+    """Run column-related rebuild scripts."""
+    for name, command in _scripts.items():
+        if is_column_script(name):
+            console.print(f"[bold cyan]→ {name}[/]")
+            command()
+
+
+@cli.command(rich_help_panel="Meta", name="all", help="Run all rebuild scripts")
+def all():
+    """Run all rebuild scripts."""
+    for name, command in _scripts.items():
+        console.print(f"[bold cyan]→ {name}[/]")
+        command()