Merge pull request #173 from ContextLab/work/priorities-and-docs

v0.2.0 (honest beta): merge train complete, gates green, real jobs verified

Author: jeremymanning

.claude/epics/clean-up-repository/updates/82/stream-C.md

@@ -1,15 +1,25 @@
 name: Fast CI
 
 on:
+  # No `paths:` filter on pull_request, deliberately.
+  #
+  # The `status-check` job below publishes the `CI Status` context, and
+  # master's branch protection lists that context as required. A required
+  # check that is never reported is not treated as passing -- GitHub blocks
+  # the merge on "Expected -- Waiting for status to be reported", forever --
+  # so while this trigger was path-filtered, a pull request touching only
+  # docs/, README.md or the notebooks could never be merged by anyone
+  # without an admin override (#169). A required check whose reporting
+  # depends on which files changed is a trap; the filter is not worth it.
+  #
+  # This repository is public, so Actions minutes are free and the cost of
+  # running the four jobs on a docs-only pull request is wall-clock time,
+  # not money.
   pull_request:
     branches: [main, master, develop]
-    paths:
-      - 'clustrix/**'
-      - 'tests/**'
-      - 'setup.py'
-      - 'pyproject.toml'
-      - 'requirements*.txt'
   push:
+    # The filter stays here. `CI Status` is not a required check for pushes
+    # to develop, so a run that never happens blocks nothing.
     branches: [develop]  # Only run on develop pushes to avoid duplication with main Tests workflow
     paths:
       - 'clustrix/**'
@@ -65,7 +75,11 @@ jobs:
             -x \
             --tb=short \
             --maxfail=3
-        timeout-minutes: 5
+        # Sized for the suite as it is now (~1,100 unit tests, several of
+        # which execute notebooks and real subprocesses): the old 5-minute
+        # step timeout was set when tests/unit held ~350 fast tests and cut
+        # the run at 60% regardless of what passed before it.
+        timeout-minutes: 15
 
   local-integration:
     name: Local Integration Test

.github/workflows/fast_ci.yml

@@ -92,6 +92,41 @@ jobs:
         # Test SSH connection
         timeout 10 ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${{ secrets.CLUSTRIX_USERNAME }}@localhost "echo 'SSH connection successful'"
 
+    - name: Populate known_hosts for host key verification
+      # Issue #148: the real-world tests no longer call
+      # `set_missing_host_key_policy(paramiko.AutoAddPolicy())`; they go
+      # through `clustrix.ssh_security.configure_host_key_policy`, whose
+      # default policy is "reject". Any host absent from known_hosts now
+      # raises HostKeyVerificationError instead of being trusted silently,
+      # so every host these tests connect to must be scanned in first.
+      #
+      # localhost/127.0.0.1 is the sshd this job installs a few steps above,
+      # and is the only host the CI run actually reaches: `test_ssh_real.py`
+      # skips itself unless the configured host is localhost.
+      #
+      # There is deliberately no `secrets.*` reference for an external
+      # cluster hostname here, because no such repository secret exists --
+      # the only cluster secrets configured are CLUSTRIX_USERNAME,
+      # CLUSTRIX_PASSWORD, HF_USERNAME and HF_TOKEN. Real cluster hosts are
+      # supplied to the suite through the CLUSTRIX_TEST_{SSH,SLURM}_HOST[_2]
+      # environment variables (see tests/real_world/credential_manager.py);
+      # this step scans whichever of those are set so that adding them later
+      # needs no further workflow change.
+      run: |
+        mkdir -p ~/.ssh
+        chmod 700 ~/.ssh
+        for host in localhost 127.0.0.1 \
+                    "$CLUSTRIX_TEST_SSH_HOST" "$CLUSTRIX_TEST_SSH_HOST_2" \
+                    "$CLUSTRIX_TEST_SLURM_HOST" "$CLUSTRIX_TEST_SLURM_HOST_2"; do
+          [ -n "$host" ] || continue
+          echo "Scanning host key for $host"
+          ssh-keyscan -H "$host" >> ~/.ssh/known_hosts
+        done
+        chmod 600 ~/.ssh/known_hosts
+        # Fail loudly rather than let the tests fail later with an opaque
+        # HostKeyVerificationError for the host this job just created.
+        ssh-keygen -F localhost -f ~/.ssh/known_hosts > /dev/null
+
     - name: Run filesystem tests
       run: |
         python scripts/run_real_world_tests.py --filesystem

.github/workflows/real-world-tests.yml

@@ -21,7 +21,7 @@ jobs:
     # This is a bound on a genuinely longer job, not a relaxed check: every
     # test still has to pass, and pytest's own --timeout=120 still bounds any
     # individual test that wedges.
-    timeout-minutes: 30
+    timeout-minutes: 45
     strategy:
       # Show every platform's failures in one run. With the default
       # fail-fast, one job failing cancelled the other six, so a
@@ -212,7 +212,15 @@ jobs:
       run: |
         cd docs
         make html
-        
+
+    # Sphinx inline markup does not nest: a ``literal`` inside a **bold**
+    # span renders as plain text, backticks and all, and `sphinx -W` builds
+    # it happily. Only the built HTML shows it, and only the built HTML has
+    # the .rst, the docstring and the nbsphinx-converted notebook cases in
+    # one place -- so this checks the output of the step above.
+    - name: Check documentation for nested inline markup
+      run: python scripts/check_docs_markup.py docs/build/html
+
     - name: Test notebook execution
       run: |
         pip install jupyter nbconvert

.github/workflows/tests.yml

@@ -60,8 +60,17 @@ docs/build/
 **/clustrix-*.key
 **/*-credentials.json
 **/*-service-account.json
-.env.local
-.env.validation
+# Credential-bearing dotenv files (see #111). `clustrix credentials setup`
+# writes ~/.clustrix/.env, but an operator following the docs can easily end
+# up with one in the working tree, and `git add .` would have committed it.
+.env
+.env.*
+# ...except a checked-in, secret-free example, if one is ever added.
+!.env.example
+# direnv's file. `.env.*` above does not match it -- no dot after "env" --
+# and it routinely holds exported credentials.
+.envrc
+.envrc.*
 **/validation-secrets.json
 **/.op/
 **/op-session-*
@@ -76,3 +85,4 @@ docs/build/
 tests/real_world/screenshots/
 tests/real_world/temp/
 .omc/
+.omo/run-continuation/

.gitignore

@@ -0,0 +1,103 @@
+# PROJECT KNOWLEDGE BASE
+
+**Generated:** 2026-08-21 16:23 UTC
+**Commit:** 7d28428
+**Branch:** work/priorities-and-docs
+
+## OVERVIEW
+
+Clustrix is a Python distributed computing framework: `@cluster` on a function serializes it (dill/cloudpickle, by value) and runs it on a configured backend — `local`, `ssh`, `slurm`, `huggingface` (HF Jobs). Those four are the whole list (`clustrix.config.SUPPORTED_CLUSTER_TYPES`); pbs/sge/kubernetes/AWS/GCP/Azure/Lambda raise `ValueError` (issues #140–#146). Python >=3.10, version 0.2.0, beta.
+
+**CLAUDE.md is the deep curated knowledge base** (architecture, security invariants, mocking policy, two-venv execution). This file is the map; read CLAUDE.md before non-trivial work.
+
+## STRUCTURE
+
+```
+clustrix/
+├── clustrix/            # the package — flat, 34 modules (see clustrix/AGENTS.md)
+├── tests/               # unit/ + real_world/ + integration/ + comprehensive/ (see tests/AGENTS.md)
+├── scripts/             # dev/ops tooling; aws/ is operator cleanup, NOT a backend
+├── docs/                # source/ (Sphinx) + evidence/ (committed proof) + build/ (generated)
+├── notes/               # session notes; per user policy, update as work proceeds
+├── .claude/             # pm command system (commands/pm, scripts/pm, rules, agents)
+├── .github/workflows/   # tests.yml, fast_ci.yml, real-world-tests.yml
+├── build/               # STALE setuptools output — see NOTES
+├── htmlcov/, performance_test_results/, docs/build/   # generated; ignore
+└── pyproject.toml       # the ONLY pytest/coverage config; black/mypy/flake8 too
+```
+
+## WHERE TO LOOK
+
+| Task | Location | Notes |
+|-|-|-|
+| Add a backend | `config.SUPPORTED_CLUSTER_TYPES` + `executor_core.py` dispatch + `executor_schedulers.py` | Gate: real job on real hardware, evidence committed |
+| Change execution flow | `executor_core.py` (ClusterExecutor), split across `executor_connections/_schedulers/_scheduler_status` | `executor.py` is a 39-line shim |
+| Touch serialization | `utils.py` `serialize_function`/`deserialize_function`, `generate_two_venv_execution_commands` | Keep each serialize/deserialize pair symmetric; never stdlib `pickle` |
+| Config change | `config.py` (ClusterConfig, configure, load_config) | No env-var overlay exists; only `CLUSTRIX_CONFIG_DIR` + `password_env_var` |
+| SSH/auth | `ssh_security.py` (host keys), `ssh_utils.py`, `auth_manager.py`, `credential_manager.py` | Never `AutoAddPolicy` directly |
+| Notebook UI | `notebook_magic_core.py` (%%remote, %clustrix), `modern_notebook_widget.py` | Widget never auto-displays on import unless `CLUSTRIX_AUTO_WIDGET=1` |
+| Data staging | `staging.py` (`data_package`, `materialize_packages`) | Declaration only; nothing inferred; nothing auto-deleted |
+| Quality gates | `scripts/pre_push_check.py` (retries 5x), `scripts/check_quality.py` | Run repeatedly until ALL pass before commit |
+| Regenerate backend evidence | `scripts/verify_cluster_usecases.py`, `scripts/collect_execution_evidence.py` | Output committed under `docs/evidence/` |
+| CI changes | `.github/workflows/` | `real-world-tests.yml` has NO push/PR trigger deliberately (credentialed jobs); secrets gate via `check-secrets` job outputs — `secrets` context is illegal in `if:` |
+
+## CODE MAP
+
+Centrality from codegraph (Python LSP not installed; ruff is lint-only).
+
+| Symbol | Type | Location | Refs | Role |
+|-|-|-|-|-|
+| `configure` | function | `clustrix/config.py:559` | 288 | Singleton config entry point; validates all keys before applying any |
+| `ClusterExecutor` | class | `clustrix/executor_core.py:27` | 81 | Dispatch, submission, HMAC-verified result retrieval |
+| `cluster` | decorator | `clustrix/decorator.py:58` | public API | `@cluster`; extras limited to `hf_*` + `key_file` — `cluster_type=` is NOT accepted |
+| `ClusterConfig` | dataclass | `clustrix/config.py` | high | Plain-`str` `cluster_type`; no ClusterType enum |
+| `ClusterfyMagics` | class | `clustrix/notebook_magic_core.py:69` | 6 | `%%remote`, `%clustrix`, deprecated `%%clusterfy` alias |
+| `LocalExecutor` | class | `clustrix/local_executor.py` | — | Real local parallelism; `choose_executor_type` at :339 |
+| `HFJobsManager` | class | `clustrix/hf_jobs.py` | — | HuggingFace Jobs backend; 256 KB payload cap |
+| `data_package` | function | `clustrix/staging.py` | — | Declared data staging; large packages go to a private HF dataset repo |
+
+## CONVENTIONS
+
+- black line-length 88, target py310, **pinned `black==26.3.1`** (unbounded `>=` broke CI, #110); flake8 max-line 88, extend-ignore E203/W503; mypy python_version 3.10, `files=["clustrix/"]`, tests ignored, `follow_imports="skip"`.
+- **pyproject.toml is the only pytest config.** No pytest.ini/tox.ini/setup.cfg — the first found shadows this block (#130); `tests/unit/test_pytest_config.py` enforces it.
+- pytest `--strict-markers`; all 6 markers registered in pyproject. `testpaths` must never list `tests/integration` (billable, #109).
+- Coverage `fail_under = 66` — a floor 2 points under measured 68%, not a target (#115).
+- Pre-commit runs on python3.12 explicitly (system python3 may be 3.9, which the project does not support).
+- Version string lives in 4 places and must stay identical: `pyproject.toml`, `setup.py`, `clustrix/__init__.py`, `docs/source/conf.py`.
+- Comments explain *why*, often with issue refs (#109–#159). Match that style; do not strip them.
+
+## ANTI-PATTERNS (THIS PROJECT)
+
+- Never `set_missing_host_key_policy(paramiko.AutoAddPolicy())` — go through `ssh_security.configure_host_key_policy`.
+- Never stdlib `pickle` in the two-venv handoffs — dill/cloudpickle only, symmetric pairs.
+- Never a mock as a fallback when the real thing is unavailable — fail instead. Production code must never know it is being tested (no `isinstance(x, Mock)`; `grep -rn "unittest.mock\|MagicMock" clustrix/` stays empty).
+- Never weaken a failing test; fix the code or explicitly rewrite a wrong assertion.
+- Never document unsupported backends (pbs/sge/k8s/cloud VMs), cost monitoring, or HF Spaces as working.
+- Never add `@cluster(cluster_type=...)` to examples — it is ignored with a warning; backend is set via `configure()`.
+- `auto_gpu_parallel` does nothing (deleted; it fabricated results). Don't document it as a feature.
+- No `cluster_put`/`cluster_get` — the `cluster_*` fs helpers are read-only by design.
+
+## UNIQUE STYLES
+
+- Honesty-first docs: verified vs unsupported backends stated up front; evidence transcripts committed under `docs/evidence/`.
+- Defensive validation with explanatory errors (removed settings raise `ValueError` naming the replacement; unknown config keys get did-you-mean).
+- One shared async executor per process (`decorator._shared_async_executor`), context-manager support on `SimpleAsyncClusterExecutor`.
+
+## COMMANDS
+
+```bash
+pip install -e ".[dev]"                 # dev env (widget extra needed for widget tests)
+python scripts/pre_push_check.py        # black+flake8+mypy+pytest, retries until clean — run before EVERY commit
+pytest tests/ -m "not real_world" --ignore=tests/real_world --ignore=tests/integration   # what CI runs
+pytest tests/unit/ -q                   # fast loop
+python scripts/collect_execution_evidence.py   # real job per reachable backend; needs credentials
+cd docs && make html                    # docs build
+```
+
+## NOTES
+
+- **`build/lib/clustrix/` is stale**: it still contains `kubernetes/`, `cloud_providers/`, `pricing_clients/`, `cost_providers/` — modules deleted from the source tree. Grep results there are ghosts; exclude `build/` from searches.
+- `htmlcov/`, `performance_test_results/`, `docs/build/` are generated output, not source.
+- `tests/integration/` provisions real billable AWS resources; refuses without `CLUSTRIX_ALLOW_BILLABLE=1` (guard reads `config.args`, deliberately).
+- `remote_work_dir` must be on a filesystem compute nodes see — `/tmp` dies with exit 127 on SLURM.
+- Fresh count: 21 of 166 test modules use `unittest.mock` (issue #117 migrates them; new tests must not add to it).