add sans-ir

Author: joenorton

.cursor/plans/sprint_1_ir_mutation_surface_c527a0dd.plan.md

@@ -0,0 +1,279 @@
+---
+name: Sprint 1 IR Mutation Surface (Revised)
+overview: Establish sans.ir as the single canonical editable IR. Remove all legacy/obsolete IR schema references. Implement deterministic normalization from IRDoc → sans.ir and sans.ir → expanded.sans printing via existing printer. Add run-ir CLI that executes sans.ir through the existing runtime core. No alternate schemas. No save splitting. Stable step handles derived from semantic anchors (not positional indices).
+todos: []
+isProject: false
+---
+
+# Sprint 1 — IR as the Mutation Surface (Authoritative Version)
+
+## Critical Clarification
+
+There is **one canonical IR format going forward: `sans.ir`.**
+
+Any pre-existing IR schema documentation (e.g. [docs/sans-ir-v0.1-schema](docs/sans-ir-v0.1-schema)) is obsolete and must either:
+
+- be **removed**, or
+- be **renamed** explicitly to describe `plan.ir.json` (execution IR with loc + transform ids)
+
+There must be **no competing IR definitions** in the repo after this sprint.
+
+---
+
+## Core Principles
+
+1. `sans.ir` is the mutation surface.
+2. `sans.ir` contains *only semantic data*.
+3. Execution continues to use in-memory `IRDoc`.
+4. Both `run` (text) and `run-ir` (IR file) feed the same `execute_plan`.
+5. **No positional step IDs.** No s1/s2 numbering — use semantic anchors only.
+
+---
+
+## 1. Define Canonical `sans.ir` Schema
+
+**Location:** `sans/ir/` package.
+
+Convert existing [sans/sans/ir.py](sans/sans/ir.py) into package:
+
+```
+sans/ir/
+  __init__.py   # exports IRDoc + related types (move current ir.py content here or to doc.py)
+  schema.py     # sans.ir schema definitions
+  normalize.py  # IRDoc → sans.ir
+  adapter.py    # sans.ir → IRDoc
+```
+
+**Do not add** `printer.py` — the printer is the existing `irdoc_to_expanded_sans(IRDoc)`.
+
+---
+
+### sans.ir Structure (Authoritative)
+
+```json
+{
+  "version": "0.1",
+  "datasources": {
+    "lb": {
+      "kind": "csv",
+      "path": "lb.csv",
+      "columns": {
+        "USUBJID": "string",
+        "VISITNUM": "int"
+      }
+    }
+  },
+  "steps": [
+    {
+      "id": "out:__t2__",
+      "op": "identity",
+      "inputs": ["__datasource__lb"],
+      "outputs": ["__t2__"],
+      "params": {}
+    },
+    {
+      "id": "out:sorted_high",
+      "op": "save",
+      "inputs": ["sorted_high"],
+      "outputs": [],
+      "params": { "path": "sorted_high.csv" }
+    }
+  ]
+}
+```
+
+---
+
+### Hard Rules
+
+**Excluded fields (must NOT appear in sans.ir):**
+
+- transform_id
+- transform_class_id
+- step_id
+- loc
+- any runtime fingerprint
+- registry ids
+
+**Step identity rule (semantic anchors only):**
+
+- If step produces **one output table:** `id = "out:<output_table_name>"`
+- For **datasource steps:** `id = "ds:<datasource_name>"`
+
+This gives stable anchors and avoids cascade renumbering on insertion (mutation-friendly).
+
+**Save steps:** Do **not** split saves into a top-level `saves[]`. `save` remains a normal step in `steps[]`. One semantic representation only.
+
+**Canonical ordering:**
+
+- datasources: sorted by key
+- steps: topologically sorted
+- params: keys sorted (recursively for nested structures)
+- JSON serialization: deterministic; no UUIDs; no timestamps
+
+---
+
+## 2. Normalization: IRDoc → sans.ir
+
+**File:** [sans/ir/normalize.py](sans/ir/normalize.py)
+
+- **Input:** IRDoc  
+- **Output:** canonical sans.ir dict
+
+**Responsibilities:**
+
+- Strip all execution-only fields.
+- Generate **semantic** step ids per rule above (`out:<table>`, `ds:<name>`).
+- Preserve step structure exactly.
+- Do **not** collapse identity steps.
+- Do **not** rewrite logic or perform semantic optimizations.
+- Canonicalize param key ordering recursively (e.g. reuse [sans/sans_script/canon.py](sans/sans_script/canon.py) style).
+
+Normalization must be **deterministic**.
+
+---
+
+## 3. Adapter: sans.ir → IRDoc
+
+**File:** [sans/ir/adapter.py](sans/ir/adapter.py)
+
+- **Input:** sans.ir dict  
+- **Output:** IRDoc
+
+**Responsibilities:**
+
+- Rebuild `OpStep` objects (and datasources, table_facts as needed).
+- Provide **synthetic Loc** objects if runtime/validator require them.
+- Preserve step order exactly as in sans.ir.
+- No mutation or inference of logic.
+
+Target invariant:
+
+```
+expanded.sans → compile → IRDoc
+IRDoc → normalize → sans.ir
+sans.ir → adapter → IRDoc
+IRDoc → irdoc_to_expanded_sans → expanded.sans
+```
+
+must be lossless (byte-identical expanded.sans and semantically identical sans.ir).
+
+---
+
+## 4. Printer
+
+**Do not** implement a new printer.
+
+Use existing [sans/sans_script/expand_printer.py](sans/sans_script/expand_printer.py):
+
+```
+sans.ir → adapter → IRDoc → irdoc_to_expanded_sans(IRDoc)
+```
+
+No alternate formatting logic.
+
+---
+
+## 5. run-ir CLI
+
+**Subcommand:**
+
+```
+sans run-ir input.sans.ir --out outdir
+```
+
+**Implementation:**
+
+- Load sans.ir JSON from `input.sans.ir`.
+- Convert via adapter → IRDoc.
+- Pass IRDoc to existing `execute_plan` (same path as `sans run`).
+- Produce identical bundle shape as `sans run` (report.json, artifacts, outputs).
+
+Execution logic must **not** fork. Register in [sans/**main**.py](sans/sans/__main__.py).
+
+---
+
+## 6. Obsolete IR Schema Cleanup
+
+- **Remove or rename** [docs/sans-ir-v0.1-schema](docs/sans-ir-v0.1-schema):
+  - Either delete it, or
+  - Rename and repurpose to document **plan.ir.json** (execution IR: loc, transform_id, step_id, etc.) so it is explicit that it is *not* the canonical mutation IR.
+
+After this sprint there must be **exactly one** canonical IR definition (sans.ir); no competing docs.
+
+---
+
+## 7. Tests
+
+### A. IR round-trip
+
+**File:** [tests/test_ir_roundtrip.py](tests/test_ir_roundtrip.py)
+
+**Pipeline:**
+
+1. `expanded.sans` → compile → IRDoc → normalize → **sans.ir A**
+2. **sans.ir A** → adapter → IRDoc → printer → **expanded.sans'**
+3. **expanded.sans'** → compile → IRDoc → normalize → **sans.ir B**
+
+**Assertions:**
+
+- `expanded.sans == expanded.sans'` (byte-identical).
+- `A == B` (canonical IR equality).
+
+Both assertions are required.
+
+---
+
+### B. Execution equivalence
+
+**File:** [tests/test_ir_execution_equivalence.py](tests/test_ir_execution_equivalence.py)
+
+- Run `sans run script.sans` and `sans run-ir script.sans.ir` (script.sans.ir produced from script.sans via normalize).
+- Assert: report.json identical status; output artifact hashes identical.
+
+---
+
+### C. Canonicalization
+
+**File:** [tests/test_ir_canonicalization.py](tests/test_ir_canonicalization.py)
+
+- No `transform_id` in sans.ir.
+- No `loc` in sans.ir.
+- Deterministic JSON ordering.
+- Multiple normalizations of same IRDoc produce identical sans.ir JSON.
+
+---
+
+## 8. Exit Criteria
+
+Sprint complete when:
+
+- There is **exactly one** canonical mutation IR (`sans.ir`).
+- No obsolete IR schema remains (removed or renamed to plan.ir.json only).
+- Round-trip is **byte-stable** (expanded.sans) and **canonical IR equality** (sans.ir A == B).
+- run-ir produces **identical execution results** (status + output hashes) to run for same logic.
+- No execution-only artifacts (transform_id, loc, step_id, etc.) appear in sans.ir.
+
+---
+
+## 9. Non-Goals
+
+- No optimizer logic.
+- No move system.
+- No IR rewriting or mapping logic.
+
+This sprint establishes the foundation only. All mutation work depends on this being correct.
+
+---
+
+## 10. File-Level Summary
+
+
+| Action        | Path                                                                                                                                                                                                               |
+| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| Refactor      | `sans/ir.py` → `sans/ir/` package with `__init__.py`, `schema.py`, `normalize.py`, `adapter.py`                                                                                                                    |
+| Modify        | [sans/**main**.py](sans/sans/__main__.py): register `run-ir` subcommand                                                                                                                                            |
+| Remove/rename | [docs/sans-ir-v0.1-schema](docs/sans-ir-v0.1-schema) — remove or rename to plan.ir.json documentation                                                                                                              |
+| Add tests     | [tests/test_ir_roundtrip.py](tests/test_ir_roundtrip.py), [tests/test_ir_execution_equivalence.py](tests/test_ir_execution_equivalence.py), [tests/test_ir_canonicalization.py](tests/test_ir_canonicalization.py) |
+
+

docs/sans-ir-v0.1-schema docs/plan-ir-v0.1-schemadocs/sans-ir-v0.1-schema renamed to docs/plan-ir-v0.1-schema

@@ -10,6 +10,9 @@
 from .validator_sdtm import validate_sdtm
 from .fmt import FMT_STYLE_ID, format_text, normalize_newlines
 from . import __version__ as _engine_version
+from .ir.adapter import sans_ir_to_irdoc
+from .ir.schema import validate_sans_ir
+from .sans_script import irdoc_to_expanded_sans
 
 
 def _parse_tables(tables_arg: str | None) -> set[str] | None:
@@ -119,6 +122,22 @@ def main(argv: list[str] | None = None) -> int:
     run_parser.add_argument("--lock-only", action="store_true", help="With --emit-schema-lock: generate lock only, do not execute (even if all datasources are typed)")
     run_parser.add_argument("--bundle-mode", choices=["full", "thin"], default="full", help="Bundle mode: full (embed datasource bytes) or thin (fingerprints only)")
 
+    run_ir_parser = subparsers.add_parser("run-ir", help="Execute a canonical sans.ir file")
+    run_ir_parser.add_argument("script", help="Path to the sans.ir file")
+    run_ir_parser.add_argument("--out", required=True, help="Output directory for plan/report and outputs")
+    run_ir_parser.add_argument("--tables", default="", help="Comma-separated table bindings name=path.csv")
+    run_ir_parser.add_argument("--format", default="csv", choices=["csv", "xpt"], help="Output format (csv, xpt)")
+    run_ir_parser.add_argument("--strict", dest="strict", action="store_true", default=True)
+    run_ir_parser.add_argument("--no-strict", dest="strict", action="store_false")
+    run_ir_parser.add_argument("--schema-lock", metavar="path", default=None, help="Path to schema.lock.json to enforce when ingesting datasources")
+    run_ir_parser.add_argument("--emit-schema-lock", metavar="path", default=None, help="After successful run, write schema.lock.json to this path")
+    run_ir_parser.add_argument("--lock-only", action="store_true", help="With --emit-schema-lock: generate lock only, do not execute (even if all datasources are typed)")
+    run_ir_parser.add_argument("--bundle-mode", choices=["full", "thin"], default="full", help="Bundle mode: full (embed datasource bytes) or thin (fingerprints only)")
+
+    ir_validate_parser = subparsers.add_parser("ir-validate", help="Validate sans.ir structure only")
+    ir_validate_parser.add_argument("script", help="Path to the sans.ir file")
+    ir_validate_parser.add_argument("--strict", action="store_true", default=False, help="Treat structural warnings as errors")
+
     schema_lock_parser = subparsers.add_parser("schema-lock", help="Generate schema.lock.json without execution (no --out required)")
     schema_lock_parser.add_argument("script", help="Path to the script file")
     schema_lock_parser.add_argument("--write", "-o", dest="write", default=None, metavar="path", help="Lock output path (default: <script_dir>/<script_stem>.schema.lock.json); relative paths resolved against script dir")
@@ -420,6 +439,84 @@ def main(argv: list[str] | None = None) -> int:
                 print(f"ok: wrote schema lock to {emit_path}{suffix}")
         return int(report.get("exit_code_bucket", 50))
 
+    if args.command == "run-ir":
+        script_path = Path(args.script)
+        out_dir = Path(args.out)
+        bindings = {}
+        if args.tables:
+            for item in args.tables.split(","):
+                if not item.strip():
+                    continue
+                if "=" not in item:
+                    return _write_failed_report(out_dir, f"Invalid table binding '{item}'")
+                name, path = item.split("=", 1)
+                name = name.strip()
+                if name in bindings:
+                    return _write_failed_report(out_dir, f"Duplicate table binding for '{name}'")
+                bindings[name] = path.strip()
+        try:
+            sans_ir = json.loads(script_path.read_text(encoding="utf-8"))
+            validate_sans_ir(sans_ir)
+            ir_doc = sans_ir_to_irdoc(sans_ir, file_name=str(script_path))
+        except OSError as exc:
+            return _write_failed_report(out_dir, str(exc))
+        except Exception as exc:
+            return _write_failed_report(out_dir, f"Invalid sans.ir: {exc}")
+
+        # Reuse existing run path and runtime core by rendering canonical expanded.sans
+        # from IR, then executing as a normal .sans script.
+        expanded = irdoc_to_expanded_sans(ir_doc)
+        virtual_script_path = script_path.with_suffix(".expanded.sans")
+        schema_lock_path = Path(args.schema_lock) if args.schema_lock else None
+        emit_schema_lock_path = Path(args.emit_schema_lock) if args.emit_schema_lock else None
+        report = run_script(
+            text=expanded,
+            file_name=str(virtual_script_path),
+            bindings=bindings,
+            out_dir=out_dir,
+            strict=args.strict,
+            output_format=args.format,
+            include_roots=None,
+            allow_absolute_includes=False,
+            allow_include_escape=False,
+            legacy_sas=False,
+            schema_lock_path=schema_lock_path,
+            emit_schema_lock_path=emit_schema_lock_path,
+            lock_only=getattr(args, "lock_only", False),
+            bundle_mode=getattr(args, "bundle_mode", "full"),
+        )
+        status = report.get("status")
+        if status == "refused":
+            primary = report.get("primary_error") or {}
+            loc = primary.get("loc") or {}
+            loc_str = f"{loc.get('file')}:{loc.get('line_start')}" if loc else ""
+            print(f"refused: {primary.get('code')} at {loc_str}".rstrip())
+        elif status == "failed":
+            primary = report.get("primary_error") or {}
+            loc = primary.get("loc") or {}
+            loc_str = f"{loc.get('file')}:{loc.get('line_start')}" if loc else ""
+            print(f"failed: {primary.get('code')} at {loc_str}".rstrip())
+        else:
+            print("ok: wrote plan.ir.json report.json registry.candidate.json runtime.evidence.json")
+        return int(report.get("exit_code_bucket", 50))
+
+    if args.command == "ir-validate":
+        script_path = Path(args.script)
+        try:
+            sans_ir = json.loads(script_path.read_text(encoding="utf-8"))
+            warnings = validate_sans_ir(sans_ir, strict=bool(getattr(args, "strict", False)))
+        except OSError as exc:
+            print(f"invalid: {exc}", file=sys.stderr)
+            return 2
+        except Exception as exc:
+            print(f"invalid: {exc}", file=sys.stderr)
+            return 2
+        if warnings:
+            print(f"ok: valid sans.ir ({len(warnings)} warning(s))", file=sys.stderr)
+        else:
+            print("ok: valid sans.ir", file=sys.stderr)
+        return 0
+
     if args.command == "schema-lock":
         from .runtime import generate_schema_lock_standalone
         script_path = Path(args.script)