Merge pull request #36 from NatLabRockies/optimize

Measure authoring reliability + OS App compatibility + E2E test

Author: brianlball

.claude/skills/measure-authoring/SKILL.md

@@ -34,6 +34,9 @@ create_measure(
 ```
 test_measure(measure_dir="/runs/custom_measures/set_lights_8w")
 ```
+Tests run against the currently loaded model (or SystemD_baseline.osm fallback),
+so measures that depend on HVAC, plant loops, or zones will work correctly.
+Use `model_path` to test against a specific model.
 
 ### 3. Apply to Model
 ```

.github/workflows/ci.yml

@@ -72,7 +72,7 @@ jobs:
               ;;
             3)
               # controls, object mgmt, loads, building, doas, hvac, measures, measure_authoring, skill_qaqc, hvac_supply_wiring
-              FILES="tests/test_component_controls.py tests/test_object_management.py tests/test_generic_access.py tests/test_create_loads.py tests/test_building.py tests/test_doas_system.py tests/test_hvac.py tests/test_measures.py tests/test_measure_authoring.py tests/test_skill_qaqc.py tests/test_hvac_supply_wiring.py"
+              FILES="tests/test_component_controls.py tests/test_object_management.py tests/test_generic_access.py tests/test_create_loads.py tests/test_building.py tests/test_doas_system.py tests/test_hvac.py tests/test_measures.py tests/test_measure_authoring.py tests/test_skill_qaqc.py tests/test_hvac_supply_wiring.py tests/test_validate_model.py"
               EXTRA_ENV=""
               ;;
             4)

CLAUDE.md

@@ -3,7 +3,7 @@
 ## Project: openstudio-mcp
 MCP server that gives AI agents full control of building energy modeling —
 create buildings, configure HVAC, run EnergyPlus simulations, and extract
-results — all through 134 MCP tools backed by the OpenStudio SDK.
+results — all through 138 MCP tools backed by the OpenStudio SDK.
 
 **Who it's for:** Building energy modelers who want to use AI assistants
 (Claude, GPT, etc.) to do real modeling work — not just chat about it.
@@ -49,14 +49,15 @@ Always use openstudio-mcp tools for BEM tasks:
 | Phase 7 | 📋 FUTURE | Advanced creation (geometry, space type wizard) |
 | Phase 8 | ✅ COMPLETE | Bundle common-measures-gem (20 measures, 11 tools: reporting, thermostat, envelope, PV, visualization) |
 | Phase 9 | ✅ COMPLETE | AI-assisted measure authoring (3 tools: create, test, edit custom measures) |
+| Phase 10 | ✅ COMPLETE | Results & error management (4 tools: errors, variables, compare, validate) |
 
 ## Current Skills
 | Skill | Tools | Phase |
 |-------|-------|-------|
 | `server_info` | `get_server_status`, `get_versions` | Phase 1 |
 | `model_management` | `create_example_osm`, `create_baseline_osm`, `inspect_osm_summary`, `load_osm_model`, `save_osm_model`, `list_files` | Phase 1 + 2 |
-| `simulation` | `validate_osw`, `run_osw`, `run_simulation`, `get_run_status`, `get_run_logs`, `get_run_artifacts`, `cancel_run` | Phase 1 |
-| `results` | `extract_summary_metrics`, `read_file`, `copy_file`, `extract_end_use_breakdown`, `extract_envelope_summary`, `extract_hvac_sizing`, `extract_zone_summary`, `extract_component_sizing`, `query_timeseries` | Phase 1 + 9 |
+| `simulation` | `validate_osw`, `run_osw`, `run_simulation`, `get_run_status`, `get_run_logs`, `get_run_artifacts`, `cancel_run`, `validate_model` | Phase 1 + 10 |
+| `results` | `extract_summary_metrics`, `read_file`, `copy_file`, `extract_end_use_breakdown`, `extract_envelope_summary`, `extract_hvac_sizing`, `extract_zone_summary`, `extract_component_sizing`, `query_timeseries`, `extract_simulation_errors`, `list_output_variables`, `compare_runs` | Phase 1 + 9 + 10 |
 | `building` | `get_building_info`, `get_model_summary` | Phase 2 |
 | `spaces` | `list_spaces`, `get_space_details`, `list_thermal_zones`, `get_thermal_zone_details`, `create_space`, `create_thermal_zone` | Phase 2 + 3 |
 | `geometry` | `list_surfaces`, `get_surface_details`, `list_subsurfaces`, `create_surface`, `create_subsurface`, `create_space_from_floor_print`, `match_surfaces`, `set_window_to_wall_ratio`, `import_floorspacejs` | Phase 2 + 7 |
@@ -77,7 +78,7 @@ Always use openstudio-mcp tools for BEM tasks:
 | `measure_authoring` | `list_custom_measures`, `create_measure`, `test_measure`, `edit_measure` | Phase 9 |
 | `skill_discovery` | `list_skills`, `get_skill` | — |
 
-**Total: 23 skills, 134 MCP tools, ~390 integration tests**
+**Total: 23 skills, 138 MCP tools, ~400 integration tests**
 
 ## Model Query Pattern
 ```python
@@ -197,7 +198,7 @@ To add a new SPM type to `set_setpoint_manager_properties`:
 
 ## Rules
 1. Keep files small where practical — aim for under 250 lines, but don't split artificially just to hit a number
-2. Every MCP tool must have a test in `tests/skills/` (Phase 2+) or `tests/` (existing)
+2. Every MCP tool must have a test in `tests/skills/` (Phase 2+) or `tests/` (existing). New behavior, bug fixes, and security hardening must also have integration tests — not just the happy path.
 3. **Integration tests must be added to `.github/workflows/ci.yml`** — add a new step following the existing pattern
 4. Operations return dicts with `{"ok": True/False, ...}` — never raise through MCP
 5. Use `openstudio` Python bindings directly.

README.md

@@ -342,7 +342,7 @@ List HVAC components via `list_model_objects("BoilerHotWater")`, loop detail too
 
 ### Measure Authoring (4 tools)
 
-Create custom OpenStudio measures with AI-generated code, test them, and apply to models. See [Example 1](docs/examples/01_custom_measure_lighting.md) and [Example 2](docs/examples/02_custom_measure_hvac.md).
+Create custom OpenStudio measures with AI-generated code, test them, and apply to models. See [Example 1](docs/examples/01_custom_measure_lighting.md), [Example 2](docs/examples/02_custom_measure_hvac.md), and [Example 19](docs/examples/19_systemd_fourpipebeam_retrofit.md) (full E2E retrofit).
 
 | Tool | Description |
 |------|-------------|
@@ -424,7 +424,7 @@ The component properties tools can query and modify these 15 HVAC component type
 
 ## Examples
 
-18 worked examples with full tool-call sequences — click to expand:
+19 worked examples with full tool-call sequences — click to expand:
 
 | # | Example | Description |
 |---|---------|-------------|
@@ -446,6 +446,7 @@ The component properties tools can query and modify these 15 HVAC component type
 | 16 | [`/new-building`](docs/examples/16_new_building.md) | Full model creation from scratch |
 | 17 | [`/retrofit`](docs/examples/17_retrofit.md) | Before/after ECM analysis |
 | 18 | [`/view`](docs/examples/18_view.md) | Interactive 3D model visualization |
+| 19 | [SystemD Four-Pipe Beam Retrofit](docs/examples/19_systemd_fourpipebeam_retrofit.md) | End-to-end: load 44-zone model, baseline sim, author measure, retrofit sim, compare |
 
 ---
 

docs/examples/02_custom_measure_hvac.md

@@ -84,6 +84,7 @@ The AI compares:
 - `edit_measure` can fix issues without recreating from scratch
 - Chilled beams are AIR TERMINALS (connected via `addBranchForZone`), NOT zone equipment
 
-## Integration Test
+## Integration Tests
 
-See `tests/llm/test_04_workflows.py::test_workflow[measure_replace_terminals_full_chain]`
+- `tests/llm/test_04_workflows.py::test_workflow[measure_replace_terminals_full_chain]` — explicit prompt with API hints
+- `tests/llm/test_04_workflows.py::test_workflow[systemd_fourpipebeam_e2e]` — natural-language prompt on 44-zone model (see [Example 19](19_systemd_fourpipebeam_retrofit.md))

docs/examples/19_systemd_fourpipebeam_retrofit.md

@@ -0,0 +1,95 @@
+# Example 19: End-to-End Retrofit — Four-Pipe Chilled Beams on Complex Model
+
+Real-world workflow: load an existing 44-zone model, set weather, run baseline, author a custom measure, apply it, re-simulate, and compare results.
+
+## Scenario
+
+A modeler has an existing SystemD building (44 zones, 2 DOAS air loops, CHW/HW/SWH plant loops) and wants to evaluate replacing the constant-volume no-reheat terminals with 4-pipe active chilled beams. They provide a natural-language request with no tool names or API hints.
+
+## Prompt
+
+> I have a model at /inputs/SystemD_baseline.osm. I want you to run the model with Boston-Logan weather file — files are in the same directory. After that, create a measure for me that changes the air terminals to 4-pipe chilled beams, apply that measure to the model, run the model, and compare the results for me. Save the measure in the same location as the model so I have a copy.
+
+## Tool Call Sequence
+
+### Setup and Baseline
+
+```
+1. list_skills()                            # discover available workflows
+2. get_skill("measure-authoring")           # load measure authoring guide
+3. load_osm_model(osm_path="/inputs/SystemD_baseline.osm")
+4. get_building_info()                      # 44 zones, 54K ft2
+5. list_air_loops()                         # 2 DOAS loops (21-zone + 23-zone)
+6. change_building_location(weather_file="/inputs/USA_MA_Boston-Logan.Intl.AP.725090_TMY3.epw")
+7. save_osm_model(save_path="/runs/SystemD_baseline.osm")
+8. run_simulation(osm_path="/runs/SystemD_baseline.osm")
+9. get_run_status(run_id=<id>)              # poll until complete
+10. extract_summary_metrics(run_id=<id>)    # baseline EUI: 28.21 kBtu/ft2
+```
+
+### Author Measure
+
+```
+11. list_plant_loops()                      # identify CHW + HW loop names
+12. create_measure(
+      name="replace_terminals_with_four_pipe_beams",
+      description="Replace all air terminals with 4-pipe chilled beams",
+      language="Ruby",
+      run_body=<generated by LLM>)
+13. test_measure(measure_dir="/runs/custom_measures/replace_terminals_with_four_pipe_beams")
+    # 2 runs, 2 assertions, 0 failures — tested against loaded model
+```
+
+### Apply and Re-simulate
+
+```
+14. load_osm_model(osm_path="/inputs/SystemD_baseline.osm")
+15. change_building_location(weather_file=...)
+16. apply_measure(measure_dir="/runs/custom_measures/replace_terminals_with_four_pipe_beams")
+17. list_air_loops()                        # verify FourPipeBeam terminals
+18. save_osm_model(save_path="/runs/SystemD_fourpipebeam.osm")
+19. run_simulation(osm_path="/runs/SystemD_fourpipebeam.osm")
+20. get_run_status(run_id=<id>)             # poll until complete
+```
+
+### Compare
+
+```
+21. compare_runs(baseline_run_id=<id1>, retrofit_run_id=<id2>)
+22. copy_file(file_path="/runs/custom_measures/replace_terminals_with_four_pipe_beams",
+             destination="/inputs")
+```
+
+## Expected Results
+
+| Metric | Baseline | Retrofit | Delta |
+|--------|----------|----------|-------|
+| EUI (kBtu/ft2) | 28.21 | 28.44 | +0.8% |
+| Unmet Hours | 58.5 | 34.5 | -41% |
+| Cooling | — | — | +5.6% |
+| Heating | — | — | +0.9% |
+| Pumps | — | — | -6.1% |
+
+34 zones had CV NoReheat terminals replaced with FourPipeBeam. 0 fatal/severe errors in both simulations. Results are deterministic across reruns.
+
+## Key Differences from Example 2
+
+| Aspect | Example 2 | This Example |
+|--------|-----------|--------------|
+| Model | 5-zone baseline | 44-zone SystemD (real complexity) |
+| Prompt style | Semi-explicit (mentions tool patterns) | Natural language (no tool names) |
+| Weather | Pre-configured | Must be set (Boston-Logan EPW) |
+| Test model | Generic | Tested against loaded model with HVAC |
+| Comparison | Manual extraction | `compare_runs` tool |
+| Measure export | Not saved | Copied to /inputs for reuse |
+| Tool calls | ~16 | ~35 |
+
+## Common Pitfalls
+
+- **Weather file not found**: The model may reference a different EPW (e.g. Baltimore). `change_building_location` must be called before `run_simulation` and again after `load_osm_model` reload.
+- **Plant loop names**: The measure must find loops by name substring ("Chilled Water", "Hot Water") — exact names vary by model.
+- **Polling frequency**: `get_run_status` should wait 1-2 minutes between calls. Complex models take 60-140s to simulate.
+
+## Integration Test
+
+See `tests/llm/test_04_workflows.py::test_workflow[systemd_fourpipebeam_e2e]`