Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,9 @@ uv run litereality run scans/<scan> --through seed
uv run litereality stage author run/<scan> --force --polish --live
```

`--polish` adds object refinement, materials, and a model-driven quality pass on top of authoring.
`--polish` adds object refinement and materials on top of authoring. The model-driven quality
pass is separate — add `--quality-pass` for it — because it is the longest agent pass on a run and
nothing downstream reads its output.
`--live` shows how everything is built in real time, alongside the agent's trace. With `--live` the
viewer starts before the room exists and waits for it, so it works on a scene's first authoring
run; it prints its url again once the first build lands.
Expand Down
14 changes: 14 additions & 0 deletions doc/Sim-Ready-intergration/Mujoco.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,20 @@ a table is a solid block — a chair tucked under it would be launched on the fi
would be a filled cupboard. Every concave body is therefore several convex pieces, and the original
mesh is kept as a visual-only geom.

### Driving it

Every joint that came from a sidecar also gets an actuator, at the effort the asset itself stated —
45 N·m for that door leaf, 800 N for that desk lift. Without them the articulation is scenery: the
only way to open a door is to push it with another body, and a limit the recipe compiled has no
effect on anything. They are `motor` elements rather than position servos on purpose. A servo holds
a setpoint, so it would clamp every door shut at `ctrl=0` and a hung door would stop swinging when
the room was shaken; a motor applies exactly `ctrl` and nothing at rest, so the passive scene is
unchanged and the joint is merely now drivable. The velocity limit the sidecar carries has nowhere
to live in MJCF, so it is kept on the actuator's `user` field for a controller to read back.

A joint recovered from the raw articulation extras gets no actuator, because nothing said what its
effort should be and inventing one is a guess. `export_report.json` lists what was actuated.

The room itself is a body on three slide joints and a yaw hinge, driven by stiff position servos.
That is what makes `--shake` an earthquake rather than a change in the direction of gravity: the
floor really accelerates under the furniture, and a hung door really swings because its frame is
Expand Down
27 changes: 23 additions & 4 deletions sanity.py
Original file line number Diff line number Diff line change
Expand Up @@ -416,11 +416,30 @@ def harness_for(role: str) -> str:
env=("LR_REFINE_PROVIDER", "claude", "other roles can stay on codex"))

if "providers" in wanted:
print("── hosted models (OpenAI images · Claude reasoning) ──")
# CHECK THE PROVIDER THAT IS ACTUALLY SELECTED. This asked for an OpenAI key unconditionally,
# so a correctly configured Gemini setup failed sanity while the run itself would have been
# fine — and, worse, an OpenAI key present alongside `LR_IMAGE_PROVIDER=gemini` passed here
# and then failed on a missing Gemini key at the first object.
provider = (os.environ.get("LR_IMAGE_PROVIDER") or "openai").strip().lower()
model = os.environ.get("LR_OPENAI_IMAGE_MODEL") or "gpt-image-2"
if provider not in ("openai", "gemini"):
provider = "gemini" if model.lower().startswith("gemini") else "openai"
print(f"── hosted models ({provider} images · Claude reasoning) ──")

if provider == "gemini":
if os.environ.get("GEMINI_API_KEY", ""):
ok(f"GEMINI_API_KEY set (image provider is gemini, "
f"{os.environ.get('LR_GEMINI_IMAGE_MODEL') or 'gemini-2.5-flash-image'})")
else:
fail("LR_IMAGE_PROVIDER=gemini but GEMINI_API_KEY is unset — every reference "
"image will fall back to the raw evidence sheet.",
env=("GEMINI_API_KEY", "<your-gemini-key>",
"create at https://aistudio.google.com/apikey"))
key = os.environ.get("OPENAI_API_KEY", "")
if not key:
fail("OPENAI_API_KEY unset — reference image-gen "
f"({os.environ.get('LR_OPENAI_IMAGE_MODEL') or 'gpt-image-2'}) will fail.",
if provider != "openai":
ok("OPENAI_API_KEY not required — images come from gemini")
elif not key:
fail(f"OPENAI_API_KEY unset — reference image-gen ({model}) will fail.",
env=("OPENAI_API_KEY", "sk-<your-openai-key>", "create at https://platform.openai.com/api-keys"))
else:
try:
Expand Down
54 changes: 54 additions & 0 deletions scripts/simready/view_scene.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
"""Open a LiteReality MuJoCo scene interactively.

MuJoCo's managed viewer (`python -m mujoco.viewer`) crashes constructing its window on this macOS,
but the PASSIVE viewer works — so this owns the loop instead of handing it to MuJoCo.

./.venv/bin/mjpython scripts/simready/view_scene.py [scene.xml] # frozen: look, don't touch
./.venv/bin/mjpython scripts/simready/view_scene.py [scene.xml] --physics # gravity on, things can be pushed

Frozen is the default because it answers a different question: it shows the room exactly as the
exporter wrote it, with nothing settled, nudged or knocked over. Nothing is stepped, so a drag
cannot move anything and no contact force is ever resolved.
"""
import sys
import time
from pathlib import Path

import mujoco
import mujoco.viewer

DEFAULT = "run/Office-Elliott/realism_authoring/mujoco/scene.xml"
args = [a for a in sys.argv[1:] if not a.startswith("-")]
physics = "--physics" in sys.argv
scene = Path(args[0]) if args else Path(DEFAULT)

model = mujoco.MjModel.from_xml_path(str(scene))
data = mujoco.MjData(model)
drives = [j for j in (mujoco.mj_name2id(model, mujoco.mjtObj.mjOBJ_JOINT, n)
for n in ("room_x", "room_y", "room_z", "room_yaw")) if j >= 0]

mujoco.mj_forward(model, data)
print(f"{scene}\n {model.nbody} bodies · {model.ngeom} geoms · {model.njnt} joints")
print(f" mode: {'PHYSICS — gravity on, bodies can be pushed' if physics else 'FROZEN — nothing is stepped, nothing can move'}")
print(" drag to orbit · scroll to zoom · double-click a body to select and name it")
print(" in the window: press F1 for help, Tab for the panel, ctrl-A to see contact forces")

with mujoco.viewer.launch_passive(model, data) as viewer:
while viewer.is_running():
t0 = time.time()
if physics:
for j in drives:
data.qpos[model.jnt_qposadr[j]] = 0.0
data.qvel[model.jnt_dofadr[j]] = 0.0
mujoco.mj_step(model, data)
else:
# Kinematics only. Poses stay exactly as exported; no force is ever integrated, so a
# mouse drag cannot displace anything and the scene cannot settle out from under you.
data.xfrc_applied[:] = 0.0
data.qacc[:] = 0.0
data.qvel[:] = 0.0
mujoco.mj_forward(model, data)
viewer.sync()
wait = model.opt.timestep - (time.time() - t0)
if wait > 0:
time.sleep(wait)
11 changes: 11 additions & 0 deletions src/litereality_agent/agent/author.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@

from __future__ import annotations

import json
import os
import time
from pathlib import Path
Expand Down Expand Up @@ -414,6 +415,16 @@ async def run(room: Path, surface_ref: Path, scan: Path, model: str, max_turns:
flush=True)
tr.think(f"[restored checkpoint] {broken}")
broken = room_compiles(room)
# WHETHER THIS ROOM WAS FINISHED OR MERELY STOPPED. A budget landing is graceful and exits 0,
# so the stage above could not tell it apart from a room the model considered done — a
# half-authored room reported as a completed stage. The distinction is only knowable here, so
# it is written down rather than left in the log for someone to notice.
try:
(room.parent / ".author_result.json").write_text(json.dumps({
"ended_early": ended_early, "calls": calls, "step_budget": step_budget,
}), encoding="utf-8")
except OSError:
pass # a summary that cannot be written is not fatal
tr.end(calls=calls, cost_usd=cost, summary=result_text)
print(f"\n== done {dt}s | calls={calls} {counts} | cost=${cost} ==\n", flush=True)
if tr.ok:
Expand Down
32 changes: 28 additions & 4 deletions src/litereality_agent/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -105,23 +105,47 @@ def _require_scene_package(target: str, context: RunContext, stage: str) -> None


def _author_options(args) -> dict:
# `--polish` deliberately does NOT include the model-driven quality pass. Refinement and
# materials each produce something the room did not have before; QC re-examines what the
# authoring pass already rendered and compared, so it is the longest agent pass on the run for
# the smallest marginal change — and nothing downstream reads its output. It stays available
# as `--quality-pass` for a room being prepared to be looked at rather than simulated.
polish = getattr(args, "polish", False)
return {
opts = {
"refine_objects": polish or getattr(args, "refine_objects", False),
"materials": polish or getattr(args, "materials", False),
"quality_pass": polish or getattr(args, "quality_pass", False),
"quality_pass": getattr(args, "quality_pass", False),
}
steps = getattr(args, "author_steps", None)
if steps:
opts["step_budget"] = steps
return opts


def _add_author_options(parser: argparse.ArgumentParser) -> None:
parser.add_argument(
"--polish",
action="store_true",
help="run object refinement, materials, and model-driven QC after authoring",
help="run object refinement and materials after authoring "
"(add --quality-pass for the model-driven QC pass as well)",
)
parser.add_argument("--refine-objects", action="store_true")
parser.add_argument("--materials", action="store_true")
parser.add_argument("--quality-pass", action="store_true")
parser.add_argument(
"--quality-pass",
action="store_true",
help="model-driven QC pass — not included in --polish",
)
# Exposed for SHORT RUNS, not lowered as a default. Authoring spends its opening steps reading
# the capture and measuring surfaces before it edits anything, so a small budget does not give
# a rougher room, it gives an unfinished one — on Office-Elliott the first edit landed at step
# 19 and the walls were not squared up until the sixties. A truncated run now says so in the
# stage summary, which is what makes a low number safe to ask for.
parser.add_argument(
"--author-steps", type=int, default=None, metavar="N",
help="tool-call budget for the authoring session (default 100); "
"a low value returns an unfinished room and says so",
)


def _simulate_options(args) -> dict:
Expand Down
14 changes: 13 additions & 1 deletion src/litereality_agent/models/object_generation/sim/properties.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,19 @@
# what gives a drawer front a believable mass of its own.
OCCUPANCY_DENSITY = {
"default": 45.0,
"chair": 14.0, "stool": 14.0, "table": 20.0, "desk": 22.0,
# 14.0 put a task chair at 7.2 kg and, after the room's fit rescaled it, 4.9 kg on the floor —
# against 12-15 kg for the real thing. Everything downstream inherits that: a body at a third
# of its weight takes a third of the push to send it across the room, which reads as the
# simulation being unstable when it is the mass that is wrong. Measured against real furniture
# over its own bounding box, a chair is 23-27 kg/m^3 whatever kind it is —
# task chair, castors + gas strut 0.49 m^3 13 kg -> 26.8
# cantilever meeting chair 0.23 m^3 5.5 kg -> 23.5
# wooden dining chair 0.21 m^3 5.0 kg -> 23.4
# so one number serves them all, which is just as well: the category that reaches here is a
# bare "chair" with no statement of which sort it is. A stool is the exception and needs its
# own, because it is small enough that its fixed frame dominates its enclosed volume (0.12 m^3,
# 5 kg -> 41.7).
"chair": 25.0, "stool": 40.0, "table": 20.0, "desk": 22.0,
"storage": 55.0, "cabinet": 55.0, "wardrobe": 50.0, "shelf": 45.0,
"bed": 35.0, "sofa": 30.0, "refrigerator": 110.0, "dishwasher": 120.0,
"oven": 120.0, "washer": 150.0, "radiator": 160.0,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
"""Run Room.py authoring and its optional polish passes."""

import json
import shutil

from litereality_agent.pipeline.context import RunContext
Expand Down Expand Up @@ -53,6 +54,24 @@ def run(context: RunContext, options: dict) -> StageResult:
if rc:
return result

# A ROOM THAT RAN OUT OF STEPS IS NOT A FINISHED ROOM. The step budget lands the session
# gracefully and exits 0, which is right — hitting it means "time's up", not "this is broken",
# and the work so far is kept. But it exits 0 through the same path as a room the model
# considered done, so without this the two are indistinguishable from outside and a truncated
# room is reported as a completed stage.
try:
summary = json.loads(
(context.authored_room.parent / ".author_result.json").read_text(encoding="utf-8"))
except (OSError, ValueError):
summary = {}
if summary.get("ended_early"):
result.warnings.append(
f"authoring stopped on {summary['ended_early']} after "
f"{summary.get('calls', '?')} tool-calls (budget "
f"{summary.get('step_budget', '?')}) — the room is as far as it got, not finished. "
f"Raise it with --author-steps."
)

passes: list[tuple[str, str, list[object]]] = []
if options.get("refine_objects"):
refine_args: list[object] = [
Expand Down
13 changes: 13 additions & 0 deletions src/litereality_agent/pipeline/scene_init/flow.py
Original file line number Diff line number Diff line change
Expand Up @@ -665,6 +665,19 @@ def summarize(results: list[dict]) -> None:
openings = r.get("openings", [])
ok_op = sum(1 for o in openings if o["status"] == "ok")
print(f" openings: {len(openings)} doors/windows ({ok_op} references ok)")

# A REFERENCE THAT FAILED IS NOT A SMALLER NUMBER, IT IS A DIFFERENT PIPELINE. On the
# fallback path the object is built from the raw evidence sheet instead of a clean render,
# so the geometry is measurably worse — and the only thing that said so was `0 references
# ok`, which reads as a count rather than as an error. A missing API key silently degraded
# every object in the room and the run still reported success.
failed = [o for o in (list(r["objects"]) + list(openings)) if o["status"] != "ok"]
if failed:
reasons = {o.get("error") or o["status"] for o in failed}
print(f" ⚠ DEGRADED: {len(failed)} of {n_obj + len(openings)} references fell back "
f"to the raw evidence sheet — these objects are built from worse input")
for reason in sorted(reasons)[:3]:
print(f" {reason}")
if "routing" in r:
print(
f" routing: {r['routing']['procedural']} procedural, {r['routing']['trellis']} trellis"
Expand Down
29 changes: 28 additions & 1 deletion src/litereality_agent/pipeline/simulate/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -106,11 +106,36 @@ def run(context: RunContext, options: dict) -> StageResult:
for key, note in (("no_sidecar", "no compiled physics"),
("unreadable_sidecar", "unreadable physics sidecar"),
("unplaceable", "physics could not be placed"),
("placement_rejected", "physics rejected as mis-placed")):
("placement_rejected", "physics rejected as mis-placed"),
("authored_no_package", "authored into Room.py, so no object package")):
names = report.get(key) or []
if names:
warnings.append(f"{len(names)} object(s) with {note}: {', '.join(map(str, names[:6]))}")

# A SCENE THAT STARTS INTERPENETRATING IS NOT SIM-READY, HOWEVER WELL IT EXPORTED. Overlap at
# t=0 is stored energy the solver has to discharge, so the first thing an episode does is throw
# furniture. This is the one check that speaks for the whole file rather than for one object.
if report.get("loads_clean") is False:
worst = report.get("initial_overlaps") or []
detail = "; ".join(f"{o['bodies']} {o['mm']}mm" for o in worst[:3])
warnings.append(f"scene starts interpenetrating in {len(worst)} place(s): {detail}")
elif report.get("loads_clean") is None and report.get("load_check_error"):
warnings.append(f"could not load the exported scene to check it: "
f"{report['load_check_error']}")

# THE HEADLINE NUMBER, SAID WHETHER OR NOT ANYTHING FAILED. Every warning above fires on a
# named object going wrong; none of them fires on the ordinary case of an authored room whose
# fixtures never had a package to begin with, and that case is the majority of the colliders.
# A scene where most of the physics was invented at export time is a worse scene, and a run
# that does not say so is reporting a success it has not earned.
coverage = report.get("sidecar_coverage")
if coverage is not None and coverage < 1.0:
warnings.append(
f"{report.get('derived_colliders', 0)} of {report.get('colliders', 0)} colliders "
f"({(1 - coverage) * 100:.0f}%) were derived at export time, not read from a compiled "
f"sidecar — mass, friction and pivots for those came from a category table"
)

if options.get("shake"):
# The report is the run's stdout, which `run_module` tees into the log. A video is asked
# for because a number saying "Chair0 moved 0.42 m" is not reviewable and a clip is.
Expand All @@ -124,6 +149,8 @@ def run(context: RunContext, options: dict) -> StageResult:
details = {"scene": str(scene_dir(context, seed=seed) / "scene.xml"),
"source": "seed" if seed else "authored",
"from_sidecar": len(report.get("from_sidecar") or []),
"sidecar_coverage": report.get("sidecar_coverage"),
"derived_colliders": report.get("derived_colliders"),
"bodies": {k: report.get(k) for k in ("structure", "free", "attached",
"articulated", "colliders")}}
return StageResult("simulate", StageStatus.COMPLETED, details=details, warnings=warnings)
Loading
Loading