fix(scene): reuse unchanged VLM inference

[lusunn111]

Problem

Scene repeatedly spent visual-model calls on unchanged inputs. The visual-tier detector polled the same cached JPEG without checking for meaningful change, and the image-grounded relation pass reran against stable visible objects, geometry, and camera state. Failures retried on the normal worker cadence, so endpoint outages could consume the shared model-call budget quickly.

Implementation

• Add exact and perceptual frame fingerprints using SHA-256 plus the maximum 4x4-block RGB RMS over a 32x32 sample. The default threshold is 0.01, which tolerates JPEG/sensor noise without averaging away small local objects.
• Reuse successful visual detections on unchanged frames. New camera deliveries refresh projected observations, while frozen deliveries neither refresh objects nor spend cache-expiry inference.
• Bound detection cache reuse with SCENE_VLM_CACHE_MAX_AGE_SEC (default 120 seconds); 0 disables reuse across new deliveries.
• Retry temporarily blocked local projection/publication from cached model output without making another model call.
• Cache whole-scene relation results against visible objects, captions, projected boxes, 3D geometry, camera intrinsics and pose, prompt input, model configuration, and frame fingerprint. Geometry is compared with explicit tolerances instead of rounded equality buckets.
• Distinguish processed, cached, failed, and backoff relation outcomes so no-call backoff rounds preserve live semantic edges without aging them, while real failures use bounded hysteresis.
• Treat parseable relation responses whose entries are all filtered as successful empty results. Detection and relation lists filter malformed items independently; only transport, parse, or invalid top-level shapes enter endpoint backoff.
• Refresh cached relation edge timestamps and reset their stale-round count.
• Log cumulative processed, skipped, retried, and failed inference counts.
• Expand the focused Scene CI job to cover scene graph, VLM geometry, image relations, and cache suppression.

Validation

• python3 -S -m pytest -q system/scene/tests/test_scene_graph.py system/scene/tests/test_vlm_inference_cache.py system/scene/tests/test_perception_vlm_geometry.py system/scene/tests/test_image_relations.py
  • 50 passed; one pre-existing asyncio deprecation warning.
• ruff check --select E9,F,I on the changed cache, detector, relation, builder, and regression-test modules.
• python3 -m compileall -q system/scene/scene_service system/scene/tests/test_vlm_inference_cache.py system/scene/tests/test_image_relations.py
• git diff --check
• python3 scripts/check_commit_authorship.py --base origin/dev --head HEAD
  • 3 commits passed.

Compatibility

• No capability contracts, RPCs, generated code, or persisted state formats change.
• Existing JPEG-only and (JPEG, float timestamp) fetchers remain accepted; the Scene service now passes the hub delivery count when available.
• Cache thresholds, cache age, and retry bounds have documented environment overrides.
• No migration is required.

Fixes #207

[enkerewpo]

@robonix-ci test

[github-actions]

✅ robonix CI — 16/16 passed (100%)

suite	scenario	result	rounds
flow	object_navigation	✅	4
builtin	fault_recovery_builtin	✅	3
builtin	file_roundtrip	✅	1
builtin	run_command	✅	1
cap	camera_snapshot	✅	1
cap	explore_smoke	✅	2
cap	lidar_snapshot	✅	1
cap	mapping_save	✅	1
cap	memgraph_failure_lesson	✅	1
cap	memgraph_roundtrip	✅	3
cap	memory_roundtrip	✅	1
cap	scene_object_fixture	✅	1
cap	speech_speak	✅	1
cap	voiceprint_list	✅	1
flow	fault_recovery	✅	2
flow	patrol_observe	✅	3

LLM analysis
This run tested the merged PR #231 'fix(scene): reuse unchanged VLM inference' on commit e3d99d1. The change adds perceptual frame fingerprinting and bounded exponential backoff to the Scene VLM object detector and image-relation inferer, plus a new CI job and unit tests. All 16 Webots scenario tests passed, and the new pytest suite for VLM cache suppression passed. No failures were observed.

Change summary: Added CI job 'scene-vlm-cache' to run pytest on system/scene/tests/test_vlm_inference_cache.py; Added vision_cache.py with FrameFingerprint, InferenceCounters, fingerprint_jpeg, fingerprint_bgr, frames_equivalent; Modified perception_vlm.py to add frame change threshold, failure backoff, and caching of last successful detections.
Test result: Webots CI ran 16 scenario tests covering navigation, exploration, mapping, memory, speech, and scene object listing; all passed (16/16). The scene service logged repeated VLM JSONDecodeError failures but the new backoff logic suppressed retries as designed, keeping the scene graph functional. The new pytest suite for VLM cache suppression passed.
Suggested fix / watchout: No fix needed. Watch for the fake-vlm JSONDecodeError in CI; it is expected and the backoff handles it. Consider adding a test that the backoff does not starve the scene graph when VLM is down.
Watchouts: The fake-vlm endpoint returns non-JSON, causing repeated VLM failures; the backoff prevents excessive retries but the scene graph never gets semantic edges in this environment.; The new cache uses a 32x32 perceptual hash; very small object changes might be missed, but tests cover meaningful changes.; The CI job installs pytest and dependencies but does not run the full scene test suite; only the new cache tests are run..

Report: open HTML report.
Action run: open GitHub Actions run.
Raw logs: scenario JSONL, provider logs, simulator logs, final caps, and boot logs are in the same artifact.

[HeartLinked]

Nice direction — the caching and retry accounting are a real improvement. Two things look blocking to me, though, because both reduce perception coverage on a healthy system under default settings.

1. The default frame-change threshold blinds the visual tier to small objects

frames_equivalent uses a whole-frame RMS over a 32x32 sample, so a local change is diluted by area (rms ~= sqrt(area_fraction) * delta / 255). Measured on 640x480 with this branch's own vision_cache, threshold=0.04:

 32x32 px object   rms=0.0285  SKIPPED
 40x40 px object   rms=0.0378  SKIPPED
 48x48 px object   rms=0.0469  inferred

Driving the real VLMObjectDetector._tick(): robot parked, a 40x40 px cup appears on a desk -> zero model calls over 150 ticks (~10 min). _DETECTION_PROMPT asks for cup / bottle / book / tool, which at ~2 m are exactly 40-60 px at 640x480.

Meanwhile the artefacts the threshold exists to absorb measure far below it — the 32x32 downsample already removes them:

q88 -> q60 re-encode  rms=0.0027     sensor noise sigma=10  rms=0.0031
default threshold     0.0400

And there is no maximum cache age: _last_success_frame only changes on a successful inference, so this is permanent suppression, not a delay.

Suggested: default ~0.005-0.01, and/or compare the max per-block RMS instead of the frame mean; plus a bounded SCENE_VLM_CACHE_MAX_AGE_SEC so a mis-tuned threshold degrades into a delay rather than blindness.

2. A parseable answer whose edges are all filtered is booked as an endpoint failure

edges = parse_image_relations(raw, box_to_oid)
if raw_edges and not edges:
    self._record_failure(self._clock())
    return []

near is deliberately outside IMAGE_RELATION_VOCAB (it is in the text path's RELATION_TYPES), and _normalize_relation maps no synonyms. So a healthy endpoint answering {"edges":[{"source":1,"target":2,"relation":"near"}]} counts as a failure. Reproduced at the 30 s rebuild cadence:

t=  0s  failed=1  retry_in=30s
t= 90s  failed=3  retry_in=120s
t=330s  failed=4  retry_in=300s   <- pinned at the cap, never recovers

Three effects stack: the backoff never clears (_last_success_edges is never written, so the cache can't short-circuit it); returning [] instead of None makes builder.py skip the text fallback, which does support near; and with MAX_STALE_ROUNDS=2 / INTERVAL_SEC=30 the semantic edges empty out after ~60 s. On dev the same response is just an authoritative empty round with no backoff, so this is a regression.

Suggested: only transport/parse failures (HTTP error, exception, non-JSON, missing or non-list edges) should call _record_failure; "parsed but no in-vocabulary edges" should count as processed and cache []. Same for _call_vlm, where any(not isinstance(item, dict) ...) discards a whole batch and enters backoff over one bad element — a per-item filter would be closer to the previous behaviour.

Checked and fine

edge.method = "cached" is safe (method is only read at builder.py:363 and in store.py); the deepcopy discipline is correct and _project_to_world doesn't mutate its input; not refreshing _last_success_frame on a cache hit correctly avoids a slow-drift ratchet; the relation signature plus sorted(nodes, key=object_id) is suitably conservative; _rgb_jpeg's tuple return has only one consumer; Pillow/numpy are already declared deps.

Minor, non-blocking: hub.latest stamps with local arrival time (_LatestSlot.write uses time.time()), not a source timestamp, so the comments are slightly off; cached edges keep their original updated_at; rounding T_cam_map to 1e-3 may make the relation cache hit far less often on real hardware than in the tests.

Happy to re-review once 1 and 2 are addressed.

[lusunn111]

@HeartLinked Thanks for the detailed measurements. Addressed in a08f5ee:

1. Small-object coverage: frame comparison now uses max 4x4-block RMS over the 32x32 sample, with a 0.01 default. Regression tests cover q88->q60/q40 recompression, sigma=10 noise, and 16/24/32/40 px local changes at 640x480. SCENE_VLM_CACHE_MAX_AGE_SEC defaults to 120 s for newly delivered frames; a frozen hub delivery never spends an expiry call.
2. Filtered-empty responses: only transport/parse/invalid top-level shapes enter failure backoff. Out-of-vocabulary or malformed list items are filtered independently, and a parsed empty result is processed and cached. Detection batches now use the same per-item policy.

The non-blocking points are covered too: Scene passes the hub delivery count instead of describing the local arrival stamp as a source timestamp; cached edges refresh updated_at and reset stale_rounds; relation geometry uses explicit tolerances instead of rounded equality buckets. Typed relation outcomes also keep no-call backoff rounds from aging live graph edges or fanning out into text calls.

Validation: 50 passed in the focused Scene suite, plus Ruff E9/F/I, compileall, diff-check, and authorship checks.

[lusunn111]

CI follow-up: the failed Rust job was unrelated to the Scene diff. Rust 1.98 introduced clippy::chunks_exact_to_as_chunks for the existing system/liaison/src/voice.rs:1292; the same line is present on current origin/dev.

I applied the behavior-preserving Clippy fix in b890baf and validated with Rust 1.98:

• cargo fmt --all -- --check
• cargo clippy --workspace --all-targets -- -D warnings
• cargo test --workspace --all-targets

All passed locally. The new fork workflows are now awaiting maintainer approval (action_required); please approve and run them when convenient.

[lusunn111]

@HeartLinked @enkerewpo Conflict resolved in 23fe10a by merging current dev and keeping the upstream voice.rs implementation from cec06ee; GitHub now reports the PR as mergeable and voice.rs is no longer part of the PR diff. Local validation passed: cargo fmt --all -- --check, workspace Clippy with warnings denied, cargo test --workspace --all-targets, and the focused Scene suite (50 passed). The refreshed Docs Check, CI, and Commit Authorship workflows are action_required for this fork update; please approve them and re-review when convenient.

[enkerewpo]

LGTM