feat(sim): spatialize world audio

[axelpey]

Summary

• derive the listener and sound-source positions from the simulator viewer's authoritative active camera, robot pose, and prop transforms
• keep robot-camera views at full loudness while third-person orbit view uses a smooth inverse-distance-style falloff with a 2 m near field and quiet audible floor
• apply the same gain path to Mad Mars motor audio, robot speech, and positioned NPC speech; queued TTS clips bind to the latest scene state when playback begins

Validation

• node --test webapp/tests/*.test.js
• npm run typecheck && npm run build:lib in sim/viewer
• pytest -q tests/test_challenge_runtime.py tests/test_household_npc.py (19 passed)
• pytest -q ros2_ws/src/brain/brain_client/test/test_tts.py (8 passed)
• targeted Ruff format/lint checks and git diff --check
• live simulator: Mad drive audio, robot TTS, and NPC TTS played through the browser; NPC transport identified resident_alex; orbit-distance falloff and first-person restoration were exercised; no audio playback errors appeared in browser diagnostics

Stacked on #750 because it owns the simulator motor-audio stream and PCM player. The stack includes #750's latest follow-up commit and is based on current main.

[greptile-apps]

Greptile Summary

The PR spatializes simulator audio using the active camera and current robot or prop positions, while preserving full volume in robot-camera views.

• Publishes authoritative audio-perspective snapshots from the simulator render loop and clears them during stage teardown.
• Applies a shared distance-based gain path to motor PCM, robot speech, and positioned NPC speech.
• Carries NPC source identifiers through the challenge, TTS transport, and browser playback pipeline.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
sim/viewer/src/simStage.ts	Publishes perspective after each primary render and now clears global audio state on both detach and final destruction.
sim/viewer/src/scene.ts	Derives listener and sound-source positions from the rendered active camera, robot pose, and visible prop transforms.
webapp/js/simSpatialAudio.js	Implements shared source tracking, orbit-view attenuation, smoothing, and full-volume fallback when no simulator perspective is active.
webapp/js/ttsAudio.js	Decodes source-aware TTS envelopes and binds each queued clip to current spatial state when playback starts.
webapp/js/motorSoundAudio.js	Routes the simulator motor stream through the shared robot-source gain.
ros2_ws/src/brain/brain_client/brain_client/transport/tts.py	Propagates queued audio-source metadata and publishes source-aware simulator TTS payloads while retaining the legacy robot payload.
sim/challenges/40_household_orders/runtime.py	Associates each resident’s speech with its authoritative simulator prop name.

Sequence Diagram

sequenceDiagram
  participant Viewer as Simulator viewer
  participant Spatial as Spatial audio state
  participant ROS as ROS audio topics
  participant Player as Browser audio player
  Viewer->>Spatial: Publish active view, listener, and source positions
  ROS->>Player: Deliver motor PCM or TTS clip with source
  Player->>Spatial: Follow source when playback begins
  Spatial-->>Player: Apply current distance-based gain
  Viewer->>Spatial: Clear perspective on detach or destroy
  Spatial-->>Player: Restore full-volume default

Loading

_{Reviews (2): Last reviewed commit: "fix(sim): clear spatial audio on stage t..." | Re-trigger Greptile}

[DavidDobas]

Rewrote this branch as a single commit on top of current main (which now has #750 merged, with the PCM player folded into motorSoundAudio.js):

• Dropped the two nav commits (feat(nav): follow the selected drive speed mode and its test). That feature was decided against; it was on the branch only because this one was cut from it. cmd_vel_mux.py, navigation_speed.py, app.cpp, the settings template and the speed-mode UI are untouched again.
• Dropped the tests (simSpatialAudio.test.js, pcmAudioPlayer.test.js, the new test_tts.py case) per the project rule; kept the signature updates to existing tests.
• simSpatialAudio.js is ~45 lines instead of 108. The viewer already dispatches a snapshot on every rendered frame, so the rAF smoothing loop and channel bookkeeping are gone (the motor path uses gain.setTargetAtTime, which smooths for free). Validation of our own in-process CustomEvent is gone too.
• /tts/audio in sim is always JSON ({audio, source}) instead of raw-base64-or-JSON, which removes the sniffing branch in both Python and the browser. Hardware never publishes on this topic, and the webapp ships with the brain, so there is no compatibility case.
• Unknown sources play at full volume. Environment speech with no prop was tagged "environment", which has no position, so in orbit view it played at the 8 % floor. Now source is null and the browser leaves it at 1.

Size: the PR is now 13 files, +165 −40 (was 28 files, +754 −60). Verified: ruff clean, basedpyright error count unchanged from baseline (the Mac has no ROS stubs), test_tts.py and the challenge runtime tests pass, sim/viewer typechecks, no tsc errors in the touched webapp files. Not yet exercised in a live simulator.

[DavidDobas]

It works and is now a small change. But idk if we want this, if I'm zoomed out, I can't hear the robot