[issue/1002] Fix and enhance singleton sensor tagging with cache persistence

.github/copilot-instructions.md

@@ -48,6 +48,13 @@ Skills are loaded on-demand based on task context to optimize token usage:
 
 Skills are detected and loaded based on file type, task keywords, and context signals. Explicitly request a skill if the auto-detection doesn't load it.
 
+### Instruction Placement Policy (Critical)
+
+- Prefer skill files under `.github/skills/` for detailed procedural rules.
+- Keep this file focused on high-level routing and references to canonical skill documents.
+- Avoid duplicating policy/checklist text across this file and skills.
+- If overlap is found, retain one canonical source and replace duplicates with a short pointer.
+
 ## Architecture Overview
 
 **Core Components:**
@@ -107,6 +114,20 @@ make -C tests unit-tests                              # Unit tests only
 make -C tests geometry-unit                           # Specific test (e.g., geometry)
 ```
 
+### Completion Gate For Test Tasks (Critical)
+
+For runtime test verification requirements, use
+`.github/skills/test-verification-gate.md`.
+
+### Containerized Test Image Freshness Gate (Critical)
+
+Use `.github/skills/test-verification-gate.md` as the single source of truth
+for image freshness checks, rebuild-before-test requirements, and retry policy
+for containerized test targets.
+
+Service-specific examples belong in each service guide (for controller, see
+`controller/Agents.md`).
+
 ## Code Patterns & Conventions
 
 **Python Packaging**:

.github/prompts/reflect.prompt.md

@@ -5,8 +5,26 @@ description: "Reflect on this conversation and suggest instruction updates"
 
 # Self-Reflection Task
 
-1. Review the entire conversation history.
-2. Identify patterns where I had to correct you or clarify my intent.
-3. Suggest specific additions or modifications to the `.github/copilot-instructions.md`, files under `.github/skills` directory, `Agents.md` in each service directory and other relevant documentation to prevent these issues in the future.
-4. Recommend any new 'Agent Skills', tools or prompts that would have made this task easier.
-5. Provide the output as a set of actionable diffs or markdown blocks.
+## Instruction Placement Rule (Critical)
+
+Before proposing documentation changes, apply this hierarchy:
+
+1. Put detailed procedural policy in the most specific skill file under `.github/skills/`.
+2. Keep `.github/copilot-instructions.md` as orchestration/entry-point guidance with pointers to skill files.
+3. Keep `Agents.md` files service-specific with concrete examples and commands, not duplicated global policy text.
+4. Do not duplicate the same checklist/policy text across global instructions and skill files.
+5. If overlap is unavoidable, keep one canonical source and replace duplicates with short references.
+
+6. Review the entire conversation history.
+7. Identify patterns where I had to correct you or clarify my intent.
+8. Suggest specific additions or modifications to the `.github/copilot-instructions.md`, files under `.github/skills` directory, `Agents.md` in each service directory and other relevant documentation to prevent these issues in the future.
+9. Recommend any new 'Agent Skills', tools or prompts that would have made this task easier.
+10. Provide the output as a set of actionable diffs or markdown blocks.
+11. Explicitly identify any missed instruction and classify the root cause as:
+    - discovery failure
+    - execution failure
+    - verification failure
+12. For test-related tasks, always include:
+    - the Makefile target that should have been run
+    - whether it was actually run
+    - the exact command and pass/fail summary (or the blocker)

.github/skills/test-verification-gate.md

@@ -0,0 +1,55 @@
+<!--
+SPDX-FileCopyrightText: (C) 2026 Intel Corporation
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# AI Agent Skill: Test Verification Gate
+
+Use this skill whenever a task adds or modifies tests.
+
+## Goal
+
+Ensure runtime verification is completed and reported consistently.
+
+## Required Checklist
+
+1. Select a repository Makefile target that covers the modified tests.
+2. Prefer a root target when practical (for example, `make run_unit_tests`).
+3. Otherwise select the narrowest scoped target in `tests/Makefile`
+   (for example, `make -C tests scenescape-unit`).
+4. If the selected target runs in a service `...-test` container image,
+   rebuild images for changed services before executing tests.
+5. Execute the target.
+6. If failures occur, confirm image freshness before code-level debugging:
+   - Rebuild the impacted service runtime and test images if not rebuilt.
+   - Rerun the same target once on fresh images.
+7. If still failing, fix and rerun the same target.
+8. Report exact command and concise pass/fail summary.
+
+## Image Freshness Mapping (Common)
+
+- Changed `controller/src/**` + `make -C tests scene-unit`:
+  - `make controller`
+  - `make -C controller test-build`
+  - then run `make -C tests scene-unit SUPASS=<password>`
+
+Apply the same pattern to other services: rebuild runtime + test image before
+running containerized test targets.
+
+## Blocked Execution Policy
+
+If execution is blocked (missing environment, skipped setup, unavailable
+runtime), report:
+
+1. What is blocked.
+2. The exact command that should be run once unblocked.
+3. Whether task completion is partial.
+
+## Not Sufficient
+
+- Lint success only
+- Syntax-only checks
+- IDE static errors only
+- Repeated reruns against stale container images
+
+These checks are useful but do not replace runtime test execution.

.github/skills/testing.md

@@ -15,6 +15,68 @@ This guide provides comprehensive instructions for AI agents to create high-qual
 - Each test should set up its own data and clean up after itself
 - Use mocking to isolate units from external dependencies
 
+## Verification Workflow For AI Agents (Mandatory)
+
+Use `.github/skills/test-verification-gate.md` for runtime verification,
+command selection, and completion reporting rules after creating or modifying
+tests.
+
+## Test Import Path Policy (Mandatory)
+
+Before adding imports or path setup in any new or modified test file, run this
+discovery workflow:
+
+1. Check shared pytest bootstrap files first:
+   - `tests/conftest.py`
+   - nearest local `conftest.py` in the target test directory tree
+2. Verify whether the required modules are already importable via existing
+   fixtures/path setup.
+3. Use direct imports (for example `from controller...`) when existing
+   `conftest.py` already establishes paths.
+4. Only add path manipulation if no appropriate shared bootstrap exists.
+5. If path setup is genuinely required, prefer adding it once in the nearest
+   relevant `conftest.py` rather than per-test-file setup.
+
+### Prohibited Pattern
+
+- Do not add `sys.path.insert(...)` blocks in individual test modules when
+  equivalent setup can live in shared `conftest.py`.
+
+### Completion Check For Test Authoring
+
+When creating or updating tests, report this explicitly:
+
+- whether `conftest.py` files were checked
+- where import-path setup is defined (file path)
+- confirmation that no unnecessary per-file `sys.path.insert` was introduced
+
+## Test Target Mapping Workflow (Mandatory)
+
+Run this workflow before executing any test command:
+
+1. Identify changed files.
+2. Classify each changed file scope: unit, functional, ui, perf, or integration.
+3. Resolve the concrete target from the relevant Makefile(s):
+   - `tests/Makefile`
+   - `tests/Makefile.functional`
+   - `tests/Makefile.user_interface`
+   - `tests/Makefile.sscape`
+4. Choose the narrowest target that directly validates the changed file(s).
+5. Run that target with required environment variables.
+
+### Anti-Miss Checklist
+
+- Do not treat unit target success as validation for functional test changes.
+- Do not run broad aggregate targets unless no narrow target exists or the user explicitly requests a sweep.
+- Do not report completion without runtime verification for the resolved target (unless blocked).
+- Always report: should-run target, whether it was run, exact command, and pass/fail summary (or blocker).
+
+### Quick Mapping Examples
+
+- `tests/functional/tc_sensors_send_mqtt_messages.py` -> `make -C tests sensors-send-events`
+- `tests/functional/tc_mqtt_sensor_roi.py` -> `make -C tests mqtt-sensor-roi`
+- `tests/functional/tc_tripwire_mqtt.py` -> `make -C tests mqtt-tripwire`
+
 ## Test Categories
 
 ### 1. Unit Tests
@@ -1047,13 +1109,16 @@ pytest -v -s  # -s shows print statements
 
 ## Test Checklist
 
-When creating a new test, verify:
+This checklist is mandatory before marking any testing task complete.
+If any item is not satisfied, the final response must explicitly state why.
+When creating or modifying tests, verify:
 
 - [ ] Test has Zephyr ID (NEX-T#####)
 - [ ] Test file named `test_*.py`
 - [ ] Test functions named `test_*`
 - [ ] Both positive and negative cases covered
 - [ ] Boundary conditions tested
+- [ ] At least one negative case exists per function or behavior under test, unless explicitly not applicable
 - [ ] Appropriate markers applied (`@pytest.mark.unit`, etc.)
 - [ ] Mocking used for external dependencies (unit tests)
 - [ ] Real data used for integration tests
@@ -1063,6 +1128,9 @@ When creating a new test, verify:
 - [ ] Test is independent (doesn't rely on other tests)
 - [ ] Fixtures used for shared data
 - [ ] Documentation strings explain what is being tested
+- [ ] Repo-preferred test command used for validation (prefer `make -C tests <target>` when available)
+- [ ] New or changed tests executed after the last code edit
+- [ ] Final response includes current pass/fail status and any warnings or known gaps
 
 ## Quick Reference
 

controller/src/controller/cache_manager.py

@@ -63,7 +63,12 @@ def refreshScenes(self):
 
       uid = scene_data['uid']
       if uid not in self.cached_scenes_by_uid:
+        # Creating new scene - check if there was an old scene with sensor cache
         scene = Scene.deserialize(scene_data)
+
+        old_scene = self._sensorNeedsRestoring(uid)
+        if old_scene:
+          self._restoreSensorCache(uid, old_scene, scene)
       else:
         scene = self.cached_scenes_by_uid[uid]
         scene.updateScene(scene_data)
@@ -73,9 +78,41 @@ def refreshScenes(self):
       for sensorID in scene.sensors.keys():
         self._cached_scenes_by_sensorID[sensorID] = scene
       self.cached_scenes_by_uid[scene.uid] = scene
+
+    # Clear old scene cache after processing all scenes
+    if hasattr(self, '_old_scene_cache'):
+      self._old_scene_cache = None
+
     self._cache_refreshed = get_epoch_time()
     return
 
+  def _sensorNeedsRestoring(self, uid):
+    # Check if any old scene has sensors with cache values that can be restored
+    if hasattr(self, '_old_scene_cache') and self._old_scene_cache:
+      return self._old_scene_cache.get(uid)
+    return None
+
+  def _restoreSensorCache(self, uid, old_scene, scene):
+    """Restore sensor cache values from old_scene to new scene"""
+    restored_count = 0
+    for sensor_id, old_sensor in old_scene.sensors.items():
+      if hasattr(scene, 'sensors') and sensor_id in scene.sensors:
+        new_sensor = scene.sensors[sensor_id]
+        restored = False
+        if hasattr(old_sensor, 'value'):
+          new_sensor.value = old_sensor.value
+          restored = True
+        if hasattr(old_sensor, 'lastValue'):
+          new_sensor.lastValue = old_sensor.lastValue
+          restored = True
+        if hasattr(old_sensor, 'lastWhen'):
+          new_sensor.lastWhen = old_sensor.lastWhen
+          restored = True
+        if restored:
+          restored_count += 1
+    if restored_count > 0:
+      log.debug(f"Restored sensor cache for {restored_count} sensor(s) in scene {uid}")
+
   def _refreshCameras(self, scene_data):
     for camera in scene_data.get('cameras', []):
       update_data = {}
@@ -188,6 +225,8 @@ def sceneWithRemoteChildID(self, childID):
     return self.cached_child_transforms_by_uid.get(childID, None)
 
   def invalidate(self):
+    # Preserve old scene cache for sensor value restoration
+    self._old_scene_cache = self.cached_scenes_by_uid if hasattr(self, 'cached_scenes_by_uid') else {}
     self.cached_scenes_by_uid = None
     if not hasattr(self, 'cached_child_transforms_by_uid') or self.cached_child_transforms_by_uid is None:
       self.cached_child_transforms_by_uid = {}

controller/src/controller/detections_builder.py

@@ -9,21 +9,21 @@
 from scene_common.timestamp import get_iso_time
 
 
-def buildDetectionsDict(objects, scene):
+def buildDetectionsDict(objects, scene, include_sensors=False):
   result_dict = {}
   for obj in objects:
-    obj_dict = prepareObjDict(scene, obj, False)
+    obj_dict = prepareObjDict(scene, obj, False, include_sensors)
     result_dict[obj_dict['id']] = obj_dict
   return result_dict
 
-def buildDetectionsList(objects, scene, update_visibility=False):
+def buildDetectionsList(objects, scene, update_visibility=False, include_sensors=False):
   result_list = []
   for obj in objects:
-    obj_dict = prepareObjDict(scene, obj, update_visibility)
+    obj_dict = prepareObjDict(scene, obj, update_visibility, include_sensors)
     result_list.append(obj_dict)
   return result_list
 
-def prepareObjDict(scene, obj, update_visibility):
+def prepareObjDict(scene, obj, update_visibility, include_sensors=False):
   aobj = obj
   if isinstance(obj, TripwireEvent):
     aobj = obj.object
@@ -37,7 +37,9 @@ def prepareObjDict(scene, obj, update_visibility):
   if not velocity.is3D:
     velocity = Point(velocity.x, velocity.y, DEFAULTZ)
 
-  obj_dict = aobj.info
+  # Build a fresh top-level dict per serialization so optional fields like
+  # sensors do not leak between scene, regulated, and external outputs.
+  obj_dict = dict(aobj.info)
   obj_dict.update({
     'id': aobj.gid, # gid is the global ID - computed by SceneScape server.
     'type': otype,
@@ -83,11 +85,37 @@ def prepareObjDict(scene, obj, update_visibility):
     if update_visibility:
       computeCameraBounds(scene, aobj, obj_dict)
 
-  chain_data = aobj.chain_data
-  if len(chain_data.regions):
-    obj_dict['regions'] = chain_data.regions
-  if len(chain_data.sensors):
-    obj_dict['sensors'] = chain_data.sensors
+  if hasattr(aobj, 'chain_data'):
+    chain_data = aobj.chain_data
+    if len(chain_data.regions):
+      obj_dict['regions'] = chain_data.regions
+
+    if include_sensors:
+      sensors_output = {}
+
+      # Copy sensor data while holding lock, then release
+      with chain_data._lock:
+        env_state_copy = dict(chain_data.env_sensor_state)
+        attr_events_copy = dict(chain_data.attr_sensor_events)
+
+      # Environmental sensors: timestamped readings
+      for sensor_id, state in env_state_copy.items():
+        values = state['readings'] if 'readings' in state and state['readings'] else []
+
+        sensors_output[sensor_id] = {
+          'values': values
+        }
+
+      # Attribute sensors: events as structured object
+      for sensor_id, events in attr_events_copy.items():
+        if events:
+          sensors_output[sensor_id] = {
+            'values': events
+          }
+
+      if sensors_output:
+        obj_dict['sensors'] = sensors_output
+
   if hasattr(aobj, 'confidence'):
     obj_dict['confidence'] = aobj.confidence
   if hasattr(aobj, 'similarity'):