fix(screenpipe): keep the chosen frame as timeline evidence through a discard

A discarded observation had its media cleared, so the frame the agent had just
chosen was thrown away. Discard means the vault needs no note about the
observation — not that the day's visual timeline should have a blank where it
happened. media_data is what evidence.py turns into an evidence item's
image_filename, which is where an episode's representative_image comes from, so
clearing it is what left gaming sessions with no picture anywhere.

The selection now applies on every branch; discard and duplicate clear the media
paths only when no frame was chosen. Putting an image inside a note stays a
separate decision behind promote_image/retain_image, which most observations
should not take.

The rest of the shortlist is dropped once it has been chosen from — it exists to
be judged, and 2-6 frames per observation is several times the storage of the one
that was picked.

Author: AnkushMalaker

backends/advanced/src/advanced_omi_backend/services/observation_curation.py

@@ -106,10 +106,11 @@ def _observation_codex_settings(settings: Any = None) -> dict[str, Any]:
 
 Images named `frame-<id>.jpg` are a shortlist sampled across this observation's span,
 not a sequence to describe. Read them for what the session actually was — the game or
-document on screen, what was being done — and set `selected_frame_id` to the one that
-best represents it, or null if none is worth keeping. Prefer clearly legible content
-over an exact moment; "good enough" is the bar. Frames are also evidence you may cite
-in the summary, not only thumbnail choices.
+document on screen, what was being done — and set `selected_frame_id` to the frame that
+best depicts it. This is the observation's thumbnail on the visual timeline and is kept
+whatever you decide below, so choose one even when the observation itself is routine and
+you discard it; use null only when no frame depicts anything. Prefer clearly legible
+content over an exact moment; "good enough" is the bar.
 
 The screenshot is sparse supporting evidence, not permission to invent. `output` audio
 is system/media audio: character dialogue, lyrics, presenters, and game dialogue are
@@ -120,7 +121,8 @@ def _observation_codex_settings(settings: Any = None) -> dict[str, Any]:
 Use `text_update` for a small Daily/YYYY-MM-DD.md entry. Use `dedicated_note` only for a
 durable event/project/topic/place/media experience; choose a safe relative `.md` path.
 Use `promote_image` only when the ScreenPipe preview or one of the supplied Immich
-thumbnail candidates adds durable value; set `immich_item_id` only for the latter. Use `duplicate` only when the supplied canonical observation id is clear.
+thumbnail candidates adds durable value to the note itself; set `immich_item_id` only
+for the latter. Use `duplicate` only when the supplied canonical observation id is clear.
 Never write a fake conversation note for screen context.
 """
 
@@ -598,30 +600,43 @@ async def apply_curation_decision(
     action = decision["decision"]
 
     updates: dict[str, Any] = {"agent_reason": agent_reason}
-    # The frame the agent chose becomes the observation's representative image, so
-    # every downstream consumer — vault promotion, the 1280px source-media fetch, the
-    # timeline thumbnail — uses the frame that was actually judged to depict this
-    # session, not the one that scored highest before anyone looked at it.
+    # The frame the agent chose becomes the observation's representative image. It is
+    # kept whatever the decision below is, because it is *timeline* evidence, not vault
+    # content: discarding an observation means the vault does not need a note about it,
+    # not that the day's visual timeline should have a blank where it happened. Only
+    # `promote_image`/`retain_image` puts an image inside a note.
     #
-    # Applied in memory here because the promotion path below reads `item.media_data`,
-    # but written only on the branches that keep an image: discard and duplicate
-    # `$unset` these same paths, and Mongo rejects a `$set` and `$unset` of one path
-    # in a single update.
+    # The shortlist itself is dropped once it has been chosen from — it exists to be
+    # judged, and keeping 2-6 frames per observation is several times the storage of
+    # the one that was picked.
     selected = _selected_preview(item, decision)
-    selection_updates: dict[str, Any] = {}
     if selected is not None:
         item.media_data = selected["data"]
         item.media_content_type = selected["content_type"]
         item.media_filename = f"frame-{selected['frame_id']}.jpg"
         item.content_hash = hashlib.sha256(selected["data"]).hexdigest()
-        selection_updates = {
-            "media_data": item.media_data,
-            "media_filename": item.media_filename,
-            "media_content_type": item.media_content_type,
-            "content_hash": item.content_hash,
-            "metadata.preview_frame_id": selected["frame_id"],
-            "metadata.thumbnail_available": True,
-        }
+        updates.update(
+            {
+                "media_data": item.media_data,
+                "media_filename": item.media_filename,
+                "media_content_type": item.media_content_type,
+                "content_hash": item.content_hash,
+                "metadata.preview_frame_id": selected["frame_id"],
+                "metadata.thumbnail_available": True,
+            }
+        )
+    if item.media_previews:
+        updates["media_previews"] = []
+        item.media_previews = []
+    # Paths the discard/duplicate branches clear when no thumbnail was chosen. Mongo
+    # rejects a `$set` and `$unset` of the same path in one update, so a chosen frame
+    # takes precedence and nothing is unset.
+    _MEDIA_PATHS = (
+        "media_data",
+        "media_filename",
+        "media_content_type",
+        "content_hash",
+    )
     unset: tuple[str, ...] = ()
     if action == "duplicate":
         target_id = decision.get("duplicate_observation_id")
@@ -638,24 +653,13 @@ async def apply_curation_decision(
             raise ValueError("agent selected an invalid duplicate observation")
         updates["duplicate_of"] = str(target.id)
         updates["curation"] = "duplicate"
-        if item.lifecycle == "closed":
-            unset = (
-                "media_data",
-                "media_filename",
-                "media_content_type",
-                "content_hash",
-            )
+        if item.lifecycle == "closed" and selected is None:
+            unset = _MEDIA_PATHS
     elif action == "discard":
         updates["curation"] = "discarded"
-        if item.lifecycle == "closed":
-            unset = (
-                "media_data",
-                "media_filename",
-                "media_content_type",
-                "content_hash",
-            )
+        if item.lifecycle == "closed" and selected is None:
+            unset = _MEDIA_PATHS
     else:
-        updates.update(selection_updates)
         retain_image = bool(decision.get("retain_image")) or action == "promote_image"
         immich_item_id = decision.get("immich_item_id") if retain_image else None
         if (

backends/advanced/tests/test_observation_curation.py

@@ -326,12 +326,12 @@ def test_only_a_shortlisted_frame_can_be_selected():
 
 
 @pytest.mark.asyncio
-async def test_discarding_a_selected_frame_does_not_set_and_unset_one_path(monkeypatch):
-    """A discarded observation drops its image; it must not also write one.
+async def test_discarded_observation_keeps_the_chosen_timeline_thumbnail(monkeypatch):
+    """Discard means "the vault needs no note", not "the timeline needs a blank".
 
-    Mongo rejects a `$set` and `$unset` of the same path in one update, so applying
-    the agent's frame choice unconditionally made every discard of a closed
-    observation fail with "Updating the path 'media_data' would create a conflict".
+    The chosen frame is timeline evidence, so it survives a discard. It must also not
+    be written and cleared in one update: Mongo rejects a `$set` and `$unset` of the
+    same path, which previously failed every discard of a closed observation.
     """
 
     item = observation(
@@ -355,6 +355,35 @@ async def capture(target, fields, *, unset=()):
         observation_revision(item),
     )
 
+    fields, unset = seen[-1]
+    assert fields["curation"] == "discarded"
+    assert fields["media_data"] == b"jpeg"
+    assert fields["metadata.preview_frame_id"] == 7
+    assert not unset
+    assert not set(fields) & set(unset)
+    # The shortlist is spent once it has been chosen from.
+    assert fields["media_previews"] == []
+
+
+@pytest.mark.asyncio
+async def test_discard_without_a_chosen_frame_still_clears_media(monkeypatch):
+    """Nothing depicted the observation, so there is no thumbnail worth storing."""
+
+    item = observation(lifecycle="closed", curation="curating")
+    seen: list[tuple[dict, tuple]] = []
+
+    async def capture(target, fields, *, unset=()):
+        seen.append((fields, unset))
+        return True
+
+    monkeypatch.setattr(observation_curation, "_apply_curation_fields", capture)
+
+    await apply_curation_decision(
+        item,
+        {"decision": "discard", "reason": "Routine", "selected_frame_id": None},
+        observation_revision(item),
+    )
+
     fields, unset = seen[-1]
     assert "media_data" in unset
     assert not set(fields) & set(unset)

docs/screenpipe.md

@@ -170,9 +170,19 @@ its store, so a missing frame is expected, does not fail the batch, and is recor
 
 The curation agent is then shown every fetched frame and picks the one that represents
 the session (`selected_frame_id`), or none. That selection — not the scorer's guess —
-becomes the observation's `media_data`, and the image it chooses is fetched at a bounded
-1280px only if it is promoted into the vault. ScreenPipe remains the high-resolution
-source and Chronicle never uploads a frame sequence.
+becomes the observation's `media_data`, and the rest of the shortlist is dropped: it
+existed to be judged, and storing every frame costs several times the one that was
+picked. ScreenPipe remains the high-resolution source and Chronicle never uploads a
+frame sequence.
+
+**A chosen frame is timeline evidence, so it survives a `discard`.** Discarding means
+the vault needs no note about the observation, not that the day's visual timeline should
+have a blank where it happened — `services/timeline/evidence.py` turns `media_data` into
+an evidence item's `image_filename`, which is what an episode's `representative_image`
+is drawn from. Only `promote_image`/`retain_image` puts an image *inside* a note, and
+that path additionally fetches the frame at a bounded 1280px and content-addresses it
+under `_media/`. The two decisions are independent: most observations should yield a
+timeline thumbnail and no vault image.
 
 The request is bounded: a shortlist is asked for at most twice, after which curation
 proceeds on text alone. It must be, because a frame ScreenPipe has pruned returns 404