Skip to content

Track host transcript sanitization and private-field filtering #688

Description

@xnne-bot

Background

memU's host bridging pipeline reads agent-owned transcript stores and materializes selected records into self-evolve inputs. Those stores routinely carry fields that are useful to the host runtime but should not be copied into memory/skill mining inputs: credentials, authorization headers, local paths, provider/session metadata, hidden instructions, reasoning payloads, or other private data.

The direct-input contract introduced by #666 is strict and deliberately asks application integrators to remove secrets before submission. Host adapters need the equivalent protection at their own normalization seam. This is not one universal parser change: each host has a different log dialect and trust boundary, so fixes should remain local to the adapter while sharing the same policy goal.

Coverage

Completed

In review

  • fix(hermes): filter private transcript fields #683 — Hermes: delete known SQLite-row timestamps and provider replay metadata while retaining the OpenAI-shaped message/tool semantics.
  • feat(hosts): add pi coding agent adapter #675 — Pi: delete known runtime/provider metadata, reasoning signatures, and tool-result details from prepared transcript output while preserving user/assistant/tool content. This sanitizer is in the open Pi host-adapter PR rather than a standalone filtering PR.

Investigated, intentionally not merged

  • fix(cursor): filter private transcript fields #685 — Cursor was investigated against real agent-transcripts/*.jsonl, then closed without merge. Its public/exported transcript boundary already contained only the conversation and text/tool fields needed by bridging; no concrete private/runtime JSON field was found that justified a permanent sanitizer. The only observed noise was a small <timestamp>…</timestamp><user_query>…</user_query> user-text envelope, whose removal changed one real session by 85 B (0.2%). That benefit did not justify binding the adapter to an additional Cursor-specific parser.

    Private/runtime-rich fields do exist in Cursor's internal store.db, but that SQLite schema is not the adapter's stable compatibility boundary. We deliberately do not switch to it merely to scrub fields. Keep using export JSONL with delete-only / preserve-unknown behavior; open a narrow Cursor filtering PR only when a real exported JSONL fixture identifies a specific field that must be removed.

Not yet audited

  • WorkBuddy
  • Cola

These unchecked hosts are not assertions that they leak data; they mean we do not yet have a reviewed raw-record sample and a documented sanitization decision. Each needs either (a) a narrow delete-only sanitizer with a regression fixture, or (b) an explicit evidence-based decision that its stable source boundary is already sufficiently clean, as with Cursor.

The generic adapter is intentionally excluded from this matrix: it processes user-selected, heterogeneous JSONL dialects and cannot safely own one universal field policy. A dedicated adapter should replace generic use once a host has a stable format; until then the caller must treat the selected source as an explicit trust boundary.

Keep this issue open

This is a living privacy/sanitization ledger, not a claim that all adapters are now conformant. Keep it open while any dedicated host can expose unreviewed private/runtime fields to bridging jobs.

For every new adapter or transcript-dialect change:

  1. inspect the actual raw records and identify fields that must never reach the materialized memory or skill JSONL;
  2. sanitize at the host adapter's existing normalization boundary, before shared bridging writes job inputs;
  3. add a focused regression fixture/assertion proving the private field is absent while required user/assistant and tool evidence remains;
  4. link the PR here and check it off when merged.

Do not solve this by silently swallowing entire transcripts or by adding a generic destructive scrubber that corrupts host-specific tool payloads. Preserve the minimum faithful conversation/tool evidence needed for evolving; remove only fields that do not belong in that input.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions