[iris] Investigate noisy memray 'Failed to compress input file' logs

[Calvin-Xu]

Describe the bug
Some Iris task logs periodically emit Failed to compress input file, often alongside a memray deactivation message, even when the task continues normally. This is noisy and makes it look like a task-local failure even when the real failures are unrelated.

To Reproduce

1. Inspect logs from a long-running qsplit replay child, e.g. under /calvinxu/dm-qsplit240-300m-6b-20260406-022411.
2. Wait for a profiling/deactivation interval or scan the archived task logs.
3. Observe repeated Failed to compress input file messages while many affected tasks keep running or later succeed.

Expected behavior
Task logs should not emit a recurring compression error unless profiling actually failed and the task should surface that as a real error.

Additional context
Current evidence suggests this is not the cause of task failures. The exact string does not appear in Marin/Iris source, including the DuckDB log-store path at lib/iris/src/iris/cluster/log_store/duckdb_store.py. Memray is only referenced in explicit profiling code paths such as lib/iris/src/iris/cluster/runtime/profile.py and lib/iris/src/iris/cluster/providers/k8s/tasks.py. That makes this look more like an upstream memray/runtime profiler issue or platform-side profiling hook than a bug in normal task scheduling or log compression. We should confirm where memray is being invoked in production task containers and suppress or downgrade this message if it is benign.

[claude]

🤖 Plan:

• Locate memray call sites in iris
• Identify source of "Failed to compress input file"
• Duplicate-work preflight
• Post analysis

Research

The string Failed to compress input file is not in marin/iris — it is emitted by upstream memray's LZ4 sink when finalizing a trace file. It is logged from inside the target process (libmemray runs in-process via ptrace injection), which is why it shows up in task stderr alongside memray deactivation messages while the task keeps running. It is benign: the on-disk .bin trace is still readable; only the post-run compression step fails (typically a short write when memray attach detaches).

In iris, memory profiling for k8s tasks calls memray attach --native against PID 1 of the task container via kubectl exec. The profiler is not started by the task itself, so any periodic appearance corresponds to someone (controller/operator) issuing ProfileTask(memory=...) against those pods, or a scheduled profiling hook.

Relevant code

• k8s/tasks.py#L1251-L1276 — _profile_memory runs memray attach in-pod via kubectl exec
• runtime/profile.py#L120-L124 — build_memray_attach_cmd (--native, --duration, --output)
• runtime/process.py#L575-L617 — process-runtime memory profile with stub fallback
• runtime/entrypoint.py#L39 — task venvs install memray unconditionally

Proposed Approach (human decision needed, not a drop-in fix)

The warning is emitted by code running inside the task process, so iris cannot filter it without either patching/pinning memray or filtering the kubelet log stream downstream. Options, ordered by cost:

1. Confirm the trigger before changing anything: grep controller logs for ProfileTask calls against /calvinxu/dm-qsplit240-300m-6b-... and correlate with the Failed to compress input file timestamps. If these come from a scheduled/periodic profile hook rather than an operator, that hook is the real fix — stop profiling healthy tasks on a timer.
2. If the hook is intentional: lower cadence, or switch to raw format (the is_raw path skips transform but the in-process compressor still runs on teardown, so this alone will not silence it).
3. Upstream fix: file a memray issue to downgrade Failed to compress input file to debug or include the underlying errno, then pin the fixed version in _build_pip_install_args.
4. Log-store filtering at duckdb_store.py is possible but fragile; not recommended.

Scope: investigation plus either a config/ops change or an upstream memray patch, not a <100-line in-repo bugfix, so I am not opening a PR. Happy to proceed with step 1 (correlating controller profile RPCs with the noisy task logs) on request.

[rjpower]

We've disabled the periodic memray profiles entirely (they were causing some crashes).