[OPIK-7279] [SDK] fix: convert trace lifecycle to upsert-only (no trace.update()/end())

[JetoPistola]

Details

QA of opik-hermes 0.1.1 surfaced the Opik SDK warning on every Hermes message:

OPIK: Calling Trace.update() shortly after creation with batching enabled may cause data loss.

That warning is the symptom of violating the codebase-wide upsert-only rule for Opik integrations: when an entity starts, send the partial (started) entity; when it finishes, re-send the same entity (same id) with the full/finished payload and let the SDK's batching layer coalesce the two sends into one row. An update()/end() shortly after create is the anti-pattern. So this is not "silence a warning" — it's adopting the upsert pattern for the trace lifecycle, which removes the warning as a side effect.

Spans were already upsert-compliant (created fully-formed in a single trace.span() call) and are left untouched — only the trace lifecycle needed the change.

Changes

• observability/opik/lifecycle.py — finish_trace now finalizes via a single client.trace(id=state.trace.id, ...) re-send instead of trace.update(output=...) + trace.end(). The re-send replays the full create payload (name / thread_id / input / start_time / metadata / tags) plus output + end_time — not just output+end_time. client.trace(id=...) builds a whole CreateTraceMessage, so any omitted field is sent as null and the backend's last-write-wins merge would clobber the create, landing an NA trace (name/thread_id → None). start_time is pinned at create time so the re-send carries the same value, not a fresh now().
• observability/opik/state.py — carries the create kwargs on TraceState.create_kwargs; evict_stale_locked finalizes evicted traces via the same full-payload upsert re-send (+ end_time) instead of trace.end(); imports get_client and fails open if no client is available.
• observability/opik/hooks.py — comment refresh only, no behavior change.

The create is still flushed as its own batch before the finalize re-send, so the NA-trace race (name/thread/input lost to batch coalescing on a fast turn) remains guarded.

Change checklist

• User facing
• Documentation updated (if needed)
• Tests added/updated (if needed)
• Breaking changes documented (if any)

Issues

• OPIK-7279

Testing

• Unit (102 pass): FakeOpik.trace() now models real Opik's upsert coalescing — a call carrying a known id merges into the existing trace as a finalize re-send rather than minting a new row. FakeTrace.update()/.end() now raise, so any regression back to the forbidden pattern fails loudly. Added test_finalize_is_upsert_not_update_or_end and test_finalize_resend_replays_full_create_payload (guards the NA-trace regression: the finalize upsert must replay the full create payload, not just output+end_time).
• E2E (e2e/run_e2e.sh, e2e/run_e2e_real_opik.sh): capture full Hermes output and assert the may cause data loss warning is absent. assert_journal.py merges trace rows last-write-wins including nulls (mirroring real Opik) and asserts the finalized trace still carries name + thread_id + output + end_time with zero trace.update() PATCHes — so a partial re-send that nulls name/thread is caught. Verified to pass the fix and fail the old pattern.
• Manual staging verification (controlled A/B): ran one real agent turn (real model) against a live Opik backend with the unfixed plugin vs. this branch, same harness. Unfixed emits the batching warning twice (Trace.update() + Trace.end()); this branch emits zero. The finalized trace queried back from the backend carried a real name/thread_id/input/output/end_time with no NA trace and clean llm+tool spans (no NA spans). This confirms the warning-absence check genuinely surfaces the warning (it fired on old code) rather than silently missing it.
• Local ruff check / ruff format --check / pytest all green (matches CI).

Documentation

N/A — no user-facing docs. Behavior change is internal to the plugin's trace lifecycle; the upsert-only rule it adopts is the existing codebase-wide integration convention.