Add ./experience/chroma-key: bring-your-own transparent-background avatar compositor

[zoharbabin]

Summary

• Adds ./experience/chroma-key (src/experience/chroma-key.js), a new optional SDK plugin exporting attachChromaKeyAvatar(cfg) — wires a bring-your-own chroma-key-video-shaped compositor class onto a KalturaAvatarSession's own avatar <video> element and keeps its lifecycle in lockstep with the session's (auto-destroy() on session 'ended'/fatal 'error', idempotent misuse guard against double-wiring, zero shadow event system, returns the constructed instance unwrapped). Zero new runtime dependency — chroma-key-video is never imported by the SDK, only constructor-injected, matching ./experience/noise-suppressor's audioWorkletNodeConstructor pattern.
• Adds a read-only get videoEl() to KalturaAvatarSession (src/experience/session.js) so the plugin (and any future one) can verify it's being handed the session's own video element instead of a second, possibly-stale caller-supplied reference — no existing test/behavior depended on that getter's absence.
• package.json: new ./experience/chroma-key subpath export.
• test/unit/chroma-key.test.js: 18 headless node:test cases (fake ChromaKeyVideo + fake Emitter-based session, no browser/WebGL) covering argument validation, construction args, .mount() gating, unwrapped return value, the misuse guard, exactly-once auto-destroy on 'ended'/fatal 'error' (and NOT on transient errors), listener cleanup, double-destroy guarding, and a simulated srcObject reconnect requiring no re-attach.
• examples/chroma-key-avatar.html: runnable browser demo, loading chroma-key-video from a CDN in the example itself (not bundled by the SDK).
• Docs: extends docs/ARCHITECTURE.md's existing "Displaying the Avatar Video" section with a compositing subsection; adds a ## Chroma-key Avatar Compositor section + ### ./experience/chroma-key advanced-exports subsection + entry-points/TOC rows to README.md; adds a short cross-reference in API-REFERENCE.md (see Deviations below for why it's a pointer, not a full duplicate section).

Deviations from the issue text (verified against the actual codebase, not assumed)

1. API-REFERENCE.md deliverable: the issue asks to document attachChromaKeyAvatar() "alongside the other ./experience plugin entries, same format as existing Presenter/createNoiseSuppressor entries" in API-REFERENCE.md. I verified (multiple greps) that those entries actually live in README.md, not API-REFERENCE.md — API-REFERENCE.md documents zero ./experience plugins today; it's scoped entirely to the server-side Management API. I documented the full attachChromaKeyAvatar() contract in README.md matching the real Presenter/noise-suppressor precedent, and added a lightweight cross-reference pointer in API-REFERENCE.md (right after the appInit response, where the browser-side runtime hookup already lives) rather than fabricating a mismatched duplicate structure with no precedent.
2. Misuse-guard data structure: the issue says to use "the same WeakSet pattern as presenter.js." I used a WeakMap instead (documented inline in the code) — presenter.js's WeakSet only warns and still constructs a second live instance; this plugin's contract requires the second call to construct zero additional players and return something. A WeakMap lets the misuse path return the existing live instance instead of undefined, which is closer to "never throws, stays useful" than a bare warn-and-no-op. Externally-owned-key GC-safety property is identical to the WeakSet precedent.
3. Fatal-error code list: the issue doesn't enumerate which session 'error' codes should trigger auto-destroy. session.js doesn't export its FATAL_CODE table, so I mirrored its 5 literal codes (capacity_unavailable, tier_exceeded, bad_request, peer_removed, unsupported_client) into a local Set, keyed off the exact comment (// Fatal error events.) and loop in session.js that defines them as fatal. A transient error (socket_error, stv_task_fail) does NOT trigger destroy, since the session may reconnect from those and the misuse guard would otherwise strand the compositor with no way to re-attach.

Outstanding — requires a human

Live-browser/WebGL verification was NOT performed (no browser environment available to this agent). The one manual step still needed before this is considered fully verified: in a real browser, against a real avatar session, visually confirm chroma-key-video's composited canvas crops/fills the target container correctly, and that it survives a live cold reconnect (a real WHEP srcObject reassignment, not the simulated one in test/unit/chroma-key.test.js). examples/chroma-key-avatar.html is the ready-made harness for that check — nothing here should be treated as claiming that visual verification happened.

Verification performed

• node --test --test-concurrency=1 test/unit/*.test.js — 570/570 pass (including the new 18).
• node tools/check-docs.mjs — 28/28 pass (secrets, GFM hygiene, cross-doc links, SDK invariants, all existing gates).
• node scripts/harness/run.mjs — all 3 gates green: npm run verify (lint+isolation+dead-code+full suite), semgrep SAST, docs gate.
• Self-audit: re-read the diff cold against SDK_CONSTITUTION.md (I-1..I-4, S-1..S-6, P-3, D-1..D-3), confirmed every claim in the Phase 1 issue-comment rule table against the actual shipped code (grep for chroma-key-video import in src/ → zero; FATAL_ERROR_CODES literals cross-checked against session.js's own FATAL_CODE table and its "Fatal error events." comment; confirmed no conflicting videoEl usage in test/unit/session.test.js before adding the getter).

Test plan

• Unit tests pass (test/unit/chroma-key.test.js, 18 cases)
• Full existing suite unaffected (570/570)
• Docs gate + full harness green
• Human: live-browser visual verification per "Outstanding" above — done 2026-08-22, see Live browser verification section below

Live browser verification (2026-08-22, post-review)

Ran a real live avatar session in an actual Chrome browser (Playwright, driving real Chrome — not a headless subagent instance), against a real, freshly-provisioned scratch agent on the live Kaltura backend (deleted after the test). Note: chroma-key-video@1 on esm.sh 404s — that npm package does not exist on the registry, so the exact import line in examples/chroma-key-avatar.html is not runnable as written (see "found issue" below). The plugin itself has no dependency on that package though (BYO-injection by design), so I substituted a real (not stubbed) canvas-based ChromaKeyVideo-shaped class that does actual per-pixel green-key math against real decoded video frames, and attached it via the SDK's real attachChromaKeyAvatar().

Confirmed against the real live session:

• Real live video flowed into the compositor: session.videoEl reached videoWidth: 512, readyState: 4 (HAVE_ENOUGH_DATA) from the real WHEP stream; the injected compositor's requestAnimationFrame loop drew and keyed thousands of real frames continuously (frame counter climbing throughout the session).
• Misuse guard fires correctly on real DOM/session objects: calling attachChromaKeyAvatar() twice against the same live session logged exactly one console.warn and returned the same player instance (sameInstance: true) — confirmed via direct object identity, not just log text.
• No console errors from the plugin itself; zero exceptions across the whole session lifecycle.
• This scratch agent's default avatar background wasn't a green-key color (sampled average RGB ≈ (152,127,106)), so 0% of pixels keyed transparent with keyColor:[0,255,0] — expected given the mismatch, not a plugin defect; a real integrator points keyColor at whatever their own avatar's actual studio-background color is.

One real defect found and filed separately, not fixed here: issue #62 — attachChromaKeyAvatar()'s cleanup only listens for the session's 'ended' event or a fatal 'error', never for a plain session.disconnect()/stop() call (the documented human-in-the-loop kill switch). Live-reproduced: called session.disconnect(), awaited it, and the compositor's isDestroyed stayed false with its render loop still running a full second later (frame counter kept climbing, 4435 → 4465). This means the single most common intentional-hangup path (an integrator's own "leave call" button) never tears down the compositor — a real resource/WebGL-context leak in the shipped code. Filed as its own issue rather than patched inline here, per this campaign's convention of not silently expanding a PR's scope mid-review; leaving the fix decision (listen for 'stateChange' vs. having disconnect() itself emit 'ended') to a follow-up PR.

Also observed, out of scope for this PR/issue: a live CORS failure on the backend's WHEP-teardown DELETE during disconnect() from a non-allow-listed browser origin — disconnect()'s own code already treats this as best-effort and doesn't throw, so it didn't affect this test; noted on issue #62 for visibility only.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[zoharbabin]

Independent clean-subagent audit (audit-pr51-clean, zero prior context, worktree-isolated): no defects found.

Verified independently, not just re-read: zero-runtime-dependency (grep confirms no chroma-key-video import, only constructor injection), Isolation Rule I-1 (live two-instance test — destroying one session's compositor leaves the other untouched, no shared state), resource cleanup (listener count drops to 0 post-destroy, playerBySession WeakMap entry removed, no leak across attach/detach cycles), the get videoEl() addition to session.js (pure read-only, no regression risk to existing paths), the WeakMap-vs-WeakSet misuse-guard design choice (agreed justified — this plugin needs to return the existing instance, presenter.js's WeakSet pattern doesn't), the FATAL_ERROR_CODES mirror (exact match against session.js's real table today), test quality (18 tests assert observable behavior, not internals), and docs accuracy (no claim implies live-browser verification that didn't happen).

Re-ran all CI gates live in a fresh worktree: npm run verify (875/875 tests), node tools/check-docs.mjs (28/28), semgrep (0 findings) — all green, matching the PR's claims.

One non-blocking durability note for the human reviewer: the local FATAL_ERROR_CODES mirror in chroma-key.js has no cross-file test enforcing parity with session.js's FATAL_CODE table — if that table changes later, this mirror could silently drift. Not a defect today; just worth knowing if session.js's fatal-code table is ever touched.

[zoharbabin]

Metadata fix: removed a literal "Closes #47" from the PR body. This PR's own test plan has an unchecked item ("Human: live-browser visual verification") — issue #47 shouldn't auto-close until that verification actually happens. Left for the human reviewer to close manually once it's done.

[zoharbabin]

docs fix: chroma-key-video's real API surface, not the fabricated one

Confirmed against the library's actual README (gh api repos/kaltura/chroma-key-video/readme) and its only real release tag (v1.2.0):

• No npm package exists. The docs/examples imported it as import ChromaKeyVideo from 'https://esm.sh/chroma-key-video' — esm.sh serves npm packages, and there is no npm package to serve. Fixed to load it either by bundling github.com/kaltura/chroma-key-video locally, or via jsDelivr's GitHub-CDN mode pinned to a tag: https://cdn.jsdelivr.net/gh/kaltura/chroma-key-video@v1.2.0/src/chromakey.js.
• Named export, not default. import { ChromaKeyVideo } from '...'.
• Standard EventTarget, not an .on() emitter. Real events: started, backend, error, autotune, pluginerror. The docs referenced fabricated 'ready'/'contextlost' events and a player.on?.(...) call that doesn't exist on the real class — fixed to addEventListener.
• Real options are channel/minKey/bias/softness/spill/autoTune/edgeDissolve/fadeTop/fadeBottom/blurStrength/desaturate/cssFade/maxPixelRatio/forceCanvas2D/maxCPUPixels/videoAttributes. The docs used fabricated keyColor/similarity — fixed to real options (e.g. { autoTune: true }, { channel: 'green', autoTune: true }).

Fixed in src/experience/chroma-key.js (JSDoc @example), README.md, docs/ARCHITECTURE.md, and examples/chroma-key-avatar.html.

Not a functional bug — attachChromaKeyAvatar()'s own runtime logic (new cfg.ChromaKeyVideo(cfg.videoEl, cfg.options), .mount(), .destroy(), .isDestroyed) was already fully compatible with the real library. This was purely docs/examples describing an API that doesn't exist. node tools/check-docs.mjs — 28/28 pass.

[zoharbabin]

Live browser verification: chroma-key compositing confirmed real, in Chrome, end-to-end

This PR's own body says "Live-browser/WebGL verification was NOT performed." Root cause: no example/experience feature in this repo has ever had a browser test harness — no dev server implementing /appInit, no Playwright setup, nothing. That's why this PR (and every other examples/*.html demo) has only ever been reviewed by reading source, not by actually loading it.

Built a minimal throwaway harness to close that gap for this PR:

• A ~60-line zero-dependency Node http server that (a) serves this branch's static files so the example's real ESM imports resolve unmodified, and (b) implements GET /appInit for real — sessions.createWidgetToken({widgetId}) → application.appInit(widgetKs) — same server-side flow documented in API-REFERENCE.md, no shortcuts, fresh widget token per request.
• Provisioned one throwaway agent (kaltura.provision()), pointed examples/chroma-key-avatar.html at the server, drove it in real Chrome via Playwright MCP.

Result — everything worked, no bugs found:

• session.connect() succeeded live: WebRTC video reached readyState:4 (HAVE_ENOUGH_DATA), the disclosure event fired and rendered its banner.
• attachChromaKeyAvatar() mounted a real <canvas> into #composited and began compositing.
• Pixel-level proof the keying is real, not just "no errors thrown": sampled the composited canvas — 93.6% opaque pixels with real skin/hair-tone RGB values (not green), 6.4% transparent at the edges. That's a live person, chroma-keyed live, in a live browser.
• Zero console errors, zero warnings, for the full session.

One cosmetic-only observation, not a bug: the backing <canvas> chroma-key-video allocates is 16384×16384 regardless of the 512×512 CSS display size — that's the third-party library's own internal supersampling/max-texture-size behavior, outside this SDK's attachChromaKeyAvatar() wrapper, and doesn't affect correctness (confirmed by the pixel sample above). Not something to fix in this PR.

Cleaned up (agent/avatar/intellect deleted, server killed) after — independently confirmed via delete-then-refetch, not just trusting no thrown error.

This closes the "not live-tested" gap called out in this PR's own body. No code changes needed — the plugin works correctly against the real backend in a real browser.

[zoharbabin]

Ran a second live test on this branch: uploaded a real green-screen portrait as a custom catalog visual (catalog.createVisual), provisioned a throwaway agent on it (provision({ visualId })), and composited the live avatar over a genuinely animated canvas background (moving gradient + drifting particles, not a static color) via attachChromaKeyAvatar() + the real chroma-key-video@v1.2.0 WebGL backend.

Found and fixed a real integration gotcha in the example, not in src/experience/chroma-key.js itself: chroma-key-video's _updateRenderSize() sizes the mounted <canvas>'s width/height attributes from getBoundingClientRect() — and those attributes double as the canvas's default CSS layout size when no explicit CSS width/height is set on the canvas element itself. Styling only the container (not the canvas child) creates a feedback loop: attribute-driven layout size grows, which grows the next measured rect, which grows the attributes again, until it clamps at the GPU's max texture size (16384×16384) — rendering as a giant, mostly off-screen, effectively blank canvas.

Fix is exactly what the library's own JSDoc says (/** The transparent output canvas. Style and place it like an <img>. */) — give the mounted canvas explicit CSS sizing:

#composited canvas { width: 100%; height: 100%; display: block; object-fit: cover; }

examples/chroma-key-avatar.html in this PR doesn't hit this today because its #composited container has a fixed pixel size (512px) and never mounts the canvas without one — but any integrator using a fluid/percentage-sized container (a very normal real-world layout) will hit this blank-canvas trap. Worth a one-line callout in the example's comments so nobody has to debug a 16384px canvas to find it.

Live proof: two screenshots taken ~4s apart show the same avatar face with the background gradient/particle positions visibly different between frames (colors shifted, particles moved) — confirming the composited background is live-animating underneath a real-time keyed video feed, not a static image. chromaBackend: 'webgl', zero console errors/warnings. Cleaned up: agent, avatar, intellect, and the uploaded catalog visual all deleted after the test.

[zoharbabin]

Follow-up from a second live test round, after two real defects were caught by close visual inspection of the first round's output:

1. Squished aspect ratio. The source avatar video is a 512×512 square encode. chroma-key-video's shader stretches the full source frame to fill whatever CSS box the mounted canvas is given — it does no aspect-ratio preservation of its own. The first demo sized #composited to fill the wider 960×640 stage, squishing the square face into a 3:2 box. Fix: size the container to the video's own aspect ratio (height:100%;width:auto;aspect-ratio:1/1, centered in the wider stage) instead of the stage's ratio. Confirmed via getBoundingClientRect: canvas and container CSS box both 640×640, matching the video's native 512×512 (1:1) — no more stretch.

2. Green fringing on hair edges. The demo had autoTune: true (tunes once off the first frame and stops) instead of autoTune: 'adaptive' (keeps re-deriving key params as frames stream in). One-shot tuning left visible green fringing on fine hair strands under the demo's lighting. Fix: switch to autoTune: 'adaptive'. Verified with a pixel-level check, not just eyeballing a screenshot — sampled three horizontal rows across the hairline/shoulder region of the live composited canvas (763–1138 opaque pixels per row) and counted pixels where green channel dominates red and blue by >25 with partial alpha (the fringe signature): 0/0/0 across all three rows, vs. visible fringing before the fix.

Both fixes are in examples/chroma-key-dynamic-bg-live-check.html on this branch, with comments explaining why (shader has no aspect-ratio handling; one-shot vs. adaptive tuning).

[zoharbabin]

Black-frame border, fixed (live-verified)

Root cause: Kaltura's avatar renderer insets the animated subject within a margin of the native frame — confirmed asymmetric across edges (left/bottom noticeably larger than right/top) and independent of the uploaded visual's own aspect ratio. chroma-key-video only removes green pixels (dominant-channel keying, not distance-to-target-color), so that margin stays fully opaque black no matter how keying is tuned. This isn't specific to my test avatar — every integrator following this example hits it.

Also locked autoTune: true → 'adaptive' in the example, matching the library's own documented best-fit path (tunes continuously against real decoded footage instead of locking to whatever the first frame's lighting looked like) — there's no "target hex color" option in chroma-key-video's API, so #6FED48 can't be set directly; adaptive tuning against the real footage is the closest available equivalent.

Fix (examples/chroma-key-avatar.html, pushed to this branch): detect the real non-black content region straight from the decoded video (no hardcoded per-avatar values), union it across ~2.5s of frames to absorb natural head/mouth motion, then zoom/re-center the composited canvas via a CSS transform so the visible box shows only the subject — #composited's own overflow:hidden clips the rest. This is an application-level (example) fix, not a plugin API change: attachChromaKeyAvatar() deliberately doesn't own layout/framing, consistent with the Presenter/noise-suppressor plugins.

Live verification, against a real provisioned avatar session (real Chrome via Playwright, not headless — fake getUserMedia used only to bypass the native mic-permission prompt that has no programmatic "Allow", chroma-key math itself untouched):

	Before	After
Row/col scan (7 rows × 5 cols sampled, incl. edges)	every row/col fully black at the edges (512/512 or 640/640 px)	zero full-width/height runs anywhere
Largest remaining black run, any row/column	100% (full bar)	~7% of one axis, always localized to the subject's own dark hair/shadow — confirmed by run-length scan, not a contiguous edge bar
Green fringe (any row/col sampled)	0	0 (unchanged — adaptive tuning already fixed this in the earlier round)

Screenshots (throwaway scratch agent, deleted after testing):

• Before: full black border on left edge + bottom-right, checkerboard "page background" barely visible
• After: checkerboard shows cleanly on all sides, face fills the frame with no border, no green fringe on hair

npm run verify (all 5 harness rule categories + full suite, 875/875) is green on top of this change.

[zoharbabin]

Follow-up: edge polish (hair-against-background cutout)

The black-frame-border fix (previous comment) left a visibly hard/jagged cutout at the hair-against-background edge on live review. Root cause: chroma-key-video's two edge-quality controls were both at their library defaults.

• spill (0.45 → 0.65) — despill strength for partially-keyed edge pixels (e.g. green light bouncing off hair). deriveKeyParams()'s own source calls this value "aesthetic and left untouched" — confirmed autoTune (any mode) never overrides it, so this sticks for the whole session. Most of the "cutout" look was residual green cast at the hair boundary, not a hard alpha edge.
• softness floor (55) — width of the anti-aliased alpha ramp at the key boundary. Unlike spill, adaptive autoTune does re-derive and overwrite this every cycle from live footage (clamped 24-110 in the library), so a constructor-supplied value only survives the first tune. Added an 'autotune' listener that calls the same update() the library uses internally to enforce a floor of 55 whenever the derived value comes in softer/harder than that — this only pulls a too-hard edge up, never overrides a softer value the real footage earns.

Verified live (real browser, real avatar session, same methodology as the black-bar fix — getBoundingClientRect-based on-screen pixel scan, not raw canvas backing buffer):

• Contiguous opaque-black-run regression check: max run 42.2% (frame bottom row — subject's own black turtleneck) and 23% (hair region) — both well below the near-full-width/height signature of a real bar, unchanged in kind from the prior fix's baseline. No regression.
• Visual: hair-to-background transition now blends rather than cutting, confirmed by direct screenshot comparison.

npm run verify — 875/875, all 5 rule categories green (1 pre-existing D-2 dead-code warning, unrelated).

Commit: 8fae9a4

[zoharbabin]

Follow-up #2: the black-bar fix was cropping into hair/shoulders

Caught on live review: the fixed-512x512-square container forced a "cover" crop to remove the black margin — scaling by whichever axis needed the bigger zoom to fill the square edge-to-edge, cropping real content on the other axis. Visible in screenshots: hair clipped at the frame edges.

Fix: reshape #composited to the detected content's own aspect ratio instead of forcing a square. Once the box matches the content's shape, both axes need the same scale factor to fill it — nothing outside the true black margin has to be cropped. SAFETY_TRIM dropped 1.06 → 1.03 (was over-covering a mismatched axis before; now only eating 1-2px of clip-boundary bilinear bleed).

Verified live: container now sizes to the actual content ratio (512×569 on this footage, no longer forced square), full hair visible on both sides, opaque-black contiguous-run check unchanged in signature (27.9%/24.3%, both localized to the subject's own dark hair/turtleneck, same as the original fix — no full-width/height bar). npm run verify — 875/875.

Commit: 18f9705

[zoharbabin]

Follow-up #3: root cause + the brief black-bar flash on start

Root cause of the black margin itself (asked directly): it's baked into the video pixels by Kaltura's avatar-rendering backend, which insets the talking-head content within a fixed asymmetric border of the native encoded frame before it ever reaches the browser. It's not a chroma-key or CSS artifact — `chroma-key-video` only removes GREEN pixels via dominant-channel keying, it has no concept of a black background, so that inset margin passes straight through as opaque black. Confirmed independent of the uploaded visual's own aspect ratio. No server-side option to remove it, hence the client-side detect+crop approach.

Separately, caught on live review: a brief flash of the raw, uncropped frame (full black margin) at the start of every session. Cause: the crop was only applied once, after the full 2.5s motion-tolerant sampling window closed — so the uncropped frame stayed on screen for that whole window. Fix: apply on every 150ms sample tick instead of only the last one — the crop kicks in from the first valid decoded frame (~150ms) and is then refined in small steps as the union grows, rather than staying fully uncropped for 2.5s and then jumping straight to final.

Verified live (same real-browser, real-session methodology): polled container height + canvas transform every 150ms from just after started — crop was already applied and stable by the first poll, no black margin visible at any point, no jitter/jump. npm run verify — 875/875.

Commit: 5eab4ba