feat(jspsych): capture <canvas> pixel state for replay fidelity

The MutationObserver cannot see drawing operations inside <canvas> — the
pixels are not in the DOM tree. Without explicit snapshots, replay
reconstructs the canvas element but renders it blank, so anything the
participant drew (sketchpad strokes, plugin-rendered visualizations) is
invisible in the playback even though cursor motion is recorded.

Introduce a generic canvas.snapshot RecordedEvent that carries a PNG
data URL keyed by node id. Every <canvas> in the trial display element
is tracked; snapshots fire on gesture release (mouseup/touchend, RAF-
deferred so the post-gesture paint has landed), at the moment a canvas
is removed from the DOM (jsPsych core clears the display element via
innerHTML = "" between trials, which would otherwise lose final state),
and on stop() of an in-flight trial. Per-canvas snapshots are throttled
to 250ms and deduped by data URL so unchanged canvases don't produce
noise. Tainted canvases (cross-origin images without CORS) throw on
toDataURL — these are caught and skipped so the recording survives.

The implementation is plugin-agnostic: it walks the display element for
any <canvas>, has no awareness of plugin-sketchpad, and works equally
for any plugin or experiment that draws to a canvas.

Author: claude

docs/reference/session-recording-schema.md

@@ -117,12 +117,11 @@ Every node in `initial_dom` is assigned a monotonically-increasing integer `id`.
 
 **Element-specific extras.**
 
-- `canvas_size`: present on `<canvas>` elements; carries `width`/`height` attributes at snapshot time. Note that the *contents* of a canvas (the rendered pixels) are not captured.
+- `canvas_size`: present on `<canvas>` elements; carries `width`/`height` attributes at snapshot time. The pixel contents at any later moment are recorded as separate [`canvas.snapshot`](#canvas-snapshots) events keyed by node id.
 - `media_src`: present on `<video>` and `<audio>` elements; carries `currentSrc` if available, falling back to the `src` attribute.
 
 **What is not captured.**
 
-- Canvas/WebGL pixel content. Only the element and its dimensions are recorded.
 - Audio/video media data. Only playback events (`media.play`, `media.pause`, etc.) and the source URL are recorded.
 - Shadow DOM. jsPsych does not use it in core; recordings will not capture mutations inside shadow roots.
 - CSSOM rule-level edits (`sheet.insertRule`, `deleteRule`, etc.) made on a captured stylesheet after `start()` without touching the owning element's children. These bypass `MutationObserver` and so are not reflected in [`stylesheet_events`](#stylesheet-events). Edits via `<style>.textContent = …` or `appendChild`/`removeChild` on the inner text node are tracked.
@@ -192,7 +191,8 @@ type RecordedEvent =
   | ClipboardRecord
   | MediaRecord
   | FocusRecord
-  | ScrollRecord;
+  | ScrollRecord
+  | CanvasSnapshot;
 ```
 
 ### DOM mutations
@@ -288,6 +288,31 @@ type MediaRecord = {
 
 `media.time` records throttled `timeupdate` events at roughly 4 Hz and is intended for replay scrubbing rather than exact synchronization.
 
+### Canvas snapshots
+
+```ts
+interface CanvasSnapshot {
+  type: "canvas.snapshot";
+  t: number;
+  node: number;     // id of the <canvas> element
+  data_url: string; // PNG data URL: "data:image/png;base64,…"
+}
+```
+
+The `MutationObserver` cannot see drawing operations inside `<canvas>` (the pixels are not in the DOM tree). Without these events, replay reconstructs the element but renders it blank, so anything the participant drew (e.g. plugin-sketchpad strokes) would be invisible.
+
+**Capture timing.** Snapshots are taken at three moments:
+
+1. After a gesture release (`mouseup` or `touchend`) anywhere within the display element. The actual `toDataURL` call is deferred to the next animation frame so the page has finished painting the post-gesture state.
+2. When a tracked canvas is removed from the DOM mid-trial (jsPsych core clears the display element via `innerHTML = ""` between trials, which would otherwise lose the final pixel state).
+3. When a recording is stopped while a trial is in flight (`stop("aborted")` etc.).
+
+**Throttling and dedupe.** Per-canvas, gesture-driven snapshots are throttled to one every 250 ms; removal- and stop-driven snapshots bypass the throttle so the final state always lands. Identical consecutive data URLs are skipped, so a canvas whose pixels did not change does not produce noise.
+
+**Tainted canvases.** If a canvas drew cross-origin images without CORS headers, `toDataURL` throws `SecurityError`. The recorder catches this, skips the offending canvas, and continues — the recording is never broken by an unreadable canvas.
+
+**Replay procedure.** Track each `<canvas>` you instantiate from `initial_dom` by node id. When a `canvas.snapshot` event arrives, decode `data_url` (e.g. into an `Image` whose `src` is the data URL) and `drawImage` it onto the canvas at `(0, 0)`. This overwrites whatever was there and brings the canvas to the recorded state.
+
 ### Focus events
 
 ```ts

packages/jspsych/src/modules/recording.ts

@@ -113,7 +113,8 @@ export type RecordedEvent =
   | ClipboardRecord
   | MediaRecord
   | FocusRecord
-  | ScrollRecord;
+  | ScrollRecord
+  | CanvasSnapshot;
 
 export type DomMutation =
   | { type: "dom.add"; t: number; parent: number; before: number | null; node: DomNode }
@@ -178,6 +179,19 @@ export type ScrollRecord =
   | { type: "scroll.window"; t: number; x: number; y: number }
   | { type: "scroll.element"; t: number; node: number; x: number; y: number };
 
+// Captures `<canvas>` pixel state as a PNG data URL. The MutationObserver
+// can't see drawing operations inside `<canvas>`; without these events a
+// replayer reconstructs the element but renders it blank, so anything the
+// participant drew (e.g. plugin-sketchpad strokes) would be invisible.
+// Emitted at gesture boundaries (mouseup/touchend) and at trial end, with
+// per-canvas throttling and a same-as-last-snapshot dedupe.
+export interface CanvasSnapshot {
+  type: "canvas.snapshot";
+  t: number;
+  node: number;
+  data_url: string;
+}
+
 export interface RngCall {
   t: number;
   fn: string;
@@ -192,6 +206,10 @@ export interface RngCall {
 const SCHEMA_VERSION = 1;
 const VIEWPORT_DEBOUNCE_MS = 100;
 const MEDIA_TIME_THROTTLE_MS = 250;
+// Per-canvas minimum gap between gesture-driven snapshots. Bounds the
+// `toDataURL` cost for users who rapidly click/release; trial-end
+// captures bypass this so the final state always lands.
+const CANVAS_SNAPSHOT_MIN_INTERVAL_MS = 250;
 
 export interface SessionRecorderOptions {
   jspsychVersion: string;
@@ -242,6 +260,14 @@ export class SessionRecorder {
   // remove their listeners when the trial ends. Cleared on detach.
   private mediaTrackedElements = new Set<HTMLMediaElement>();
 
+  // Strong ref to canvas elements within the current trial's display
+  // subtree. Used to drive deferred snapshotting after gestures and at
+  // trial end. Per-canvas throttle/dedupe state is held alongside.
+  private canvasTrackedElements = new Set<HTMLCanvasElement>();
+  private canvasLastSnapshot = new WeakMap<HTMLCanvasElement, string>();
+  private canvasLastSnapshotTime = new WeakMap<HTMLCanvasElement, number>();
+  private canvasSnapshotScheduled = false;
+
   // Reference to whatever `Math.random` was immediately before the recorder
   // started. Restored verbatim on stop so opting into recording does not
   // permanently alter `Math.random`, even if we auto-seeded.
@@ -335,6 +361,7 @@ export class SessionRecorder {
       this.flushPendingMouse();
       this.flushPendingScroll();
       this.flushPendingInput();
+      this.captureCanvasSnapshots(true);
       this.currentTrial.t_end = this.t();
       this.currentTrial = null;
     }
@@ -374,13 +401,15 @@ export class SessionRecorder {
     this.currentTrial.initial_dom = this.serializeNode(this.displayElement);
     this.attachTrialListeners();
     this.scanForMediaElements(this.displayElement);
+    this.scanForCanvasElements(this.displayElement);
   }
 
   onTrialFinish(trialData: unknown) {
     if (!this.currentTrial) return;
     this.flushPendingMouse();
     this.flushPendingScroll();
     this.flushPendingInput();
+    this.captureCanvasSnapshots(true);
     this.detachTrialListeners();
     this.currentTrial.t_end = this.t();
     this.currentTrial.trial_data = serializeJson(trialData);
@@ -506,6 +535,8 @@ export class SessionRecorder {
       this.mutationObserver = null;
     }
     this.detachMediaListeners();
+    this.canvasTrackedElements.clear();
+    this.canvasSnapshotScheduled = false;
     // Tear down only the trial-scoped handlers; session-scoped handlers
     // (resize, focus, blur, fullscreenchange) stay attached until stop().
     const remaining: typeof this.boundHandlers = [];
@@ -543,6 +574,8 @@ export class SessionRecorder {
             this.pushEvent({ type: "dom.add", t, parent: parentId, before, node });
             if (added instanceof HTMLMediaElement) this.attachMediaListeners(added);
             else if (added instanceof Element) this.scanForMediaElements(added);
+            if (added instanceof HTMLCanvasElement) this.trackCanvasElement(added);
+            else if (added instanceof Element) this.scanForCanvasElements(added);
           }
         } else if (r.type === "attributes") {
           const id = this.nodeIds.get(r.target);
@@ -578,6 +611,14 @@ export class SessionRecorder {
   }
 
   private releaseSubtree(node: Node) {
+    if (node instanceof HTMLCanvasElement && this.canvasTrackedElements.has(node)) {
+      // jsPsych core clears the display element via `innerHTML = ""`
+      // immediately after each trial, before `onTrialFinish` fires. Take
+      // a final snapshot here so the canvas's last pixel state is in the
+      // recording instead of being silently dropped on removal.
+      this.snapshotCanvas(node, this.t(), true);
+      this.canvasTrackedElements.delete(node);
+    }
     if (this.nodeIds.has(node)) {
       this.nodeIds.delete(node);
     }
@@ -919,6 +960,9 @@ export class SessionRecorder {
         button: e.button,
         target: this.targetId(e.target),
       });
+      // Gesture release is the moment a stroke completes; snapshot any
+      // tracked canvases so the drawing it produced reaches the replay.
+      if (type === "mouse.up") this.scheduleCanvasSnapshot();
     };
   }
 
@@ -931,6 +975,7 @@ export class SessionRecorder {
         y: tt.clientY,
       }));
       this.pushEvent({ type, t: this.t(), touches });
+      if (type === "touch.end") this.scheduleCanvasSnapshot();
     };
   }
 
@@ -1034,6 +1079,81 @@ export class SessionRecorder {
     this.mediaTrackedElements.clear();
   }
 
+  // -------- canvas snapshotting --------
+
+  // Walks `root` for `<canvas>` elements and registers each one for
+  // snapshotting. Called on trial load and when subtrees are added
+  // mid-trial via `dom.add`.
+  private scanForCanvasElements(root: Element) {
+    if (root instanceof HTMLCanvasElement) {
+      this.trackCanvasElement(root);
+      return;
+    }
+    const els = root.querySelectorAll("canvas");
+    for (const el of Array.from(els)) {
+      this.trackCanvasElement(el as HTMLCanvasElement);
+    }
+  }
+
+  private trackCanvasElement(canvas: HTMLCanvasElement) {
+    this.canvasTrackedElements.add(canvas);
+  }
+
+  // Defers actual snapshotting to the next animation frame so we wait
+  // until the page has had a chance to paint the post-gesture state
+  // (otherwise `toDataURL` could return the canvas as it was *before*
+  // the up event's listeners ran). Coalesced via a single scheduled flag
+  // so a flurry of mouseups doesn't queue redundant work.
+  private scheduleCanvasSnapshot() {
+    if (this.canvasSnapshotScheduled) return;
+    if (this.canvasTrackedElements.size === 0) return;
+    this.canvasSnapshotScheduled = true;
+    requestAnimationFrame(() => {
+      this.canvasSnapshotScheduled = false;
+      this.captureCanvasSnapshots(false);
+    });
+  }
+
+  // Throttles to at most one snapshot per canvas per
+  // `CANVAS_SNAPSHOT_MIN_INTERVAL_MS`, and dedupes by data URL so a
+  // canvas whose pixels did not actually change does not produce noise.
+  // `force` bypasses the throttle for trial-end captures so the final
+  // state of every canvas is always recorded.
+  private captureCanvasSnapshots(force: boolean) {
+    if (this.canvasTrackedElements.size === 0) return;
+    const now = this.t();
+    for (const canvas of this.canvasTrackedElements) {
+      this.snapshotCanvas(canvas, now, force);
+    }
+  }
+
+  // Pushes a `canvas.snapshot` event for one canvas, applying the
+  // per-canvas throttle (unless `force`) and skipping when the data URL
+  // is unchanged from the last snapshot. Pulled out of the loop above so
+  // it can also be called from `releaseSubtree` to capture the final
+  // pixel state at the moment a canvas is removed from the trial DOM —
+  // jsPsych core clears the display element via `innerHTML = ""` before
+  // `onTrialFinish` runs, so a trial-end-only flush would miss it.
+  private snapshotCanvas(canvas: HTMLCanvasElement, t: number, force: boolean) {
+    const id = this.nodeIds.get(canvas);
+    if (id === undefined) return;
+    if (!force) {
+      const last = this.canvasLastSnapshotTime.get(canvas) ?? -Infinity;
+      if (t - last < CANVAS_SNAPSHOT_MIN_INTERVAL_MS) return;
+    }
+    try {
+      const dataUrl = canvas.toDataURL();
+      if (this.canvasLastSnapshot.get(canvas) === dataUrl) return;
+      this.canvasLastSnapshot.set(canvas, dataUrl);
+      this.canvasLastSnapshotTime.set(canvas, t);
+      this.pushEvent({ type: "canvas.snapshot", t, node: id, data_url: dataUrl });
+    } catch {
+      // toDataURL throws SecurityError on tainted canvases (canvases
+      // that drew cross-origin images without CORS headers). Skip and
+      // never break the experiment.
+    }
+  }
+
   // -------- RNG --------
 
   private isNativeMathRandom(fn: typeof Math.random) {

packages/jspsych/tests/core/record-session.test.ts

@@ -329,6 +329,126 @@ describe("record_session option", () => {
     });
   });
 
+  describe("canvas snapshot capture", () => {
+    test("captures a final canvas.snapshot at trial end for canvases in the initial DOM", async () => {
+      const jsPsych = initJsPsych({ record_session: true });
+      await startTimeline(
+        [
+          {
+            type: htmlKeyboardResponse,
+            stimulus: '<canvas id="c" width="50" height="50"></canvas>',
+            on_load: () => {
+              const c = document.getElementById("c") as HTMLCanvasElement;
+              const ctx = c.getContext("2d")!;
+              ctx.fillRect(10, 10, 20, 20);
+            },
+          },
+        ],
+        jsPsych
+      );
+      await pressKey("a");
+
+      const rec = jsPsych.getSessionRecording()!;
+      const snaps = rec.trials[0].events.filter((e) => e.type === "canvas.snapshot");
+      expect(snaps.length).toBeGreaterThan(0);
+      const last = snaps[snaps.length - 1];
+      expect(typeof last.node).toBe("number");
+      expect(last.data_url).toEqual(expect.stringMatching(/^data:image\/png/));
+    });
+
+    test("emits no canvas.snapshot events when the trial has no canvas", async () => {
+      const jsPsych = initJsPsych({ record_session: true });
+      await startTimeline(
+        [{ type: htmlKeyboardResponse, stimulus: "<p>no canvas here</p>" }],
+        jsPsych
+      );
+      await pressKey("a");
+
+      const rec = jsPsych.getSessionRecording()!;
+      const snaps = rec.trials[0].events.filter((e) => e.type === "canvas.snapshot");
+      expect(snaps).toHaveLength(0);
+    });
+
+    test("tracks canvases added mid-trial via dom.add", async () => {
+      const jsPsych = initJsPsych({ record_session: true });
+      await startTimeline(
+        [
+          {
+            type: htmlKeyboardResponse,
+            stimulus: '<div id="host"></div>',
+            on_load: () => {
+              const c = document.createElement("canvas");
+              c.id = "late";
+              c.width = 32;
+              c.height = 32;
+              document.getElementById("host")!.appendChild(c);
+            },
+          },
+        ],
+        jsPsych
+      );
+      await pressKey("a");
+
+      const rec = jsPsych.getSessionRecording()!;
+      const snaps = rec.trials[0].events.filter((e) => e.type === "canvas.snapshot");
+      expect(snaps.length).toBeGreaterThan(0);
+    });
+
+    test("captures the canvas's final pixel state when it is removed mid-trial", async () => {
+      // jsPsych core clears the display element via `innerHTML = ""` after
+      // every trial, before `onTrialFinish` fires. The recorder must take
+      // its final snapshot at the moment of removal — a trial-end-only
+      // flush would race with the cleanup and miss the canvas entirely.
+      const jsPsych = initJsPsych({ record_session: true });
+      await startTimeline(
+        [
+          {
+            type: htmlKeyboardResponse,
+            stimulus: '<canvas id="c" width="20" height="20"></canvas>',
+            on_load: () => {
+              const c = document.getElementById("c") as HTMLCanvasElement;
+              c.getContext("2d")!.fillRect(0, 0, 10, 10);
+              c.remove();
+            },
+          },
+        ],
+        jsPsych
+      );
+      await pressKey("a");
+
+      const rec = jsPsych.getSessionRecording()!;
+      const snaps = rec.trials[0].events.filter((e) => e.type === "canvas.snapshot");
+      expect(snaps).toHaveLength(1);
+      expect(snaps[0].data_url).toEqual(expect.stringMatching(/^data:image\/png/));
+    });
+
+    test("a tainted canvas (toDataURL throws) does not break the recording", async () => {
+      const jsPsych = initJsPsych({ record_session: true });
+      await startTimeline(
+        [
+          {
+            type: htmlKeyboardResponse,
+            stimulus: '<canvas id="c" width="10" height="10"></canvas>',
+            on_load: () => {
+              const c = document.getElementById("c") as HTMLCanvasElement;
+              c.toDataURL = () => {
+                throw new DOMException("tainted", "SecurityError");
+              };
+            },
+          },
+        ],
+        jsPsych
+      );
+      await pressKey("a");
+
+      const rec = jsPsych.getSessionRecording()!;
+      // Recording still ends cleanly and the trial completes.
+      expect(rec.end_reason).toBe("finished");
+      const snaps = rec.trials[0].events.filter((e) => e.type === "canvas.snapshot");
+      expect(snaps).toHaveLength(0);
+    });
+  });
+
   describe("form-state capture", () => {
     test("captures input.value events for typed text", async () => {
       const jsPsych = initJsPsych({ record_session: true });