feat(agent): HTML-to-PDF, ffmpeg and speech-to-text for the agent - #1738
Merged
Conversation
…cribe The three capabilities the agent kept refusing (#1738), each traced to a real prod refusal rather than invented: html-to-pdf nashs, 2026-09-01 — "save this 8.5x11 sign as a PDF". The PyMuPDF fallback ignored flexbox, the base64 logo and the web fonts, and emitted a garbled 400x600pt page. media kinneyk, 2026-09-01 — a .mov to convert and verify for Facebook/Instagram upload. transcribe demontek, 2026-09-03 — an .m4a to turn into a printable script. That failure named the cause exactly: no Transcribe permission on the execution role. SEPARATE from hyperframes-render on purpose, despite the overlapping toolset. That function's role may write ONLY to public-images/; this one reads and writes the owner's PRIVATE workspace prefix, because a scanned IEP or a staff recording must not become a public-by-link object. Merging them would widen a production role to avoid a second image — the wrong trade. Nothing here writes to public-images/ at all; publishing stays a deliberate separate act through psd-publish-file and its sensitivity gate. Security model: the function never accepts an identity. The relay injects workspacePrefix from the web-verified invocation context, and the caller supplies only workspace-relative paths, validated against absolute forms, `..` traversal, backslashes and control characters (DEL included). A caller cannot reach another owner's workspace because it never states who it is. ffmpeg is exposed as three named presets rather than caller-supplied argv: arbitrary arguments into ffmpeg are a command-injection and resource-exhaustion surface, and every real request so far has been one of these shapes. A test asserts no preset argument contains a shell metacharacter. Verified against the built image, not just in unit tests: - html-to-pdf round-tripped through the Runtime Interface Emulator returned a real %PDF- with MediaBox 612x792pt — exactly 8.5x11, the precise thing that was wrong before. Rendered to PNG and inspected: full-bleed gradient, flexbox space-between layout, the embedded base64 image and the serif font all present. - the social-mp4 preset turned a ProRes .mov into h264 High / yuv420p + aac, with the moov atom at byte 36 ahead of mdat at 3591 — +faststart doing the job that stops uploaders rejecting a file as corrupt. - the Dockerfile fails the build if Chromium cannot emit a %PDF- or the static ffmpeg/ffprobe cannot execute, so a broken image never reaches a push. 21 handler tests pass. One of them earned its keep immediately: an unknown ffmpeg preset was being validated AFTER the input was pulled from S3, so a typo would download hundreds of megabytes and then fail with "WORKSPACE_BUCKET is not configured" — the wrong error for the wrong reason. Everything free to check now runs before any I/O. Not yet wired: CDK construct and IAM, the relay route, and the skills. This commit is the capability itself, verified in isolation.
… skills Completes #1738. The Lambda landed in the previous commit; this is everything that makes it reachable, plus the three skills that expose it. CDK (infra/lib/constructs/compute/agent-media-function.ts) A separate function from hyperframes-render, on purpose. That role may write ONLY to public-images/; this one reads and writes owners' PRIVATE workspace prefixes, and carries an explicit DENY on public-images/ so "never publishes" is an IAM property rather than a code convention that survives only as long as nobody edits the handler. Transcribe is scoped to the `transcription-job/agent-media-*` name prefix the handler mints, so this role cannot read or disturb jobs started by anything else in the account — that missing permission is exactly what demontek's 2026-09-03 failure named. Relay (infra/agent-image/mantle_proxy.py) One fixed route, same shape as the HyperFrames one: no caller-selected target, no ARN, no identity. Identity fields are REJECTED by the payload allowlist rather than silently overwritten, so an attempt to reach another owner's workspace is visible instead of quietly corrected. `_resolve_invocation_identity` deliberately has NO old-tier fallback, unlike `_resolve_invocation_owner`. The workspace prefix decides whose objects the function may touch; if the web tier is too old to return it, media stays unavailable until the deploy catches up. Same web-tier-first ordering as every other cross-unit change here. Transport (skills/_shared/media-relay.js) Bytes go through the workspace broker into `.media-scratch/`, which is added to checkpointExclusions in workspace-policy.json. That is load-bearing, not tidiness: an ad-hoc upload into a checkpoint-MANAGED path is deleted by the next ensureWorkspaceCheckpoint, which rolls the workspace back to its manifest and delete-markers anything unexpected — scratch objects would vanish mid-turn. Being excluded also keeps them out of the container, which is right for a transport. Scratch is removed on every exit path; for psd-transcribe that matters more than tidiness, because the transport copy is someone's voice. Skills psd-print-pdf HTML -> real PDF. nashs asked twice. psd-media ffprobe + three named presets. kinneyk's .mov. psd-transcribe audio -> text. demontek's .m4a. Presets, never caller-supplied ffmpeg argv: arbitrary arguments into ffmpeg are a command-injection and resource-exhaustion surface, and every real request so far has been one of these three shapes. Each SKILL.md says plainly that its output is a LOCAL file — publishing stays a separate, deliberate act through psd-publish-file and its sensitivity gate. Verified rather than assumed - Rebuilt the image AFTER the complexity refactor and re-ran the smoke test, because the earlier pass no longer described the code: byte-identical result, 20924-byte PDF at MediaBox 612x792pt. - Drove the error paths through the real container: an .aiff refusal that names the converter, and `../escape.mov` stopped at the traversal check. - Proved the relay's path regex against 13 escape attempts before trusting it. - `cdk synth` succeeds; the two new AgentMedia IAM warnings are the same LOW tag-condition class the existing HyperframesRender role already carries, for the documented reason that object-level S3 grants cannot satisfy one. Also wired agent-media into CI with its own dependency install — the same gap that made the agent-cron suite run nowhere until CI caught it. Gate: lint clean (zero warnings — the fs access moved onto the shared validated-fs wrapper rather than being suppressed, and three over-complex functions were split), typecheck clean, 5918 jest, 904 agent-image python (+9 relay boundary tests), 21 handler tests.
krishagel
added a commit
that referenced
this pull request
Sep 6, 2026
psd-print-pdf shipped in #1738 and never printed a page in production. Found when Kris re-tested the interruption fix and asked for a PDF of the Ion Sudoku artifact: "headless Chromium is crashing... consistent across 4 attempts". The agent was blameless — psd-print-pdf has no local Chromium and correctly called the relay. psd-agent-media-dev was invoked 5 times: one that hung to the 120s Chromium timeout, then four ~1s failures. THE ACTUAL ERROR, and how it was obtained Not inferred. Captured by invoking the DEPLOYED function directly with a minimal html-to-pdf payload: GPU process launch failed: error_code=1002 FATAL: GPU process isn't usable. Goodbye. ptrace: Operation not permitted (1) Zygote could not fork: process_type utility numfds 6 child_pid -1 write: Broken pipe (32) Three subprocess spawns Lambda's sandbox refuses, so each new flag answers one line rather than being copied from a blog post: --in-process-gpu `--disable-gpu` was already there and is not enough: it disables GPU RENDERING, the GPU PROCESS is still spawned, and Chromium treats its failure as FATAL. --disable-software-rasterizer no SwiftShader fallback process either. --no-zygote the zygote cannot fork here. --single-process belt to those braces. print-to-PDF is a one-shot non-interactive load, the workload single-process is safe for. --disable-crash-reporter crashpad's ptrace attach is denied and the only thing it contributed was noise in the error the user ended up reading. --user-data-dir / --crash-dumps-dir in scratch, plus HOME and XDG_* pointed at /tmp: the filesystem is read-only everywhere else and Chromium's default profile path is $HOME/.config/chromium. Held as a frozen CHROMIUM_SANDBOX_FLAGS constant with a test per flag naming the stderr line it answers, because dropping any one of them fails at runtime, in production, silently. THE FUNCTION LOGGED NOTHING Its whole CloudWatch group held START/END/REPORT and not one application line. The diagnostic went to the caller, so the only record of why production was broken was a screenshot of a chat message. runBinary and the top-level catch now console.error before returning. WHY THE GATES PASSED A BUILD THAT COULD NOT PRINT The Dockerfile probe runs during `docker build`; the smoke test runs the RIE under Docker Desktop. Both are ordinary containers. Real Lambda is a locked-down microVM, and this does NOT reproduce locally — verified: with `--cap-drop=ALL --security-opt no-new-privileges` Docker still writes a correct PDF. So a local pass proves OUTPUT QUALITY and nothing about whether the binary can start. The test-file header and README used to claim otherwise. Both now say what local runs can and cannot prove, and README gains a Post-deploy check — an `aws lambda invoke` one-liner against the deployed function, required after any change to the Dockerfile, the flags, or the base image. It is the only test that runs inside the sandbox that broke this. VERIFIED Local runs prove the half they can: the new flag set renders nashs's actual sign (flexbox, @page, embedded base64 image, serif stack) to a 20924-byte PDF at MediaBox 612x792pt — byte-identical in size to the pre-change output, so --single-process costs nothing here. Rendered to PNG and looked at it: gradient, space-between layout, embedded image, font all intact. Known trade-off: under --single-process Chromium cannot resolve `@font-face { src: local(...) }` (it needs the browser process's font service) and logs a warning. Generated artifacts embed or link their fonts, and a missing local() face is a smaller loss than no PDF at all. Gate: lint clean (zero warnings), typecheck clean, 5918 jest, 30 agent-media bun tests (+9 new), cdk synth green — only the pre-existing LOW tag-condition warnings. No agent-image change: the media Lambda is a CDK DockerImageAsset, built and pushed by `cdk deploy`, so this ships with the AgentPlatform deploy alone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes item 7 of the week's agent-failure work: the three capabilities the agent kept having to refuse.
The refusals this ends
Pulled from prod
agent_failuresrather than summarised, because the detail changed the design:OCR — the fourth ask — shipped in #1736 and is already live.
Design
One new container Lambda (
infra/agent-media/) with three operations, behind the existing root-owned loopback relay.Separate from
hyperframes-render, deliberately, despite the overlapping Chromium+FFmpeg toolset. The reason is IAM, not convenience: that function's role may write only topublic-images/. This one reads and writes owners' private workspace prefixes and carries an explicit DENY onpublic-images/, so "a scanned IEP never becomes a public-by-link object" is an IAM property rather than a code convention that lasts only as long as nobody edits the handler. Merging them would widen a production role to save an image.The function never accepts an identity. The relay injects
workspacePrefixfrom the web-verified invocation context; the model supplies only workspace-relative paths, validated against absolute forms,.., backslashes and control characters at two independent layers. A caller cannot reach another owner's workspace because it never states who it is. Identity fields in the payload are rejected, not silently overwritten — an attempt to reach elsewhere should be visible, not quietly corrected._resolve_invocation_identityhas no old-tier fallback, unlike its owner-only sibling. The prefix decides whose objects the function may touch, so if the web tier is too old to return it, media stays unavailable until the deploy catches up.Transport: bytes go through the workspace broker into
.media-scratch/, newly added tocheckpointExclusions. That is load-bearing. An ad-hoc upload into a checkpoint-managed path is deleted by the nextensureWorkspaceCheckpoint, which rolls the workspace back to its manifest and delete-markers anything unexpected — scratch would vanish mid-turn. Exclusion also keeps it out of the container, which is correct for a transport. Scratch is removed on every exit path; forpsd-transcribethat matters beyond tidiness, because the transport copy is someone's voice.Presets, never caller-supplied ffmpeg argv. Arbitrary arguments into ffmpeg are a command-injection and resource-exhaustion surface, and every real request so far has been one of three shapes. A test asserts no preset argument contains a shell metacharacter.
Verified against the running container, not just unit tests
html-to-pdf — reproduced nashs's sign (flexbox,
@page, embedded base64 image, web font stack) through the Lambda Runtime Interface Emulator:612×792pt is exactly 8.5×11. Rendered it to PNG and looked at it: full-bleed gradient,
space-betweenlayout, the embedded image, the serif font — every element the old fallback destroyed.social-mp4 — ProRes
.mov→h264 High / yuv420p+aac, 3.5 MB → 54 KB. Checked the+faststartclaim empirically rather than trusting the flag:moovat byte 36,mdatat 3591. That ordering is what stops uploaders rejecting a file as corrupt.Error paths through the real container — an
.aiffrefusal that hands over the exact conversion command, and../escape.movstopped at the traversal check.The relay's path regex — proven against 13 escape attempts before being trusted.
Re-verified after refactoring. I split three over-complex functions to clear the zero-warning lint gate, then rebuilt the image and re-ran the smoke test, because the earlier pass no longer described the code. Byte-identical result.
The Dockerfile also fails the build if Chromium cannot emit a
%PDF-or the static ffmpeg cannot execute, so a broken image never reaches a push.Things worth a reviewer's attention
WORKSPACE_BUCKET is not configured— the wrong error for the wrong reason. Everything free to check now runs before any I/O.security/detect-non-literal-fs-filenamewarnings were pointing at something real: I had imported the sharedvalidated-fswrapper and then used rawfsanyway. All filesystem access moved onto the wrapper, matching every other handler and skill.cdk synthsucceeds. The two newAgentMediaIAM warnings are the same LOW tag-condition class the existingHyperframesRenderrole already carries, for the documented reason that an object-levels3:PutObjectpresents no resource tags at authorization time.Deploy order
Same web-tier-first rule as #1736, and here it is enforced rather than merely documented: the relay fails closed if
/api/agent/invocation-identitydoes not yet returnworkspacePrefix.AIStudio-AgentPlatformStackwith the new agent image and the media LambdaThe three evals are L2/capability, matching
psd-hyperframes— they exercise a deployed function, so they skip until it exists rather than failing red.Verification
bun run lintclean (zero warnings) ·bun run typecheckclean · 5,918 jest · 904 agent-image Python (+9 new relay boundary tests) · 21 handler tests ·cdk synthgreen.Also wired agent-media into CI with its own dependency install — the same gap that left the agent-cron suite running nowhere until CI caught it in #1736.