Protocol pipelining via client-allocated object refs (POC: ~17-20% faster synth)

[mrgrain]

Context

Large CDK (Python) synths spend most of their time in the synchronous host↔kernel protocol: a strict request/response ping-pong where the host blocks on every create/invoke/get. Profiling (see #5166) showed each side idle ~half the time waiting for the other, while JSON serialization is only ~2% — so the lever is overlapping round-trips, not the wire format.

This issue tracks a working proof-of-concept of protocol pipelining via client-allocated object references, and the work needed to productionize it. Draft PR: (link below).

Prototyped by Kiro (an AI agent), working with @mrgrain.

Mechanism

• Any call whose declared return is a reference type (class/interface) is fired without waiting: the host mints the object id, sends it with the request, and uses a synthetic handle immediately.
• The kernel honors a client-supplied objid on create, and on invoke/sinvoke/get/sget aliases that id to whatever object the call actually produced — fresh or pre-existing. Multiple ids → one object is fine; kernel-side identity stays correct.
• The host drains acks lazily and only blocks at true sync points: value returns (e.g. reading a token string), callbacks, end of synth. A pending cap bounds outstanding requests to avoid pipe-buffer deadlock.
• ref-vs-value is decided from the generated binding's declared return type.

Results (POC)

~2,200-resource synth, best-of-5, same runtime, byte-identical CloudFormation output:

variant	wall
baseline	~5.8–6.1 s
pipelined (create+invoke+sinvoke+sget)	~4.7–4.9 s

~17–20% faster. Findings:

• create-only pipelining ≈ 0% — interleaved synchronous invokes are barriers that drain the pipeline. Pipelining invokes too is what unlocks the win.
• get/sget added little here — instance gets in this app are all value reads (bucket_name, queue_url, table_name → token strings), which are irreducible sync points; only static sgets returned references.
• The remaining gap to the theoretical ~2× ceiling is the kernel's serial work (construct JS + dispatch), not the protocol. Pipelining overlaps the host with the kernel; once overlapped, you're kernel-work-bound.

Remaining TODOs / gaps

Productionizing this is a real cross-cutting project:

• Codegen instead of runtime introspection. The POC reads the caller's return annotation at runtime (frame walk + get_type_hints, cached). Production should have pacmak emit the return fqn into the binding call (jsii.invoke(self, "m", args, return_fqn=...)) — faster, deterministic, and removes the frame-walk hack. Needed per target language.
• Deferred error semantics. A failed pipelined call surfaces at the next sync point, not the call site. Requests need tagging so errors map back to the originating call with a usable stack. The POC ignores errors on the drained path.
• Callbacks. Overrides (host ← kernel callbacks) are hard sync points; the pipeline must drain/coordinate around them. The POC asserts/raises if a callback appears mid-pipeline (the test app has none).
• Object identity. Client-minted ids mean two reads of the same underlying object yield two distinct host proxies (a.prop is a.prop can become False). Kernel-side identity is preserved; host-side needs a story (e.g. cache ref→proxy, or reconcile at sync points). Did not affect output here, but identity-sensitive code could differ.
• Id-space / collision discipline. The kernel must accept and trust client ids in a defined, non-colliding range (POC uses fqn@<1e9+seq>); formalize and validate.
• Capability negotiation. Gate on a handshake flag so old/new hosts and kernels interoperate; never send objid to a kernel that doesn't understand it.
• Pending-cap tuning / backpressure. POC uses a fixed cap of 64; needs principled backpressure to avoid pipe-buffer deadlock across platforms.
• All language runtimes. POC is Python-only; the kernel side is language-agnostic, but Java/.NET/Go hosts need the same client work.
• Async (begin/end) and del interactions with the pipeline are unexplored (absent in the test app).
• Tests for the kernel alias behavior and the host pipeline/drain/correctness; broader app coverage beyond the synthetic stack.

Related

• Synth perf: wire protocol is a synchronous ping-pong (~2x headroom); JSON is not the bottleneck — pipeline via client-allocated object refs #5166 — wire-protocol profiling and analysis (why JSON isn't the bottleneck; the ~2× ceiling).

[mrgrain]

Capability negotiation. Gate on a handshake flag so old/new hosts and kernels interoperate; never send objid to a kernel that doesn't understand it.

• The kernel should send an initial message allocating the runtime client a namespace.
• Runtime will only allocate IDs if it has received the allocation message, otherwise behave as before
• Verify: If a runtime that doesn't support this will ignore the message.
• Otherwise: Runtime should initiate by asking the kernel (again, verify that an older kernel doesn't throw here)
• Note: Runtime and Kernel are always bundled, so this might not be an issue even.

[mrgrain]

Pending-cap tuning / backpressure. POC uses a fixed cap of 64; needs principled backpressure to avoid pipe-buffer deadlock across platforms

Test if a higher value performes better.

[mrgrain]

Object identity. Client-minted ids mean two reads of the same underlying object yield two distinct host proxies (a.prop is a.prop can become False). Kernel-side identity is preserved; host-side needs a story (e.g. cache ref→proxy, or reconcile at sync points). Did not affect output here, but identity-sensitive code could differ.

Can we make object identity a jsii operation as well? that way the kernel can decide.

[mrgrain]

All language runtimes. POC is Python-only; the kernel side is language-agnostic, but Java/.NET/Go hosts need the same client work.

Python only for now.