fix(extract-code-examples): three Python parser fixes (streaming wrapper, path-param wrapper, request body extraction) - #5
Merged
Conversation
… streaming SDK calls Fern's Python generator wraps streaming endpoint tests in `for _ in client.foo(...): pass`. The regex captured the call expression but also the trailing `:` from the `for` header, producing syntactically invalid Python in `sdkCallSource`. Adds `truncateAfterMatchingParen` to slice the captured string at the close paren matching its first open paren, dropping any trailing punctuation from the compound-statement wrapper. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…sonable_encoder/url_encode
Fern's current Python generator emits `encode_path_param(name)` inside path
f-strings. The hardcoded wrapper list missed it, so manifest keys leaked the
helper call (e.g. `GET /agent/{encode_path_param(id)}`) — breaking any
downstream lookup by canonical OpenAPI path. Replaces the two specific
replaces with one generic `\{\w+\((\w+)\)\}` that matches any future Fern
wrapper without further changes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
… Python SDKs
Python wire tests pass body data as kwargs (e.g. `client.foo.create(name="x", ...)`)
rather than as an explicit body literal — there's no `rawRequestBody`
variable like in the TS parser. The Python path through the extractor was
hard-coding `requestBody: null` and `sdkCallArgs: []`, so every Python
example in the manifest reported "no request data" regardless of method.
Adds a hand-rolled Python kwarg parser (`pyParseKwargs` + tokenizing
helpers) that walks the captured `sdkCallSource`, splits at top-level
commas, and emits ordered `{name, value}` pairs. Values are JSON-parsed
after translating Python's True/False/None; non-literal values fall back
to `<expr:...>` to match the TS escape hatch.
In `buildManifest`, when `example.requestBody` is null and the method is
POST/PUT/PATCH, derive the body from the kwargs minus any whose name
appears as a `{name}` slot in the endpoint's path template. Other
languages keep their already-populated body via the `??` short-circuit.
TS audit: the TS parser was already extracting `rawRequestBody` and call
args correctly — added a regression test against the existing fixture to
lock that behavior in.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…m restating comments Three Python char-walking helpers shared the same string-literal + bracket-depth preamble. Extract it into a `pyWalkTopLevel` generator and have `pyExtractArgsPortion`, `pySplitTopLevel`, and `pyFindTopLevelEquals` each become a short consumer of the iterator. Centralizes the string- handling logic and removes a class of "fix one walker but forget the others" risk. Also trims doc comments on `pyParseKwargs` and `deriveBodyFromKwargs` to drop signature-restating WHAT paragraphs, leaving only the load-bearing WHY (Python AST absence, query/header param leakage caveat). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…n Python kwarg values Python allows — and Black's multi-line formatter emits — trailing commas in lists/dicts (e.g. `items=["a", "b",]`). JSON.parse rejects these, so without stripping them every multi-line kwarg value fell through to the `<expr:...>` escape hatch, making `request.body` wrong for almost every real Fern-generated Python wire test. Adds a string-literal-aware `stripTrailingCommas` pass between `pyToJsonLiteral` and `JSON.parse`. Commas inside quoted strings (e.g. `label="a,]b"`) are preserved. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…erived bodies
`deriveBodyFromKwargs` was using path-param exclusion only, so any kwarg
that wasn't a path param ended up in `request.body` — including header
kwargs like `phenoml_on_behalf_of` and query kwargs.
Extract the precise set of body-field kwargs from the generated raw
client's `json={...}` dict literal (`pyExtractBodyParamNames`) and store
it on `EndpointMapping.bodyParamNames`. The body derivation now uses this
as a positive allowlist when present, so kwargs that go to `headers={...}`
or `params={...}` in the raw client are correctly excluded.
Falls back to the old path-param-exclusion heuristic when the raw client
doesn't use a dict literal for `json=` (rare); also keeps that path live
for TS/Java parsers that already extract body from test literals.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…not SDK kwarg names
Fern can alias the wire-side JSON field name to differ from the local
kwarg name (e.g., `json={"someField": some_field}`). The previous fix
keyed derived bodies by the kwarg name, so for aliased fields the
manifest's request.body didn't match what the SDK actually sends over
HTTP.
Replace `bodyParamNames: string[]` with `bodyParamMap: Record<string, string>`
that maps kwarg name → JSON field name. `pyExtractBodyParamMap` now
captures BOTH sides of each `"jsonKey": kwarg` entry. `deriveBodyFromKwargs`
looks up each kwarg in the map and emits the JSON-side key.
The non-aliased common case (`"name": name`) is unchanged because both
sides match. Fallback path (no bodyParamMap) still uses the kwarg name
as the body key — best-effort when the raw client doesn't use a dict
literal for `json=`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
gavinsharp
force-pushed
the
gavinsharp/fix-python-code-extraction
branch
from
May 12, 2026 00:14
44e9ee3 to
754d17b
Compare
… extracting body kwargs Current Fern Python output wraps body field values with helpers like `convert_and_respect_annotation_metadata(object_=conversation_config, ...)` to attach serialization metadata. The map-extraction regex captured the wrapper function name (`convert_and_respect_annotation_metadata`) as the kwarg, which never matches the real SDK kwarg, so the field disappeared from the derived body. Add `pyUnwrapBodyValue` that handles three shapes: - bare identifier (existing behavior) - positional wrapper like `jsonable_encoder(x)` (used in older Fern paths) - `object_=<kwarg>` keyed wrapper (Fern's current serialization helper) Drops the entry rather than guessing when the wrapper has no recoverable inner kwarg (e.g., only `name=value` style args) — better than emitting a body keyed by a function name that nothing matches. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…e tests target the same endpoint
When more than one wire test mapped to the same endpoint, buildManifest
unconditionally overwrote the existing manifest entry. With the current
fixtures that meant test_long_body.py (no kwargs) erased the derived
body that test_authtoken_auth.py (with kwargs) produced for the same
endpoint — and the integration test that should have caught this was
filtering out the poor example, hiding the bug from CI.
Score each candidate by how many of {body, sdkCallArgs, responseBody,
sdkCallSource} carry data, and only overwrite when the new entry has a
strictly higher score. Ties keep the first writer (insertion-order
stable).
Drop the workaround filter from the existing integration test so it now
exercises the real production code path, and add focused order-independence
tests covering both insertion orders.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three bug fixes to the Python wire-test parser in
extract-code-examples, each surfaced when running the action againstphenoml-python-sdk(PR #145):sdkCallSource— Fern wraps streaming tests infor _ in client.foo(...): pass, and the captured snippet included the trailing:from theforheader. Added atruncateAfterMatchingParenhelper that slices the captured call at the close-paren matching the first open-paren.encode_path_param(id)leaked into manifest keys (GET /agent/{encode_path_param(id)}) because the hardcoded unwrap list only knew about older Fern helpers (jsonable_encoder,url_encode). Replaced with a generic\{\w+\((\w+)\)\}regex that covers any current or future Fern wrapper.request.bodyandsdkCallArgswere always empty for Python SDKs. The Python parser hard-coded both to null/[] and never inspected the SDK call's kwargs. Added a hand-rolled kwarg parser (no Python AST in Node) that walks the capturedsdkCallSource, splits at top-level commas, and emits{name, value}pairs.buildManifestthen derivesrequest.bodyfrom those kwargs minus path-param names for POST/PUT/PATCH endpoints. TS path was already correct; added a regression test to lock that in.Test plan
bun test— 60 tests pass (45 existing + 15 new across the three fixes)code-examples.jsonmanifest now has populated bodies, noencode_path_paramleakage, and valid Python in streamingsdkCallSource🤖 Generated with Claude Code
Note
Medium Risk
Medium risk: changes parsing heuristics and manifest generation for Python SDKs, which could alter extracted endpoints/bodies if edge-case Fern output isn’t handled as expected.
Overview
Fixes Python code-example extraction so manifests include accurate request bodies and valid SDK call sources.
The Python parser now (1) strips any single-function wrapper around f-string path params (e.g.
encode_path_param(...)), (2) truncates streamingfor _ in client.foo(...):captures to the matching)to avoid invalidsdkCallSource, and (3) parses SDK-call kwargs into structuredsdkCallArgs.Manifest generation is updated to derive
request.bodyforPOST/PUT/PATCHfrom kwargs, using a newbodyParamMapextracted from raw-clientjson={...}literals to both filter out non-body kwargs (headers/query/path) and emit aliased wire field names; when multiple tests hit the same endpoint, a richness score now prefers the most complete example. Tests/fixtures were expanded to cover these cases.Reviewed by Cursor Bugbot for commit bdbe6fc. Bugbot is set up for automated code reviews on this repo. Configure here.