Skip to content

Take composite OIDs from the caller instead of guessing 16385 - #102

Merged
adriangb merged 4 commits into
mainfrom
claude/composite-oids
Aug 15, 2026
Merged

Take composite OIDs from the caller instead of guessing 16385#102
adriangb merged 4 commits into
mainfrom
claude/composite-oids

Conversation

@adriangb

Copy link
Copy Markdown
Owner

Closes #96 (item 1; item 2, Json.oid(), is #100). Breaking, deliberately batched into the
pending semver-major.

The bug

PostgresType::UserDefined::oid() returned a hard-coded 16385. Postgres tolerated that only
because it assumes an OID it does not recognise belongs to a sender-side type — and 16385 is
the first OID a cluster hands out for user objects
. On any database that has ever created a
user-defined type, record_recv finds a real (and different) type at 16385 and rejects the COPY.
Fresh embedded test clusters never have one, which is why the suite never saw it.

pgpq cannot know a composite's OID: the server allocates it when the type is created. So the
caller supplies it.

API

let encoder = ArrowToPostgresBinaryEncoder::try_new(&schema)?
    .with_composite_oids(&HashMap::from([("s_t".into(), oid_from_pg_type)]))?;
  • PostgresType::UserDefined carries oid: Option<u32>.
  • StructEncoderBuilder::with_oid for one, ArrowToPostgresBinaryEncoder::with_composite_oids
    for a whole map — keyed by the type name the generated DDL uses (<field>_t), which is what
    you look up in pg_type after running the DDL. Unknown names are reported, so a typo fails
    loudly instead of silently leaving the wrong OID in place.
  • Exposed to Python as ArrowToPostgresBinaryEncoder.with_composite_oids(mapping), with a stub
    entry.

Behaviour change

Encoding a nested composite without its OID is now an error that names the type and the query
that finds it:

field s is a composite type whose OID in the target database is unknown; supply it with
`with_composite_oids` (look it up with `select oid from pg_type where typname = 's_t'`)

Building the encoder tree still succeeds — the OID arrives afterwards — so only shapes that can
never carry an OID (arrays of composites, arrays of arrays) still fail at construction, as
before. This is the one judgement call worth flagging: it turns a silent wrong-OID-on-the-wire
into an explicit error, at the cost of requiring nested-struct callers to do a lookup they
previously got away without on a fresh database.

Tests

  • The roundtrip harness now creates the types, asks pg_type for the OIDs it just allocated, and
    encodes with those — so the suite no longer depends on any composite landing on a particular
    OID.
  • nested_composite_uses_the_databases_own_oid creates unrelated types first to force the
    composite off 16385, asserts the real OID differs, asserts the encoded bytes carry it (and
    differ from the placeholder encoding), and roundtrips successfully.
  • Python: test_nested_struct_needs_composite_oids covers the raise, the success with the OID
    declared, and the unknown-name error.

The byte-exact snapshots still encode with 16385 — that is what produced them and what a fresh
cluster allocates — but that value now lives in the test harness where it is visible, rather than
in the library where it applied to every user's database.

`UserDefined.oid()` returned a hard-coded 16385. Postgres tolerated it only
because it assumes an OID it does not recognise belongs to a sender-side type —
and 16385 is the *first* OID a cluster hands out for user objects, so on any
database that has ever created one, `record_recv` found a real and different
type there and rejected the COPY. Fresh test clusters never had one, which is
why nothing caught it.

pgpq cannot know a composite's OID: the server allocates it when the type is
created. So the caller supplies it.

* `PostgresType::UserDefined` carries `oid: Option<u32>`.
* `StructEncoderBuilder::with_oid`, and
  `ArrowToPostgresBinaryEncoder::with_composite_oids` for a whole map at once,
  keyed by the type name the generated DDL uses (`<field>_t`). Unknown names are
  reported rather than ignored, so a typo fails instead of silently leaving the
  wrong OID. Exposed to Python as `with_composite_oids` too.
* Encoding a nested composite without its OID is now an error naming the type and
  the query that finds it, rather than putting a guess on the wire. Building the
  tree still works — the OID is supplied afterwards — so only shapes that can
  never carry an OID (arrays of composites or of arrays) still fail at
  construction.

The roundtrip harness now creates the types, asks `pg_type` for the OIDs it just
allocated and encodes with those, so the suite no longer depends on a composite
landing on any particular OID. `nested_composite_uses_the_databases_own_oid`
creates unrelated types first to force the composite elsewhere, and asserts the
encoded bytes follow the server.

The byte-exact snapshots still encode with 16385 — that is what produced them and
what a fresh cluster allocates — but the value now lives in the test harness where
it is visible, not in the library where it applied to everyone's database.

BREAKING: nested composites require their OID; `PostgresType::UserDefined` gained
a field.
@adriangb
adriangb enabled auto-merge (squash) August 15, 2026 04:41
CI caught `test_roundtrip_structs`: the Python roundtrip encoded before running
the DDL, so a nested struct had no OID to encode with. Same ordering problem the
Rust harness had, and the same fix — create the types, ask the server, encode.

Doing that from Python needed a way to know *which* type names to look up, and
the Python `UserDefined` exposes no field traversal. Rather than make every
caller write one, the encoder now answers the question directly:

    for name in encoder.composite_type_names():
        oids[name] = look_up(name)
    encoder.with_composite_oids(oids)

`composite_type_names()` is on the Rust encoder too, and both test harnesses now
use it instead of the two hand-written schema walks this branch had grown.

Verified with a real server this time: `PATH=~/.theseus/postgresql/*/bin` (the
trick DEVELOPMENT.md already documents) runs the 9 Postgres-backed Python tests
that are otherwise skipped locally. All 25 pass.
# Conflicts:
#	core/tests/integration_tests.rs
@adriangb
adriangb merged commit 1b5eb30 into main Aug 15, 2026
27 checks passed
@adriangb
adriangb deleted the claude/composite-oids branch August 15, 2026 05:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Composite/array OID correctness: dummy 16385 collides with real user OIDs; Json.oid() returns jsonb's 3802

1 participant