Skip to content

Document how to reproduce the "46,066/46,066" regression suite claim from a fresh clone #65

Description

@MauricioPerera

Summary

The README badge and text (README.md:9,32,56) advertise a 100% pass rate on "all 46,066 tests in Postgres' regression suite," linking to an external blog post for methodology. I could not find, anywhere in a fresh clone of this repo, the actual base regression test corpus (upstream PostgreSQL's src/test/regress/sql/*.sql and expected/*.out, ~200 files) that this claim would be run against.

What I checked

  • regress/overlay/sql/ contains only pgrust-specific overlay files (e.g. partition_aggregate.sql, partition_join.sql, partition_prune.sql) — these read as additions/modifications layered on top of a base suite, not the base suite itself.
  • fixtures/pg-regress/ exists and does contain a handful of files with upstream-style names (combocid.sql, guc.sql, transactions.sql) — but only 3 files total, nowhere near the ~200-file upstream corpus, and none partition-related.
  • No .gitmodules, no vendored PostgreSQL source tarball, and no fetch/download script (checked for references to postgresql.org, a Postgres source tarball, etc.) that would pull in the missing base suite at build or test time.
  • No CONTRIBUTING.md or docs under regress/ explaining how to obtain/place the base suite to reproduce the regression run.

Why this matters

This is one of the project's headline correctness claims (it's the first badge in the README), and it's reasonable for someone evaluating or contributing to the project to want to verify it themselves. As it stands, a fresh clone doesn't contain enough to run the full regression suite — only pgrust's own overlay/fixture additions are present, not the upstream base files they overlay onto.

I'm not claiming the underlying tests don't pass — the linked blog post and the overlay files suggest real, serious regression-testing work exists — just that the reproduction path isn't documented or vendored in this checkout, so the badge's number can't currently be independently verified by cloning the repo.

Suggested fix

Document (e.g. in README.md or a new regress/README.md) either:

  1. Where/how to obtain the upstream PostgreSQL 18.3 src/test/regress corpus and how it's expected to be laid out alongside regress/overlay/ to run the full suite, or
  2. If the base suite is fetched automatically by some tool/script not present in this repo, point to that tooling, or
  3. If the base suite is intentionally not redistributed (e.g. licensing/size reasons) but instructions exist to regenerate the exact 46,066-test run locally, link/include those instructions.

Environment / where found

Found via a static review of a fresh clone of this repo while researching the partitioning test coverage specifically (looking for partition.sql/partition_index.sql, the upstream core DDL/bound-validation regression tests) — I couldn't confirm those run anywhere in this checkout, which led to checking for the base suite more broadly. This is a documentation/reproducibility gap, not a claim that the underlying engine work is incorrect.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions