Skip to content

ci: weekly compatibility canary for all language connectors#616

Open
amaksimo wants to merge 3 commits into
mainfrom
compat-canary
Open

ci: weekly compatibility canary for all language connectors#616
amaksimo wants to merge 3 commits into
mainfrom
compat-canary

Conversation

@amaksimo

Copy link
Copy Markdown
Contributor

Summary

Mirror of awslabs/aurora-dsql-orms#499 — same early-warning system, applied to every language connector. Weekly recompile + unit tests against the latest upstream Postgres driver / SDK (instead of the pinned version). A failure auto-files a single dedup'd compat-canary tracking issue.

The motivating bug: a customer hit NoClassDefFoundError: PgJdbcHelper when Spring Boot 4 pulled Hibernate 7 into our dialect. That class of break — linkage / API relocation — would have been caught by simply recompiling against latest. This canary does that, weekly, with no live cluster needed.

Coverage

8 connectors, all with cluster-free unit suites (verified by reading each *-ci.yml):

Connector Probe
Python (psycopg / psycopg2 / asyncpg) pytest tests/unit after uv pip install --upgrade of all three
Java JDBC ./gradlew test -PpgjdbcVersion=+
Go pgx go test ./dsql/... after go get -u jackc/pgx aws-sdk-go-v2/...
node-postgres npm run test:unit after npm install pg@latest --save-peer
postgres-js npm run test:unit after npm install postgres@latest --save-peer
.NET Npgsql dotnet test after dotnet add package Npgsql --prerelease
Rust sqlx cargo test --lib after cargo add sqlx@latest
Ruby pg bundle exec rake unit via overlay Gemfile that relaxes ~> 1.5 to track latest

PHP pdo_pgsql intentionally omitted — it's a built-in PHP extension, not an upgradable package, so there's no upstream to canary at the connector layer.

Mechanics

  • .github/workflows/compat-canary.yml — weekly (Mon 06:00 UTC) + manual. Independent jobs per connector; never gates PRs/releases. Includes a dry-run workflow_dispatch input that skips issue filing — use it for the first manual run on main to validate wiring without spamming issues.
  • .github/actions/report-canary-failure/ — composite action; opens or comments on a single dedup'd compat-canary issue per connector.
  • build.gradle.kts — pgjdbc version is now overridable via -PpgjdbcVersion=… (default 42.7.11 unchanged). Verified locally: override flows through, default falls back, compileJava is clean. No other manifest edits needed — Python / Node / Go / Rust / .NET / Ruby all support install-time version overrides.

Honest caveats (unchanged from the ORMs PR)

  • YAML files parse, but real GitHub-Actions schema validation only happens once the workflow exists on main (workflow_dispatch 404s on PR branches).
  • The Gradle override is the only logic I tested locally; the cargo add sqlx@latest, dotnet add package --prerelease, Ruby Gemfile overlay, etc. are best-reading-of-docs, not observed runs.
  • Recommended first run after merge: gh workflow run compat-canary.yml -f dry-run=true. Reveals broken probes without filing noise.

Why no dependabot edit (unlike the ORMs PR)

The ORMs repo had stale >=7 / >=4 ignores muting the exact signal this canary now provides; that needed fixing. The connectors repo's dependabot.yml has no upstream-version ignores — only build-tool pins (junit, jgit) — so there's nothing to roll forward.

amaksimo added 3 commits June 23, 2026 16:12
Mirrors the same early-warning system added to aurora-dsql-orms: weekly
recompile + unit tests against the LATEST upstream version of each connector's
Postgres driver / SDK. A failure auto-files a dedup'd tracking issue.

Coverage (8 connectors, all with real cluster-free unit suites):
- Python (psycopg/psycopg2/asyncpg latest, tests/unit)
- Java JDBC (pgjdbc latest, ./gradlew test)
- Go pgx (pgx + aws-sdk-go-v2 latest, go test ./dsql/...)
- node-postgres (pg latest, npm run test:unit)
- postgres-js (postgres latest, npm run test:unit)
- .NET Npgsql (Npgsql latest, dotnet test)
- Rust sqlx (sqlx latest, cargo test --lib)
- Ruby pg (pg gem latest, rake unit)

PHP pdo_pgsql intentionally omitted — it is a built-in PHP extension, not an
upgradable package, so there is no upstream to canary at the connector layer.

build.gradle.kts: pgjdbc version is now overridable via -PpgjdbcVersion (used
by the canary to probe latest); default 42.7.11 unchanged.

The workflow includes a workflow_dispatch `dry-run` boolean for safe manual
smoke runs that skip filing tracking issues — used to validate wiring on the
first manual run after merge.
Self-review on PR #616 (16-agent fan-out, ≥60 confidence threshold).
Sourced commit. 19 high-confidence findings; 17 fixed in code, 2
deferred to PR-body edit.

Bug fixes:
- Rust sqlx: `cargo add sqlx@latest` is invalid (`latest` is not a
  cargo version requirement). Use bare `cargo add sqlx` which already
  takes the latest, including across majors.

Hardening:
- SHA-pin every action to the same SHAs the rest of the repo uses,
  matching the repo's existing supply-chain convention.
- Add `persist-credentials: false` to every `actions/checkout` step.
- Add `concurrency: { group: compat-canary }` so an overlapping
  schedule + workflow_dispatch can't race the dedup search.
- Probe label existence before `gh label create` instead of swallowing
  every error class with `2>/dev/null || true`.
- Pass title via `--arg` to jq to remove quote-injection surface.
- Write to `$GITHUB_STEP_SUMMARY` so reporter self-failure is
  distinguishable from a normal canary-fired run.

Comment / accuracy fixes:
- Issue body said "the canary will reopen it if the break recurs" but
  dedup uses `--state open` only; reword to match behavior.
- build.gradle.kts said "probes the next major" but `+` resolves to
  absolute latest (any major); reword.
- dry-run input description didn't say the run still goes red on
  probe failure; clarify.
- Document that tier-1 doesn't catch runtime DSQL semantic regressions.
- Document the deliberate `@latest`-from-public-registry supply-chain
  surface and the `issues: write`-only mitigation.
- Document why `--no-sync`, why unversioned `gem "pg"` overrides the
  gemspec ceiling, and why `go get -u` won't cross majors.

Style:
- `findProperty(...) as String?` → `findProperty(...)?.toString()` —
  defensive Kotlin DSL idiom; survives non-String property values.
- Drop redundant `working-directory` on `ruby/setup-ruby` step.
Iteration-2 self-review caught a real regression introduced by iter-1:

`gh issue list --arg` is not a recognized flag (`gh` does not forward
unknown args to its embedded `--jq` engine). Under `set -euo pipefail`,
the dedup probe crashed before any issue-filing logic ran — meaning the
entire user-visible output of the canary (dedup'd tracking issues) would
never have fired on a real break. Verified empirically with gh 2.83.1:
`unknown flag: --arg`.

Fix: pipe `gh issue list --json` to standalone `jq -r --arg t "${TITLE}" ...`,
which preserves the quote-injection guard while actually running.

Also caught:
- Rust step name still said "including next major"; iter-1 reworded the
  inline comment but missed the user-visible step name. Renamed to
  "absolute latest (including new majors)".
- Rust job used inline `working-directory:` on every step instead of
  `defaults.run.working-directory: rust/sqlx` like every other
  multi-step job. Restored the defaults block.
@amaksimo

Copy link
Copy Markdown
Contributor Author

Self-review (3 iterations, 16+ sub-agents per pass)

Procedure: 19 sub-agent fan-out per iteration; ≥60 confidence threshold; convergence validated by iteration 3 with zero new findings.

Final state: 3 commits on this branch — original 7ce24a4, iter-1 hardening 839f3b4, iter-2 regression-fix 7d48b8d.

Tracking table

# Conf Area Finding Disposition Commit
1 95 Bug cargo add sqlx@latest is not a valid cargo version requirement (verified: error: invalid version requirement \latest``); the Rust canary would have failed before any test ran. fixed: dropped @latest — bare cargo add sqlx already takes latest including across majors 839f3b4
2 90 Convention / supply-chain All 8 marketplace actions used floating tags; every other workflow in the repo SHA-pins. fixed: SHA-pinned to the same SHAs the rest of the repo uses (checkout v7.0.0, setup-uv v8.2.0, setup-go v6.4.0, setup-node v6.4.0, setup-java v5.3.0, setup-dotnet v5.3.0, ruby/setup-ruby v1.314.0, rust-toolchain stable) 839f3b4
3 80 Comment vs behavior Issue body said "the canary will reopen it if the break recurs", but dedup uses --state open only — closed issues never reopen. fixed: reworded to "a fresh issue is filed if the break recurs after closure"; added inline comment explaining open-only semantics 839f3b4
4 75 Comment accuracy build.gradle.kts said "probes the next major"; + resolves to absolute latest. Sibling orms#499 fixed the same defect. fixed: reworded to "absolute latest released version, including new majors" 839f3b4
5 75 Race No concurrency: group; overlapping schedule + workflow_dispatch could double-file. fixed: concurrency: { group: compat-canary, cancel-in-progress: false } 839f3b4
6 75 Hardening Dedup jq filter shell-interpolated ${TITLE} — quote/backslash injection if a future caller's connector name contains " or \. fixed: pass via --arg t "${TITLE}" (see also #22) 839f3b47d48b8d
7 70 Resilience gh label create ... 2>/dev/null || true swallowed every error class. fixed: probe via gh label list | grep -qx compat-canary first; only create when missing 839f3b4
8 70 Observability Self-failure of the report step looked identical to a normal canary-fired run on the Actions UI. fixed: write outcome to $GITHUB_STEP_SUMMARY 839f3b4
9 70 Hardening actions/checkout defaults to persist-credentials: true. fixed: with: persist-credentials: false on every checkout 839f3b4
10 65 Comment dry-run description didn't say the run still goes red. fixed: reworded 839f3b4
11 65 Comment The dedup --search + client-side select(.title == ...) looked redundant. fixed: comment explaining substring vs. exact-match roles 839f3b4
12 65 Comment Top-of-file comment didn't disclose what tier-1 cannot catch. fixed: added one sentence about runtime-semantic regressions being out of scope 839f3b4
13 65 Comment Ruby Gemfile heredoc didn't explain why unversioned gem "pg" overrides the gemspec ceiling. fixed: extended comment 839f3b4
14 65 Comment Python --no-sync is load-bearing (without it, uv run reverts the upgrade); was undocumented. fixed: one-line explanation 839f3b4
15 60 Comment / observability No supply-chain rationale in the workflow header. fixed: added a paragraph noting the deliberate @latest surface and the issues: write-only mitigation 839f3b4
16 60 Idiom findProperty(...) as String? throws on non-String values. fixed: replaced with ?.toString() ?: … 839f3b4
17 60 Nit Redundant working-directory: on ruby/setup-ruby step. fixed: removed 839f3b4
22 95 Iter-1 regression gh issue list --arg is not a recognized flag (gh does not forward unknown flags to its embedded --jq); under set -e the dedup probe would have crashed before any issue-filing logic ran — meaning the entire user-visible output of the canary would never have fired on a real break. Verified empirically with gh 2.83.1: unknown flag: --arg. fixed: pipe to standalone jq -r --arg t "${TITLE}" ... which preserves the --arg injection guard while actually running 7d48b8d
23 70 Comment Rust step name still said "Upgrade sqlx to latest (including next major)"; iter-1 missed the user-visible step name. fixed: renamed to "absolute latest (including new majors)" 7d48b8d
24 65 Idiom Rust job used inline working-directory: on each step instead of the defaults.run.working-directory pattern used by every other multi-step job. fixed: restored defaults block 7d48b8d
18 65 PR body Probe table understates 4 jobs (node-postgres, postgres-js, .NET, Rust): omits npm run build, two dotnet add invocations, --features pool,occ for cargo test. deferred: PR-body-only edit, no source change. Recommend updating the Coverage table when convenient — current diff is functionally correct, just under-described. n/a
25 50 PR body Probe table Rust row says cargo add sqlx@latest; "Honest caveats" also references the literal cargo add sqlx@latest invocation that no longer exists. deferred: PR-body-only edit. Replace both occurrences with cargo add sqlx --features runtime-tokio,postgres,tls-rustls-ring. n/a
19 60 Title Title says "all language connectors"; PHP is carved out only in body. deferred: stylistic; under 70 chars matters. Author's call. n/a
20 30 Test No bats smoke test for the dedup bash. deferred: out of scope; would expand the PR meaningfully n/a
FP False positive Iter-1 sub-agent claimed actions/checkout@v7, setup-go@v6, setup-node@v6, setup-java@v5, setup-dotnet@v5, setup-uv@v8 don't exist. false-positive: stale training data. Verified empirically via GitHub releases API — every version was published before today. n/a
FP False positive Claim that npm install pg@latest --save-peer doesn't update node_modules when an older pg is locked. false-positive: empirically tested in /tmp project — npm replaces locked 8.10.0 with latest 8.22.0 after the command. n/a
FP False positive Claim that this PR should also patch dependabot.yml (sibling orms PR did). false-positive: PR body explicitly justifies — connectors dependabot.yml has no upstream-version ignores, only build-tool pins (junit, jgit), so there's nothing to roll forward. Verified. n/a
FP False positive Claim that npm install pg@latest excludes pre-releases. false-positive: by-design — canary is for stable releases. n/a

Iteration log

Iter High-conf findings Resolved this pass Remaining open
1 19 (incl. 2 PR-body) 17 source fixes (839f3b4); 2 deferred to PR-body edit 0
2 3 (1 regression from iter-1, 2 nits) 3 source fixes (7d48b8d); 1 PR-body item added (Rust row stale) 0
3 0 (validation pass) n/a 0

Convergence: Iteration 3 reported zero new high-confidence findings; no prior fix regressed. Procedure complete inside the 3-iteration cap. The four remaining items (#18, #25, #19 PR-body edits; #20 test gap) are explicitly deferred and listed for human judgment.

@anwesham-lab

Copy link
Copy Markdown
Member

LGTM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants