feat(catalog): add GCP Cloud KMS support for secret decryption by whites11 · Pull Request #4243 · PeerDB-io/peerdb

whites11 · 2026-04-26T10:33:37Z

Summary

Add PEERDB_KMS_PROVIDER env var to dispatch between AWS KMS and GCP Cloud KMS for catalog secret decryption
Rust (nexus/catalog): GCP Cloud KMS REST API via reqwest, using a Workload Identity token fetched from the GKE metadata server
Go (flow/internal/env.go): GCP Cloud KMS via the official cloud.google.com/go/kms/apiv1 client (uses Application Default Credentials, which resolves to Workload Identity on GKE)
Provider switch on both sides enumerates aws (and empty for back-compat), gcp, and returns an explicit error for unknown values; constants KmsProviderAWS / KmsProviderGCP introduced in Go to mirror the Rust pattern
Defaults to aws everywhere — no behavior change for existing AWS deployments

Test plan

End-to-end test on GCP GKE cluster with Workload Identity, decrypting a catalog secret encrypted with a GCP KMS key
Verify PEERDB_KMS_PROVIDER= (empty), unset, and PEERDB_KMS_PROVIDER=aws all still hit the AWS path
Verify PEERDB_KMS_PROVIDER=invalid returns the new error rather than silently falling through to AWS

🤖 Generated with Claude Code

github-actions · 2026-04-26T10:55:57Z

🔄 Flaky Test Detected

Analysis: All three failing jobs show flakiness indicators: one hit the 15-minute timeout (panic: test timed out after 15m0s), and two others failed with failed to teardown postgres schema during cleanup — not due to test assertion failures.
Confidence: 0.9

✅ Automatically retrying the workflow

View workflow run

github-actions · 2026-04-26T11:15:02Z

🔄 Flaky Test Detected

Analysis: Both failures share the same transient infrastructure error — FATAL: terminating connection due to administrator command (SQLSTATE 57P01) — meaning the catalog PostgreSQL connection was forcibly killed mid-test by an external event, not a code regression.
Confidence: 0.93

✅ Automatically retrying the workflow

View workflow run

codecov · 2026-04-27T15:21:34Z

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
2226	2	2224	203

View the top 2 failed test(s) by shortest run time

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuitePG_CH_Cluster

Stack Traces | 0.01s run time

=== RUN   TestPeerFlowE2ETestSuitePG_CH_Cluster
=== PAUSE TestPeerFlowE2ETestSuitePG_CH_Cluster
=== CONT  TestPeerFlowE2ETestSuitePG_CH_Cluster
--- FAIL: TestPeerFlowE2ETestSuitePG_CH_Cluster (0.01s)
2026/04/27 15:25:00 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/04/27 15:25:00 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuitePG_CH_Cluster/Test_Binary_Format_Raw

Stack Traces | 139s run time

=== RUN   TestPeerFlowE2ETestSuitePG_CH_Cluster/Test_Binary_Format_Raw
=== PAUSE TestPeerFlowE2ETestSuitePG_CH_Cluster/Test_Binary_Format_Raw
=== CONT  TestPeerFlowE2ETestSuitePG_CH_Cluster/Test_Binary_Format_Raw
2026/04/27 15:22:40 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/04/27 15:22:40 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/04/27 15:22:41 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,c1,t FROM e2e_test_pgchcl_ed97mc0q.\"test_exclude_ch\" ORDER BY id"
2026/04/27 15:22:41 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,c1,t FROM e2e_test_pgchcl_ed97mc0q.\"test_exclude_ch\" ORDER BY id"
2026/04/27 15:22:41 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_11864272989103652787 CURSOR FOR SELECT id,c1,t FROM e2e_test_pgchcl_ed97mc0q.\"test_exclude_ch\" ORDER BY id" args=[]
2026/04/27 15:22:41 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,c1,t FROM e2e_test_pgchcl_ed97mc0q.\"test_exclude_ch\" ORDER BY id" channelLen=0
2026/04/27 15:22:41 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_11864272989103652787
2026/04/27 15:22:41 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_11864272989103652787 records=4 bytes=80 channelLen=3
2026/04/27 15:22:41 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,c1,t FROM e2e_test_pgchcl_ed97mc0q.\"test_exclude_ch\" ORDER BY id" rows=4 bytes=80 channelLen=3
2026/04/27 15:22:41 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_11864272989103652787
2026/04/27 15:22:41 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_11864272989103652787 records=0 bytes=0 channelLen=0
2026/04/27 15:22:41 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,c1,t FROM e2e_test_pgchcl_ed97mc0q.\"test_exclude_ch\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/04/27 15:22:41 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/04/27 15:22:41 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,c1,t FROM e2e_test_pgchcl_ed97mc0q.\"test_exclude_ch\" ORDER BY id" rows=4 bytes=80 channelLen=0
    clickhouse_test.go:1064: WaitFor waiting for CDC count 2026-04-27 15:22:44.820286601 +0000 UTC m=+472.997751827
    clickhouse_test.go:1068: WaitFor waiting for CDC count 2026-04-27 15:22:44.828812912 +0000 UTC m=+473.006278143
    clickhouse.go:76: 
        	Error Trace:	.../flow/e2e/congen.go:44
        	            				.../flow/e2e/clickhouse.go:76
        	            				.../flow/e2e/clickhouse.go:154
        	            				.../flow/e2e/test_utils.go:240
        	            				.../flow/e2e/test_utils.go:852
        	            				.../flow/e2e/test_utils.go:237
        	            				.../flow/e2e/clickhouse_test.go:1068
        	            				.../flow/e2e/clickhouse_test.go:1083
        	Error:      	Received unexpected error:
        	            	unable to establish connection with catalog: FATAL: terminating connection due to administrator command (SQLSTATE 57P01)
        	Test:       	TestPeerFlowE2ETestSuitePG_CH_Cluster/Test_Binary_Format_Raw
    clickhouse.go:107: begin tearing down postgres schema pgchcl_xibjps6w
2026/04/27 15:23:00 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_erh8jgyu.\"table1\" ORDER BY id"
2026/04/27 15:23:00 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_erh8jgyu.\"table1\" ORDER BY id"
2026/04/27 15:23:00 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_10620002973562392176 CURSOR FOR SELECT id,val FROM e2e_test_api_erh8jgyu.\"table1\" ORDER BY id" args=[]
2026/04/27 15:23:00 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_erh8jgyu.\"table1\" ORDER BY id" channelLen=0
2026/04/27 15:23:00 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_10620002973562392176
2026/04/27 15:23:00 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_10620002973562392176 records=2 bytes=19 channelLen=1
2026/04/27 15:23:00 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_erh8jgyu.\"table1\" ORDER BY id" rows=2 bytes=19 channelLen=1
2026/04/27 15:23:00 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_10620002973562392176
2026/04/27 15:23:00 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_10620002973562392176 records=0 bytes=0 channelLen=0
2026/04/27 15:23:00 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_erh8jgyu.\"table1\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/04/27 15:23:00 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/04/27 15:23:00 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_erh8jgyu.\"table1\" ORDER BY id" rows=2 bytes=19 channelLen=0
    clickhouse.go:107: 
        	Error Trace:	.../flow/e2e/pg.go:147
        	            				.../flow/e2e/clickhouse.go:107
        	            				.../flow/e2eshared/e2eshared.go:38
        	            				.../hostedtoolcache/go/1.26.2.../src/testing/testing.go:1317
        	            				.../hostedtoolcache/go/1.26.2.../src/testing/testing.go:1667
        	            				.../hostedtoolcache/go/1.26.2.../src/testing/testing.go:2030
        	            				.../hostedtoolcache/go/1.26.2.../src/runtime/panic.go:694
        	            				.../hostedtoolcache/go/1.26.2.../src/testing/testing.go:1022
        	            				.../flow/e2e/congen.go:44
        	            				.../flow/e2e/clickhouse.go:76
        	            				.../flow/e2e/clickhouse.go:154
        	            				.../flow/e2e/test_utils.go:240
        	            				.../flow/e2e/test_utils.go:852
        	            				.../flow/e2e/test_utils.go:237
        	            				.../flow/e2e/clickhouse_test.go:1068
        	            				.../flow/e2e/clickhouse_test.go:1083
        	            				.../hostedtoolcache/go/1.26.2.../src/reflect/value.go:586
        	            				.../hostedtoolcache/go/1.26.2.../src/reflect/value.go:369
        	            				.../flow/e2eshared/e2eshared.go:40
        	Error:      	failed to teardown postgres schema
        	Test:       	TestPeerFlowE2ETestSuitePG_CH_Cluster/Test_Binary_Format_Raw
        	Messages:   	pgchcl_xibjps6w: failed to drop replication slots: ERROR: replication slot "peerflow_slot_ch_binary_format_raw_pgchcl_xibjps6w" is active for PID 22166 (SQLSTATE 55006)
--- FAIL: TestPeerFlowE2ETestSuitePG_CH_Cluster/Test_Binary_Format_Raw (139.45s)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

github-actions · 2026-04-27T15:25:00Z

❌ Test Failure

Analysis: Deterministic compile error: internal/env.go:135 uses strings package which was not imported in the commit under test, causing the build to fail before any tests ran.
Confidence: 0.98

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-04-27T15:29:48Z

🔄 Flaky Test Detected

Analysis: The e2e test TestPeerFlowE2ETestSuitePG_CH_Cluster/Test_Binary_Format_Raw failed due to FATAL: terminating connection due to administrator command (SQLSTATE 57P01), a PostgreSQL admin-side connection kill that is unrelated to test logic and indicates transient CI infrastructure instability.
Confidence: 0.93

✅ Automatically retrying the workflow

View workflow run

github-actions · 2026-04-27T15:43:31Z

🔄 Flaky Test Detected

Analysis: Virtually all e2e tests timed out at ~31s across every suite (MySQL, PG, Mongo, PG_CH_Cluster), indicating the peer-flow worker became unresponsive during the run rather than any individual test logic regression.
Confidence: 0.93

✅ Automatically retrying the workflow

View workflow run

Add a PEERDB_KMS_PROVIDER env var ("aws" default, "gcp") to select the KMS backend for catalog credential decryption. When set to "gcp", uses the GCP metadata server for Workload Identity access tokens and calls the Cloud KMS REST API to decrypt. The key ID is the raw GCP resource name (projects/.../cryptoKeys/...). AWS behavior is unchanged and remains the default. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The Go flow worker, maintenance worker, and other flow components also decrypt catalog credentials via KMS. Add the same PEERDB_KMS_PROVIDER=gcp dispatch to the Go side, using the GCP metadata server for Workload Identity tokens and the Cloud KMS REST API for decryption. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Rust: remove needless Ok(x?) → x (clippy::needless_question_mark) - Go: use http.NoBody instead of nil request body (gocritic::httpNoBody) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Replace manual metadata token fetch + raw REST API calls with the official cloud.google.com/go/kms client library. The client handles authentication via Application Default Credentials (Workload Identity on GKE) automatically. This reduces ~50 lines of HTTP plumbing to ~15 lines of typed SDK calls and eliminates manual token management, JSON serialization, and error response parsing. Pinned cloud.google.com/go/kms at v1.24.0 to stay within the google.golang.org/api v0.257.0 constraint. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Add back strings import (still used by strings.TrimSpace) - Register cloud.google.com/go/kms v1.24.0 in check_pinned_versions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Per review on #4243 (comment) — define KmsProviderAWS/KmsProviderGCP constants, list all cases (aws + empty for back-compat, gcp), default branch returns an error matching the Rust side in nexus/catalog/src/lib.rs:43-47.

claude · 2026-05-07T11:55:56Z

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

masterashu

Looks good.
Any reason to use api calls to use http calls instead of google_cloud_kms library?

whites11 · 2026-05-07T12:55:18Z

Looks good. Any reason to use api calls to use http calls instead of google_cloud_kms library?

Let me give you claude's explanation, althought I have no rust experience to evaluate it

The reasoning is dependency-footprint pragmatism:                                                                                  
  
  1. The whole "gcp" path is feature-gated (gcp = ["dep:reqwest"]). The catalog crate compiles without any GCP deps when the feature 
  is off; turning it on adds only reqwest. A real Google Cloud SDK crate (e.g. google-cloud-kms/gcloud-sdk) would pull in tonic,
  prost, generated protobuf modules, h2, and a chain of transitive crates — even when feature-gated, just enabling it would balloon  
  compile time and binary size.                               
  2. The surface needed is tiny — two HTTP calls:
    - GET metadata.google.internal/.../token for a Workload Identity bearer token
    - POST cloudkms.googleapis.com/v1/{key}:decrypt                                                                                  
                                                                                                                                     
  That's ~30 LOC with reqwest. An SDK would be more code via builder objects with no functional gain.                                
  3. The Rust GCP-SDK ecosystem isn't as canonical as Go's. In Go there's exactly one obvious choice (cloud.google.com/go/kms/apiv1),
   and that's why the Go side of this same PR was refactored to it (commit acf7c23a). In Rust, the options (gcloud-sdk,              
  google-cloud-kms, google-cloud-rust) are competing third-party crates with different trade-offs — committing to one is a
  non-trivial choice.                                                                                                                
  4. Workload Identity on GKE is intentionally trivial over HTTP — the metadata server is designed to be hit with curl-equivalent
  calls. Using it that way is idiomatic, not a hack.                                                                                 
  
  Should it be changed? Probably not for this PR. If/when the catalog crate grows more GCP touchpoints (Pub/Sub source on the catalog
   side, etc.), promoting to a real SDK becomes worth the dep cost. For one decrypt call, raw reqwest is the right call.

whites11 self-assigned this Apr 26, 2026

whites11 force-pushed the customer-gcp-kms-support branch from 898c481 to 405dfc6 Compare May 5, 2026 11:46

whites11 and others added 5 commits May 7, 2026 13:32

fix(catalog,flow): address clippy and golangci-lint warnings

6bf16ae

- Rust: remove needless Ok(x?) → x (clippy::needless_question_mark) - Go: use http.NoBody instead of nil request body (gocritic::httpNoBody) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix(flow): restore strings import and register KMS pin

018b7e6

- Add back strings import (still used by strings.TrimSpace) - Register cloud.google.com/go/kms v1.24.0 in check_pinned_versions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

whites11 force-pushed the customer-gcp-kms-support branch from 405dfc6 to 018b7e6 Compare May 7, 2026 11:33

whites11 commented May 7, 2026

View reviewed changes

Comment thread flow/internal/env.go

whites11 marked this pull request as ready for review May 7, 2026 11:50

whites11 requested a review from a team as a code owner May 7, 2026 11:50

masterashu approved these changes May 7, 2026

View reviewed changes

whites11 merged commit f1c8c85 into main May 11, 2026
27 checks passed

whites11 deleted the customer-gcp-kms-support branch May 11, 2026 20:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(catalog): add GCP Cloud KMS support for secret decryption#4243

feat(catalog): add GCP Cloud KMS support for secret decryption#4243
whites11 merged 6 commits into
mainfrom
customer-gcp-kms-support

whites11 commented Apr 26, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Apr 26, 2026

Uh oh!

github-actions Bot commented Apr 26, 2026

Uh oh!

codecov Bot commented Apr 27, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Apr 27, 2026

Uh oh!

github-actions Bot commented Apr 27, 2026

Uh oh!

github-actions Bot commented Apr 27, 2026

Uh oh!

Uh oh!

claude Bot commented May 7, 2026

Uh oh!

masterashu left a comment

Uh oh!

whites11 commented May 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

whites11 commented Apr 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

github-actions Bot commented Apr 26, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented Apr 26, 2026

🔄 Flaky Test Detected

Uh oh!

codecov Bot commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 2 Tests Failed:

Uh oh!

github-actions Bot commented Apr 27, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented Apr 27, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented Apr 27, 2026

🔄 Flaky Test Detected

Uh oh!

Uh oh!

claude Bot commented May 7, 2026

Code review

Uh oh!

masterashu left a comment

Choose a reason for hiding this comment

Uh oh!

whites11 commented May 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

whites11 commented Apr 26, 2026 •

edited

Loading

codecov Bot commented Apr 27, 2026 •

edited

Loading