Return structured gRPC errors on source table missing by ilidemi · Pull Request #4194 · PeerDB-io/peerdb

ilidemi · 2026-04-18T00:24:39Z

Helps to unify the handling across connectors and consume on the caller without string matching.

Also clean up unused api error and move string constants to pkg. Would be nicer to have everything in protos but string constants are not supported there and not all our constants are uppercase to be enums.

codecov · 2026-04-18T00:41:21Z

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
2224	2	2222	203

View the top 2 failed test(s) by shortest run time

github.com/PeerDB-io/peerdb/flow/e2e::TestApiPg

Stack Traces | 0.01s run time

=== RUN   TestApiPg
=== PAUSE TestApiPg
=== CONT  TestApiPg
--- FAIL: TestApiPg (0.01s)
2026/04/20 22:22:46 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_lzjkq2np.\"myMixedCaseTable\" ORDER BY id"
2026/04/20 22:22:46 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_lzjkq2np.\"myMixedCaseTable\" ORDER BY id"
2026/04/20 22:22:46 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_7041997332280922776 CURSOR FOR SELECT id,\"key\" FROM e2e_test_pgch_lzjkq2np.\"myMixedCaseTable\" ORDER BY id" args=[]
2026/04/20 22:22:46 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_lzjkq2np.\"myMixedCaseTable\" ORDER BY id" channelLen=0
2026/04/20 22:22:46 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/04/20 22:22:46 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_7041997332280922776
2026/04/20 22:22:46 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_7041997332280922776 records=2 bytes=15 channelLen=1
2026/04/20 22:22:46 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_lzjkq2np.\"myMixedCaseTable\" ORDER BY id" rows=2 bytes=15 channelLen=1
2026/04/20 22:22:46 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_7041997332280922776
2026/04/20 22:22:46 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_7041997332280922776 records=0 bytes=0 channelLen=0
2026/04/20 22:22:46 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_lzjkq2np.\"myMixedCaseTable\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/04/20 22:22:46 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/04/20 22:22:46 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,\"key\" FROM e2e_test_pgch_lzjkq2np.\"myMixedCaseTable\" ORDER BY id" rows=2 bytes=15 channelLen=0

github.com/PeerDB-io/peerdb/flow/e2e::TestApiPg/TestCancelTableAddition_WithRemoval

Stack Traces | 24.2s run time

=== RUN   TestApiPg/TestCancelTableAddition_WithRemoval
=== PAUSE TestApiPg/TestCancelTableAddition_WithRemoval
=== CONT  TestApiPg/TestCancelTableAddition_WithRemoval
    cancel_table_addition_test.go:361: WaitFor wait for initial load to finish 2026-04-20 22:21:15.756286239 +0000 UTC m=+544.928337170
    cancel_table_addition_test.go:365: WaitFor t1 initial load 2026-04-20 22:21:15.756670536 +0000 UTC m=+544.928721478
2026/04/20 22:21:15 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_xjuse7ab.\"t1\" ORDER BY id"
2026/04/20 22:21:15 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_xjuse7ab.\"t1\" ORDER BY id"
2026/04/20 22:21:15 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_5043154007557543504 CURSOR FOR SELECT id,val FROM e2e_test_api_xjuse7ab.\"t1\" ORDER BY id" args=[]
2026/04/20 22:21:15 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_xjuse7ab.\"t1\" ORDER BY id" channelLen=0
2026/04/20 22:21:15 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5043154007557543504
2026/04/20 22:21:15 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5043154007557543504 records=1 bytes=9 channelLen=0
2026/04/20 22:21:15 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_xjuse7ab.\"t1\" ORDER BY id" rows=1 bytes=9 channelLen=0
2026/04/20 22:21:15 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5043154007557543504
2026/04/20 22:21:15 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_5043154007557543504 records=0 bytes=0 channelLen=0
2026/04/20 22:21:15 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_xjuse7ab.\"t1\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/04/20 22:21:15 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/04/20 22:21:15 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_xjuse7ab.\"t1\" ORDER BY id" rows=1 bytes=9 channelLen=0
    cancel_table_addition_test.go:366: WaitFor t2 initial load 2026-04-20 22:21:15.770296333 +0000 UTC m=+544.942347264
2026/04/20 22:21:15 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_xjuse7ab.\"t2\" ORDER BY id"
2026/04/20 22:21:15 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_xjuse7ab.\"t2\" ORDER BY id"
2026/04/20 22:21:15 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_3390404670187857547 CURSOR FOR SELECT id,val FROM e2e_test_api_xjuse7ab.\"t2\" ORDER BY id" args=[]
2026/04/20 22:21:15 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_xjuse7ab.\"t2\" ORDER BY id" channelLen=0
2026/04/20 22:21:15 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_3390404670187857547
2026/04/20 22:21:15 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_3390404670187857547 records=1 bytes=9 channelLen=0
2026/04/20 22:21:15 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_xjuse7ab.\"t2\" ORDER BY id" rows=1 bytes=9 channelLen=0
2026/04/20 22:21:15 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_3390404670187857547
2026/04/20 22:21:15 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_3390404670187857547 records=0 bytes=0 channelLen=0
2026/04/20 22:21:15 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_xjuse7ab.\"t2\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/04/20 22:21:15 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/04/20 22:21:15 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_xjuse7ab.\"t2\" ORDER BY id" rows=1 bytes=9 channelLen=0
    cancel_table_addition_test.go:367: WaitFor t3 initial load 2026-04-20 22:21:15.776666033 +0000 UTC m=+544.948716974
2026/04/20 22:21:15 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_xjuse7ab.\"t3\" ORDER BY id"
2026/04/20 22:21:15 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_xjuse7ab.\"t3\" ORDER BY id"
2026/04/20 22:21:15 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_13920649951805865243 CURSOR FOR SELECT id,val FROM e2e_test_api_xjuse7ab.\"t3\" ORDER BY id" args=[]
2026/04/20 22:21:15 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_xjuse7ab.\"t3\" ORDER BY id" channelLen=0
2026/04/20 22:21:15 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_13920649951805865243
2026/04/20 22:21:15 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_13920649951805865243 records=1 bytes=9 channelLen=0
2026/04/20 22:21:15 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_xjuse7ab.\"t3\" ORDER BY id" rows=1 bytes=9 channelLen=0
2026/04/20 22:21:15 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_13920649951805865243
2026/04/20 22:21:15 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_13920649951805865243 records=0 bytes=0 channelLen=0
2026/04/20 22:21:15 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_xjuse7ab.\"t3\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/04/20 22:21:15 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/04/20 22:21:15 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_xjuse7ab.\"t3\" ORDER BY id" rows=1 bytes=9 channelLen=0
    cancel_table_addition_test.go:375: WaitFor wait for pause for add table 2026-04-20 22:21:15.790652703 +0000 UTC m=+544.962703644
    cancel_table_addition_test.go:376: UNEXPECTED ERROR unable to establish connection with catalog: FATAL: terminating connection due to administrator command (SQLSTATE 57P01)
    api_test.go:49: begin tearing down postgres schema api_xjuse7ab
--- FAIL: TestApiPg/TestCancelTableAddition_WithRemoval (24.24s)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

github-actions · 2026-04-18T00:45:17Z

🔄 Flaky Test Detected

Analysis: The e2e test suite timed out at 900.562s (just 0.5s over the 900s limit) on the MariaDB 8.0 matrix job, a classic flaky timeout with no assertion failures.
Confidence: 0.9

✅ Automatically retrying the workflow

View workflow run

github-actions · 2026-04-18T01:19:29Z

🔄 Flaky Test Detected

Analysis: Two e2e tests failed due to transient "conn closed" postgres connection errors during teardown and under high parallel load (32 concurrent tests, ~760s of 900s timeout), with different tests failing across different matrix configurations — classic flaky infrastructure behavior.
Confidence: 0.9

✅ Automatically retrying the workflow

View workflow run

ilidemi · 2026-04-20T19:28:14Z

Will add a change to return all missing tables at once, resulting contract might be different

github-actions · 2026-04-20T21:55:51Z

🔄 Flaky Test Detected

Analysis: The BigQuery e2e test Test_Inheritance_Table_With_Dynamic_Setting timed out waiting on STATUS_SNAPSHOT, a transient external-service timing failure rather than a code defect.
Confidence: 0.9

✅ Automatically retrying the workflow

View workflow run

github-actions · 2026-04-20T22:07:02Z

❌ Test Failure

Analysis: Deterministic build failure: common.NewSourceTableMissingError is undefined but referenced in three connector files (mysql, bigquery, postgres), indicating an incomplete implementation on the table-missing-structured-errors branch.
Confidence: 0.99

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-04-20T22:09:26Z

🔄 Flaky Test Detected

Analysis: The entire e2e test suite hit the 900s wall-clock timeout (panic: test timed out after 15m0s), cascading context deadline exceeded failures across unrelated suites (PG, MySQL, Mongo, BQ) — classic over-parallelized (-p 32) e2e timeout flake on a 16-core runner.
Confidence: 0.9

✅ Automatically retrying the workflow

View workflow run

github-actions · 2026-04-20T22:14:32Z

🔄 Flaky Test Detected

Analysis: The test TestCancelTableAddition_NoRemovalAssumedWithRemoval failed because PostgreSQL terminated the connection with SQLSTATE 57P01 (admin_shutdown), indicating a transient infrastructure-level connection kill rather than any code logic bug.
Confidence: 0.93

✅ Automatically retrying the workflow

View workflow run

github-actions · 2026-04-20T22:27:20Z

🔄 Flaky Test Detected

Analysis: TestCancelTableAddition_WithRemoval failed with a Postgres admin_shutdown error (SQLSTATE 57P01) mid-test, indicating a race condition where another parallel test's teardown terminated the catalog connection unexpectedly.
Confidence: 0.88

✅ Automatically retrying the workflow

View workflow run

jgao54 · 2026-04-22T02:47:03Z


+	// ValidateMirrorSource checks that the source is ready to replicate the configured tables.
+	// MUST return *common.SourceTablesMissingError when a mapped source table is absent.
 	ValidateMirrorSource(context.Context, *protos.FlowConnectionConfigsCore) error


for my understanding, what's the reason for not just changing the return type here to *common.SourceTablesMissingError

nvm, i see that the function can return other types of errors as well.

eventually we will get to a place where we can replace all the error.fmt with structured errors 💪

Return structured gRPC errors on resync table missing

eec7756

Support BQ

78fb576

ilidemi requested a review from jgao54 April 18, 2026 01:00

ilidemi removed the request for review from jgao54 April 20, 2026 19:27

ilidemi marked this pull request as draft April 20, 2026 19:27

ilidemi added 2 commits April 20, 2026 14:38

Return machine-readable list of tables, add not in publication error

8f62215

Check tables before the publication

ef61b3a

ilidemi marked this pull request as ready for review April 20, 2026 21:59

ilidemi requested a review from jgao54 April 20, 2026 21:59

plural, simplify

d900edf

bad rename

c3de992

jgao54 reviewed Apr 22, 2026

View reviewed changes

jgao54 approved these changes Apr 22, 2026

View reviewed changes

ilidemi merged commit 1a2dfd1 into main Apr 22, 2026
23 of 24 checks passed

ilidemi deleted the table-missing-structured-errors branch April 22, 2026 04:46

Uh oh!

Conversation

ilidemi commented Apr 18, 2026

Uh oh!

codecov Bot commented Apr 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 2 Tests Failed:

Uh oh!

github-actions Bot commented Apr 18, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented Apr 18, 2026

🔄 Flaky Test Detected

Uh oh!

ilidemi commented Apr 20, 2026

Uh oh!

github-actions Bot commented Apr 20, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented Apr 20, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented Apr 20, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented Apr 20, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented Apr 20, 2026

🔄 Flaky Test Detected

Uh oh!

jgao54 Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jgao54 Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

jgao54 Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Apr 18, 2026 •

edited

Loading

jgao54 Apr 22, 2026 •

edited

Loading

jgao54 Apr 22, 2026 •

edited

Loading