Ingest CI test execution stats by pfcoperez · Pull Request #4191 · PeerDB-io/peerdb

pfcoperez · 2026-04-17T19:47:41Z

This PR adds a flow step to the repository build ingesting test results for internal analysis.

📝 So far, this is implemented as a step within this repository CI flow, it can be easily factored out as an action if we want to follow the same strategy in other repositories.

The step:

Converts the go tests XML reports into NDJSON format using test-results.sh which is just a wrapper for a yq query.
Uploads the NDJSON to a ClickHouse ingestion Query Endpoint. If the endpoint failes (no response, connection interrupted or status code >= 400), the upload is retried a up to 5 times with power-of-two exponential back-off.

This is done as the last step in the build so failures are not disruptive.

For internal reviewers

You should check the following details related to internal management of this data:

Ready when

Pipeline ingestion is tested.

What does this PR resolve?

Part of: https://linear.app/clickhouse/issue/DBI-671

Copilot

Pull request overview

Adds ingestion of CI test execution results from the Flow workflow into an internal analytics endpoint for observability/analysis (DBI-671).

Changes:

Add a new GitHub Actions step in flow.yml to normalize JUnit XML into NDJSON and upload it to an external endpoint.
Introduce a new .github/scripts/test-results.sh script to transform JUnit XML into per-test NDJSON records enriched with workflow metadata.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 9 comments.

File	Description
.github/workflows/flow.yml	Adds a post-test step that normalizes JUnit XML and uploads NDJSON to an analytics endpoint, then checks uploaded row count.
.github/scripts/test-results.sh	New yq/jq-based converter from JUnit XML to NDJSON with workflow metadata fields.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

codecov · 2026-04-17T20:39:39Z

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
2215	2	2213	202

View the top 2 failed test(s) by shortest run time

github.com/PeerDB-io/peerdb/flow/e2e::TestSwitchboardPostgres/Test_QueryComplexity_WindowFunction

Stack Traces | 0s run time

=== RUN   TestSwitchboardPostgres/Test_QueryComplexity_WindowFunction
=== PAUSE TestSwitchboardPostgres/Test_QueryComplexity_WindowFunction
=== CONT  TestSwitchboardPostgres/Test_QueryComplexity_WindowFunction
    e2eshared.go:34: 
        	Error Trace:	.../flow/e2e/congen.go:44
        	            				.../flow/e2e/pg.go:190
        	            				.../flow/e2e/switchboard_pg_test.go:38
        	            				.../flow/e2eshared/e2eshared.go:34
        	Error:      	Received unexpected error:
        	            	unable to establish connection with catalog: FATAL: terminating connection due to administrator command (SQLSTATE 57P01)
        	Test:       	TestSwitchboardPostgres/Test_QueryComplexity_WindowFunction
--- FAIL: TestSwitchboardPostgres/Test_QueryComplexity_WindowFunction (0.00s)

github.com/PeerDB-io/peerdb/flow/e2e::TestSwitchboardPostgres

Stack Traces | 0.01s run time

=== RUN   TestSwitchboardPostgres
=== PAUSE TestSwitchboardPostgres
=== CONT  TestSwitchboardPostgres
--- FAIL: TestSwitchboardPostgres (0.01s)
2026/04/20 10:21:31 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/04/20 10:21:31 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/04/20 10:21:31 INFO Executing and processing query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_mbh4gyhg.\"original\" ORDER BY id"
2026/04/20 10:21:31 INFO Executing and processing query stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_mbh4gyhg.\"original\" ORDER BY id"
2026/04/20 10:21:31 INFO [pg_query_executor] declared cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursorQuery="DECLARE peerdb_cursor_10689333627283282704 CURSOR FOR SELECT id,val FROM e2e_test_api_mbh4gyhg.\"original\" ORDER BY id" args=[]
2026/04/20 10:21:31 INFO [pg_query_executor] fetching rows start x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_mbh4gyhg.\"original\" ORDER BY id" channelLen=0
2026/04/20 10:21:31 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_10689333627283282704
2026/04/20 10:21:31 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_10689333627283282704 records=2 bytes=19 channelLen=1
2026/04/20 10:21:31 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_mbh4gyhg.\"original\" ORDER BY id" rows=2 bytes=19 channelLen=1
2026/04/20 10:21:31 INFO [pg_query_executor] fetching from cursor x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_10689333627283282704
2026/04/20 10:21:31 INFO processed row stream x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart cursor=peerdb_cursor_10689333627283282704 records=0 bytes=0 channelLen=0
2026/04/20 10:21:31 INFO [pg_query_executor] fetched rows x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_mbh4gyhg.\"original\" ORDER BY id" rows=0 bytes=0 channelLen=0
2026/04/20 10:21:31 INFO [pg_query_executor] committing transaction x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart
2026/04/20 10:21:31 INFO [pg_query_executor] committed transaction for query x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} partitionId=testpart query="SELECT id,val FROM e2e_test_api_mbh4gyhg.\"original\" ORDER BY id" rows=2 bytes=19 channelLen=0

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

github-actions · 2026-04-17T20:43:14Z

🔄 Flaky Test Detected

Analysis: TestCancelTableAdditionRemoveAddRemove failed due to a PostgreSQL SQLSTATE 57P01 (admin_shutdown) error — a transient catalog connection termination by the database server, not a code bug.
Confidence: 0.93

✅ Automatically retrying the workflow

View workflow run

github-actions · 2026-04-17T20:45:21Z

🔄 Flaky Test Detected

Analysis: Both failures share the same root cause: the PostgreSQL catalog connection was killed mid-test by an administrator command (SQLSTATE 57P01), a transient CI infrastructure event unrelated to the code change.
Confidence: 0.97

✅ Automatically retrying the workflow

View workflow run

github-actions · 2026-04-17T20:46:27Z

🔄 Flaky Test Detected

Analysis: TestResyncCompleted failed due to PostgreSQL SQLSTATE 57P01 ("terminating connection due to administrator command") — an external connection kill unrelated to the code under test.
Confidence: 0.93

✅ Automatically retrying the workflow

View workflow run

…existing files.

github-actions · 2026-04-19T20:41:39Z

🔄 Flaky Test Detected

Analysis: TestGenericBQ/Test_Inheritance_Table_With_Dynamic_Setting timed out waiting for STATUS_SNAPSHOT to complete — a known flaky pattern under high CI concurrency, consistent with explicit comments in the test suite about BigQuery flakiness.
Confidence: 0.92

✅ Automatically retrying the workflow

View workflow run

github-actions · 2026-04-19T20:45:36Z

🔄 Flaky Test Detected

Analysis: The test TestPeerFlowE2ETestSuitePG_CH_Cluster/Test_Binary_Format_Base64 failed because the PostgreSQL catalog connection was forcibly terminated by an administrator command (SQLSTATE 57P01), a transient CI infrastructure issue unrelated to any code change.
Confidence: 0.93

✅ Automatically retrying the workflow

View workflow run

…atus code <400

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

github-actions · 2026-04-20T08:39:13Z

🔄 Flaky Test Detected

Analysis: The test TestApiPg/TestEditTablesBeforeResync failed due to a PostgreSQL catalog connection being forcibly terminated by an external admin command (SQLSTATE 57P01 / admin_shutdown), which is a transient infrastructure failure unrelated to any code change.
Confidence: 0.95

✅ Automatically retrying the workflow

View workflow run

github-actions · 2026-04-20T08:56:46Z

🔄 Flaky Test Detected

Analysis: E2E test TestApiPg/TestCancelAddCancel timed out in a WaitFor polling loop waiting for async "initial load MV error messages" from a Temporal workflow, a classic flaky pattern in cancel/re-add table orchestration tests.
Confidence: 0.88

✅ Automatically retrying the workflow

View workflow run

github-actions · 2026-04-20T10:10:59Z

🔄 Flaky Test Detected

Analysis: CI infrastructure failure — Docker port 42002 was already in use on the runner when attempting to bind it for the toxiproxy container, preventing container startup before any tests ran.
Confidence: 0.97

✅ Automatically retrying the workflow

View workflow run

Two improvements following up #4191 1. Include information on the sources versions matrix combination. 2. For PRs, the commit information is now the last commit pushed by the author rather than the resulting commit from the automatic merge from main. --------- Co-authored-by: Ilia Demianenko <ilia.demianenko@clickhouse.com>

pfcoperez added 3 commits April 17, 2026 10:33

First version of the test results normalization script

ae86adc

Working extraction script

36f5954

Step to ingest test results

97f2aa9

pfcoperez requested a review from Copilot April 17, 2026 19:47

Copilot started reviewing on behalf of pfcoperez April 17, 2026 19:48 View session

Copilot AI reviewed Apr 17, 2026

View reviewed changes

pfcoperez and others added 3 commits April 17, 2026 22:23

PR Comments: Fix comments typos

b14ecec

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

PR Comments: Fix comments typos II

399e9eb

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Retry number is intended to be the number of attempts

40affbc

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

pfcoperez added 3 commits April 19, 2026 22:21

Add commit SHA

c5a56e8

Fail at workflow ingestion step on missing mandatory fields

41840f6

No need to use tee in append mode, it is safer to overwrite possible …

01fd1fc

…existing files.

pfcoperez and others added 2 commits April 20, 2026 10:18

Add retry logic for SQL endpoint: Success when server returns with st…

685daa7

…atus code <400

zc

86f23b0

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

pfcoperez added 2 commits April 20, 2026 11:34

Update script comments

68f9d6e

Change steps order to avoid disruptions.

c364877

pfcoperez requested review from ilidemi, jgao54 and masterashu April 20, 2026 10:53

pfcoperez self-assigned this Apr 20, 2026

pfcoperez marked this pull request as ready for review April 20, 2026 10:59

masterashu approved these changes Apr 20, 2026

View reviewed changes

pfcoperez merged commit 4154230 into main Apr 20, 2026
20 of 23 checks passed

pfcoperez deleted the DBI-671/test/o11y/store-peerdb-ci-test-results-for-flake-analysis branch April 20, 2026 20:07

ilidemi reviewed Apr 21, 2026

View reviewed changes

Comment thread .github/scripts/test-results.sh

Comment thread .github/workflows/flow.yml

pfcoperez mentioned this pull request Apr 21, 2026

CI Tests results ingestion: Follow-up improvements #4209

Merged

Uh oh!

Conversation

pfcoperez commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

For internal reviewers

Ready when

What does this PR resolve?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 2 Tests Failed:

Uh oh!

github-actions Bot commented Apr 17, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented Apr 17, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented Apr 17, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented Apr 19, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented Apr 19, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented Apr 20, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented Apr 20, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented Apr 20, 2026

🔄 Flaky Test Detected

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pfcoperez commented Apr 17, 2026 •

edited

Loading

codecov Bot commented Apr 17, 2026 •

edited

Loading