Ingest CI test execution stats (#4191)

This PR adds a flow step to the repository build ingesting test results
for internal analysis.

:memo: So far, this is implemented as a step within this repository CI
flow, it can be easily factored out as an action if we want to follow
the same strategy in other repositories.

The step:

1. Converts the go tests XML reports into NDJSON format using
`test-results.sh` which is just a wrapper for a `yq` query.
2. Uploads the NDJSON to a ClickHouse ingestion Query Endpoint. If the
endpoint failes (no response, connection interrupted or status code >=
400), the upload is retried a up to 5 times with power-of-two
exponential back-off.

This is done as the last step in the build so failures are not
disruptive.

## For internal reviewers

You should check the following details related to internal management of
this data:

- Data model:
https://linear.app/clickhouse/issue/DBI-671/store-peerdb-ci-test-results-for-flake-analysis#comment-62e1e006
- QA validation:
https://linear.app/clickhouse/issue/DBI-671/store-peerdb-ci-test-results-for-flake-analysis#comment-0b4514cb
- Internal documentation:
https://www.notion.so/PeerDB-Test-Infrastructure-3482bd19f29080ea8199dcf57d822e88?v=3212bd19f29080be82b2000c45d4fd04&source=copy_link#3482bd19f2908073878cffb9992a25aa

## Ready when

- [x] Pipeline ingestion is tested.

## What does this PR resolve?

Part of: https://linear.app/clickhouse/issue/DBI-671

Author: pfcoperez

.github/scripts/test-results.sh

@@ -0,0 +1,53 @@
+#!/bin/bash
+
+set -eo pipefail # For jq to propagate yq failures.
+
+# This script processes the XML tests results and outputs a NDJSON stream with one document per test execution.
+# Hence, each document corresponds to a testcase entry in the XML. It contains the following fields:
+#
+#   - 'timestamp': As the testsuite (parent of the entry) timestamp field.
+#   - 'suite_name': As the entry class name.
+#   - 'test': As the entry name.
+#   - 'result': success/failure/skipped from the optional child node of the entry. No child node means success.
+#   - 'reason': As the entry child (skipped or failure) contents. For example, in the source, <failure>SOURCE FOR REASON</failure>.
+#   - 'duration_seconds': As the time field of the entry.
+#   - 'workflow_run_id': As the content of the environment variable 'WORKFLOW_RUN_ID'
+#   - 'workflow_run_link': As the content of the environment variable 'WORKFLOW_RUN_LINK'
+#   - 'workflow_head_branch': As the content of the environment variable 'WORKFLOW_HEAD_BRANCH'
+#   - 'workflow_retry_number': As the content of the environment variable 'WORKFLOW_RUN_NUMBER'
+#   - 'commit_sha': As the content of the environment variable 'COMMIT_SHA'
+#
+# INPUT:
+#   - The XML file(s) to process. For example, the test results generated by JUnit in the 'target/surefire-reports' directory.
+#   - The following environment variables to enrich with the workflow information:
+#     - WORKFLOW_RUN_ID
+#     - WORKFLOW_RUN_LINK
+#     - WORKFLOW_HEAD_BRANCH
+#     - WORKFLOW_RUN_NUMBER
+#     - COMMIT_SHA
+#
+# OUTPUT: NDJSON stream through stdout.
+#
+yq -p=xml -o=json \
+  '
+  ((.testsuites.testsuite | select(kind == "seq") | .[]), (.testsuites.testsuite | select(kind == "map"))) |
+  ."+@timestamp" as $ts |
+  select(has("testcase")) |
+  ((.testcase | select(kind == "seq") | .[]), (.testcase | select(kind == "map"))) |
+  {
+    "timestamp": $ts,
+    "suite_name": ."+@classname",
+    "test": ."+@name",
+    "result": ((keys | map(select(. == "failure" or . == "skipped")) | .[0]) // "success"),
+    "reason": ((.failure."+content" // .skipped."+@message") // null),
+    "duration_seconds": (."+@time" | select(. != "") | tonumber // null),
+    "workflow_run_id": (strenv(WORKFLOW_RUN_ID) | tonumber),
+    "workflow_run_link": strenv(WORKFLOW_RUN_LINK),
+    "workflow_head_branch": strenv(WORKFLOW_HEAD_BRANCH),
+    "workflow_retry_number": (strenv(WORKFLOW_RUN_NUMBER) | tonumber),
+    "commit_sha": strenv(COMMIT_SHA)
+  } |
+  select(.suite_name != null)
+  ' \
+  "$@" \
+  | jq -c . # jq for compacting the output to NDJSON.

.github/workflows/flow.yml

@@ -116,6 +116,10 @@ jobs:
           sudo apt-get remove --purge man-db
           sudo apt-get install libgeos-dev
 
+      - name: install retry tool
+        run: |
+          sudo apt-get install -y retry
+
       - run: go mod download
         working-directory: ./flow
 
@@ -550,3 +554,39 @@ jobs:
         uses: codecov/codecov-action@57e3a136b779b570ffcdbf80b3bdc90e7fab3de2 # v6
         with:
           token: ${{ secrets.CODECOV_TOKEN }}
+      - name: Ingest tests results for analysis
+        if: success() || failure()
+        env:
+          WORKFLOW_RUN_ID: ${{ github.run_id }}
+          WORKFLOW_RUN_LINK: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+          WORKFLOW_HEAD_BRANCH: ${{ github.head_ref || github.ref_name }}
+          WORKFLOW_RUN_NUMBER: ${{ github.run_attempt }}
+          COMMIT_SHA: ${{ github.sha }}
+          SUMMARY_HEADER: "x-clickhouse-summary"
+        run: |
+          if [ -z "${{ secrets.CI_O11Y_TARGET_API_KEY_ID }}" ] \
+            || [ -z "${{ secrets.CI_O11Y_TARGET_API_KEY_SECRET }}" ] \
+            || [ -z "${{ secrets.CI_O11Y_TARGET_QUERY_ENDPOINT }}" ]; then
+              echo "Secrets not configured; Test results ingestion is disabled."
+              exit 0
+          fi
+          .github/scripts/test-results.sh logs/test-results.xml > logs/normalized-test-results.ndjson
+          if [ ! -s logs/normalized-test-results.ndjson ]; then
+              echo "Failed to extract test results"
+              exit 1
+          fi
+          # Retry upload with exponential backoff
+          retry --times=5 --delay "1,2,4,8,16,32,64,128" -- \
+          curl --fail-with-body -i -H "Content-Type: application/octet-stream" \
+            --user "${{ secrets.CI_O11Y_TARGET_API_KEY_ID }}:${{ secrets.CI_O11Y_TARGET_API_KEY_SECRET }}" \
+            "${{ secrets.CI_O11Y_TARGET_QUERY_ENDPOINT }}" \
+            --data-binary @logs/normalized-test-results.ndjson \
+            | tee upload.log
+          WRITTEN_ROWS=$(cat upload.log | grep $SUMMARY_HEADER | sed "s/$SUMMARY_HEADER: //" | jq '.written_rows' -r)
+          if [ -z "$WRITTEN_ROWS" ] || [ "$WRITTEN_ROWS" -eq 0 ]; then
+              echo "No results uploaded. Query client logs:"
+              cat upload.log
+              exit 2
+          fi
+          echo "Number of ingested test results: $WRITTEN_ROWS"
+          exit 0