Skip to content

Commit aafb741

Browse files
authored
Replicate greenlight PR state from S3 to ClickHouse (#8410)
**Impact:** CI data pipeline only **Risk:** low ## What Adds a `greenlight_pr_state` adapter and registers it so objects landing under that S3 path are replicated into the `misc.greenlight_pr_state` ClickHouse table. ## Why The greenlight/PR-eval workflow writes per-PR verdicts (status, reason, eval/agent job refs, version) to S3, and those records need to be queryable in ClickHouse. This wires up the replicator so the data flows automatically, following the same JSONEachRow pattern used by the other adapters (e.g. `runner_fleet_count`). # Notes - Purely additive: a new adapter function plus two dictionary entries (`SUPPORTED_PATHS`, `OBJECT_CONVERTER`). No existing paths are touched. - The schema here must stay in sync with the `misc.greenlight_pr_state` table definition; column order matches the table's ordinary columns and `JSONEachRow` maps by field name. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
1 parent 492c1ba commit aafb741

1 file changed

Lines changed: 25 additions & 0 deletions

File tree

aws/lambda/clickhouse-replicator-s3/lambda_function.py

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -727,6 +727,29 @@ def runner_fleet_count_adapter(table, bucket, key):
727727
general_adapter(table, bucket, key, schema, ["gzip", "none"], "JSONEachRow")
728728

729729

730+
def greenlight_pr_state_adapter(table, bucket, key):
731+
# Column order must match the misc.greenlight_pr_state table's ordinary
732+
# columns (minus `_meta`, which general_adapter appends via `SELECT *,
733+
# (bucket, key)`). JSONEachRow maps by field name. LowCardinality(String)
734+
# table columns are read as plain String here; ClickHouse widens on insert.
735+
# NB: no timezone literal here -- general_adapter wraps this schema in single
736+
# quotes, so an inner 'UTC' would break the s3() SQL. `version` is a UTC
737+
# wall-clock string with millisecond precision.
738+
schema = """
739+
`repo` String,
740+
`pr_number` Int64,
741+
`head_sha` String,
742+
`status` String,
743+
`reason` String,
744+
`eval_hash` String,
745+
`message` String,
746+
`eval_job` String,
747+
`agent_job` String,
748+
`version` DateTime64(3)
749+
"""
750+
general_adapter(table, bucket, key, schema, ["gzip", "none"], "JSONEachRow")
751+
752+
730753
SUPPORTED_PATHS = {
731754
"merges": "default.merges",
732755
"queue_times_historical": "default.queue_times_historical",
@@ -753,6 +776,7 @@ def runner_fleet_count_adapter(table, bucket, key):
753776
"ghci-related": "infra_metrics.cloudwatch_metrics",
754777
"test_jsons_while_running": "tests.all_test_runs",
755778
"runner_fleet_count": "misc.runner_fleet_count",
779+
"greenlight_pr_state": "misc.greenlight_pr_state",
756780
}
757781

758782
OBJECT_CONVERTER = {
@@ -780,6 +804,7 @@ def runner_fleet_count_adapter(table, bucket, key):
780804
"misc.autorevert_advisor_verdicts": autorevert_advisor_verdicts_adapter,
781805
"infra_metrics.cloudwatch_metrics": cloudwatch_metrics_adapter,
782806
"misc.runner_fleet_count": runner_fleet_count_adapter,
807+
"misc.greenlight_pr_state": greenlight_pr_state_adapter,
783808
}
784809

785810

0 commit comments

Comments
 (0)