Skip to content

Commit 82c1e49

Browse files
gaspergromgaspergrom
committed
feat: expose health score v2 sub-signal detail via new pipe chain (CM-IN-1212)
Promotes previously computed-but-discarded sub-signal scores and availability flags in the 3 category pipes (health_score_v2_maintainer, health_score_v2_security, health_score_v2_development), and adds a new pipe chain (health_score_v2_signal_detail -> project_insights_health_breakdown_copy -> project_insights_health_breakdown) rolling repo-level signal detail up to project level for the insights frontend's health-breakdown UI. No scoring/business logic changed, only column exposure and rollup. Signed-off-by: Gašper Grom <gasper.grom@gmail.com>
1 parent c279f16 commit 82c1e49

11 files changed

Lines changed: 644 additions & 17 deletions

‎services/libs/tinybird/datasources/health_score_v2_development_ds.datasource‎

Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,41 @@ DESCRIPTION >
44
- `repoUrl` is the repository URL — the join key back to `repositories`.
55
- `developmentActivityScoreV2` — NULL when covered sub-signal weight is <40% of the 25pt max
66
(spec Layer 1 graceful degradation), otherwise the rescaled 0-25 score.
7+
- `releaseCadenceScore`/`commitActivityScore`/`issueResolutionScore`/`prMergeScore` (IN-1212) are
8+
the raw per-signal sub-scores before rescaling, as computed in `health_score_v2_development.pipe`.
9+
- `releaseCadenceAvailable`/`issueResolutionAvailable`/`prMergeAvailable` (IN-1212) are the Layer 1
10+
coverage flags — 0 means the signal was `blocked` (no data), not scored 0. Commit activity has no
11+
`Available` flag: it always contributes via the `repos.lastCommitAt` fallback (see pipe DESCRIPTION).
12+
- `daysSinceLatest`/`daysBetweenRecent` (IN-1212) are the raw release-cadence signal; NULL when the
13+
repo has no published packages with release dates.
14+
- `commitsLast6m` (IN-1212) is the raw commit count behind `commitActivityScore`; NULL when the repo
15+
has no `authored-commit` activity (the score then falls back to `lastCommitAt`).
16+
- `lastCommitAt` (IN-1212) is the repo's last commit timestamp, used as the commit-activity fallback.
17+
- `closed12m`/`opened12m`/`medianCloseS` (IN-1212) are the raw issue-resolution counts/median behind
18+
`issueResolutionScore`; NULL when unavailable.
19+
- `merged12m`/`closedUnmerged12m`/`medianMergeS` (IN-1212) are the raw PR-merge counts/median behind
20+
`prMergeScore`; NULL when unavailable.
721

822
SCHEMA >
923
`repoUrl` String,
10-
`developmentActivityScoreV2` Nullable(UInt8)
24+
`developmentActivityScoreV2` Nullable(UInt8),
25+
`releaseCadenceScore` UInt8,
26+
`releaseCadenceAvailable` UInt8,
27+
`daysSinceLatest` Nullable(Int64),
28+
`daysBetweenRecent` Nullable(Int64),
29+
`commitActivityScore` UInt8,
30+
`commitsLast6m` Nullable(UInt64),
31+
`lastCommitAt` Nullable(DateTime64(3)),
32+
`issueResolutionScore` UInt8,
33+
`issueResolutionAvailable` UInt8,
34+
`closed12m` Nullable(UInt64),
35+
`opened12m` Nullable(UInt64),
36+
`medianCloseS` Nullable(Float64),
37+
`prMergeScore` UInt8,
38+
`prMergeAvailable` UInt8,
39+
`merged12m` Nullable(UInt64),
40+
`closedUnmerged12m` Nullable(UInt64),
41+
`medianMergeS` Nullable(Float64)
1142

1243
ENGINE MergeTree
1344
ENGINE_SORTING_KEY repoUrl

‎services/libs/tinybird/datasources/health_score_v2_maintainer_ds.datasource‎

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,36 @@ DESCRIPTION >
44
- `repoUrl` is the repository URL — the join key back to `repositories`.
55
- `maintainerHealthScoreV2` — NULL when covered sub-signal weight is <40% of the 40pt max
66
(spec Layer 1 graceful degradation), otherwise the rescaled 0-40 score.
7+
- `busFactorScore`/`orgDiversityScore`/`responsivenessScore` (IN-1212) are the raw per-signal
8+
sub-scores before rescaling, as computed in `health_score_v2_maintainer.pipe`.
9+
- `busFactorAvailable`/`orgDiversityAvailable`/`responsivenessAvailable` (IN-1212) are the Layer 1
10+
coverage flags — 0 means the signal was `blocked` (no data), not scored 0.
11+
- `busFactorCount`/`orgCount` (IN-1212) are the raw counts behind the bus-factor and org-diversity
12+
scores; NULL when the signal is blocked.
13+
- `medianPrResponseS`/`medianIssueResponseS` (IN-1212) are the raw median response times (seconds)
14+
behind the responsiveness score; NULL when unavailable.
15+
- `isGerrit`/`isExcluded` (IN-1212) are the repo classification flags used to decide responsiveness
16+
availability and scoring branch.
17+
- `coveredWeight` (IN-1212) is the sum of available sub-signal weights (max 40) used as the
18+
rescale denominator for `maintainerHealthScoreV2` — exposed so the content spec's "category
19+
coverage" UI can show what fraction of the 40pt max was actually scored.
720

821
SCHEMA >
922
`repoUrl` String,
10-
`maintainerHealthScoreV2` Nullable(UInt8)
23+
`maintainerHealthScoreV2` Nullable(UInt8),
24+
`busFactorScore` UInt8,
25+
`busFactorAvailable` UInt8,
26+
`busFactorCount` Nullable(UInt64),
27+
`orgDiversityScore` UInt8,
28+
`orgDiversityAvailable` UInt8,
29+
`orgCount` Nullable(UInt64),
30+
`responsivenessScore` UInt8,
31+
`responsivenessAvailable` UInt8,
32+
`medianPrResponseS` Nullable(Float64),
33+
`medianIssueResponseS` Nullable(Float64),
34+
`isGerrit` UInt8,
35+
`isExcluded` UInt8,
36+
`coveredWeight` UInt8
1137

1238
ENGINE MergeTree
1339
ENGINE_SORTING_KEY repoUrl

‎services/libs/tinybird/datasources/health_score_v2_security_ds.datasource‎

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,40 @@ DESCRIPTION >
44
- `repoUrl` is the repository URL — the join key back to `repositories`.
55
- `securitySupplyChainScoreV2` — NULL when covered sub-signal weight is <40% of the 35pt max
66
(spec Layer 1 graceful degradation), otherwise the rescaled 0-35 score.
7+
- `openVulnScore`/`scorecardScorePts`/`securityPracticesScore`/`dependencyHealthScore` (IN-1212)
8+
are the raw per-signal sub-scores before rescaling, as computed in `health_score_v2_security.pipe`.
9+
- `scorecardAvailable`/`securityPracticesAvailable`/`dependencyHealthAvailable` (IN-1212) are the
10+
Layer 1 coverage flags — 0 means the signal was `blocked` (no data), not scored 0.
11+
- `openCriticals`/`openHighs`/`openModerates` (IN-1212) are the raw open-vulnerability counts by
12+
severity; NULL when the repo has no rows in `vulnerabilities`.
13+
- `scorecardScore` (IN-1212) is the raw OpenSSF Scorecard aggregate score (0-10 as a string,
14+
passed through from `repos.scorecardScore`; empty string when unavailable).
15+
- `securityPolicyEnabled`/`branchProtectionEnabled`/`branchProtectionRequiredReviews`/
16+
`branchProtectionRequiresStatusChecks`/`branchProtectionAllowsForcePush` (IN-1212) are the raw
17+
security-practice flags behind `securityPracticesScore`; NULL when unavailable.
18+
- `vulnerableDeps` (IN-1212) is the raw count of dependencies with an open HIGH/CRITICAL advisory
19+
behind `dependencyHealthScore`; NULL when the repo has no published packages.
720

821
SCHEMA >
922
`repoUrl` String,
10-
`securitySupplyChainScoreV2` Nullable(UInt8)
23+
`securitySupplyChainScoreV2` Nullable(UInt8),
24+
`openVulnScore` UInt8,
25+
`openCriticals` Nullable(UInt64),
26+
`openHighs` Nullable(UInt64),
27+
`openModerates` Nullable(UInt64),
28+
`scorecardScorePts` UInt8,
29+
`scorecardAvailable` UInt8,
30+
`scorecardScore` String,
31+
`securityPracticesScore` UInt8,
32+
`securityPracticesAvailable` UInt8,
33+
`securityPolicyEnabled` Nullable(UInt8),
34+
`branchProtectionEnabled` Nullable(UInt8),
35+
`branchProtectionRequiredReviews` Nullable(Int32),
36+
`branchProtectionRequiresStatusChecks` Nullable(UInt8),
37+
`branchProtectionAllowsForcePush` Nullable(UInt8),
38+
`dependencyHealthScore` UInt8,
39+
`dependencyHealthAvailable` UInt8,
40+
`vulnerableDeps` Nullable(UInt64)
1141

1242
ENGINE MergeTree
1343
ENGINE_SORTING_KEY repoUrl

‎services/libs/tinybird/datasources/health_score_v2_signal_detail_ds.datasource‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
DESCRIPTION >
2+
- `health_score_v2_signal_detail_ds` holds the per-repo flat signal-detail row for Health Score v2
3+
(IN-1212 Content Spec) — every promoted per-signal raw value and `*Available` coverage flag from
4+
the 3 category datasources (`health_score_v2_maintainer_ds`, `health_score_v2_security_ds`,
5+
`health_score_v2_development_ds`), joined on `repoUrl`. Populated by
6+
`health_score_v2_signal_detail.pipe`.
7+
- `repoUrl` is the repository URL — the join key back to `repositories`.
8+
- Maintainer signals: `busFactorScore`, `busFactorAvailable`, `busFactorCount`,
9+
`orgDiversityScore`, `orgDiversityAvailable`, `orgCount`, `responsivenessScore`,
10+
`responsivenessAvailable`, `medianPrResponseS`, `medianIssueResponseS`, `isGerrit`, `isExcluded`
11+
— see `health_score_v2_maintainer_ds` for definitions.
12+
- Security signals: `openVulnScore`, `openCriticals`, `openHighs`, `openModerates`,
13+
`scorecardScorePts`, `scorecardAvailable`, `scorecardScore`, `securityPracticesScore`,
14+
`securityPracticesAvailable`, `securityPolicyEnabled`, `branchProtectionEnabled`,
15+
`branchProtectionRequiredReviews`, `branchProtectionRequiresStatusChecks`,
16+
`branchProtectionAllowsForcePush`, `dependencyHealthScore`, `dependencyHealthAvailable`,
17+
`vulnerableDeps` — see `health_score_v2_security_ds` for definitions.
18+
- Development signals: `releaseCadenceScore`, `releaseCadenceAvailable`, `daysSinceLatest`,
19+
`daysBetweenRecent`, `commitActivityScore`, `commitsLast6m`, `lastCommitAt`,
20+
`issueResolutionScore`, `issueResolutionAvailable`, `closed12m`, `opened12m`, `medianCloseS`,
21+
`prMergeScore`, `prMergeAvailable`, `merged12m`, `closedUnmerged12m`, `medianMergeS` — see
22+
`health_score_v2_development_ds` for definitions.
23+
- All columns except `repoUrl` are Nullable here (even ones that are non-Nullable in their source
24+
`_ds`) because this table is built via LEFT JOIN — a repo missing from one of the 3 source
25+
datasources surfaces NULL for that category's columns rather than dropping the repo.
26+
27+
SCHEMA >
28+
`repoUrl` String,
29+
`busFactorScore` Nullable(UInt8),
30+
`busFactorAvailable` Nullable(UInt8),
31+
`busFactorCount` Nullable(UInt64),
32+
`orgDiversityScore` Nullable(UInt8),
33+
`orgDiversityAvailable` Nullable(UInt8),
34+
`orgCount` Nullable(UInt64),
35+
`responsivenessScore` Nullable(UInt8),
36+
`responsivenessAvailable` Nullable(UInt8),
37+
`medianPrResponseS` Nullable(Float64),
38+
`medianIssueResponseS` Nullable(Float64),
39+
`isGerrit` Nullable(UInt8),
40+
`isExcluded` Nullable(UInt8),
41+
`openVulnScore` Nullable(UInt8),
42+
`openCriticals` Nullable(UInt64),
43+
`openHighs` Nullable(UInt64),
44+
`openModerates` Nullable(UInt64),
45+
`scorecardScorePts` Nullable(UInt8),
46+
`scorecardAvailable` Nullable(UInt8),
47+
`scorecardScore` Nullable(String),
48+
`securityPracticesScore` Nullable(UInt8),
49+
`securityPracticesAvailable` Nullable(UInt8),
50+
`securityPolicyEnabled` Nullable(UInt8),
51+
`branchProtectionEnabled` Nullable(UInt8),
52+
`branchProtectionRequiredReviews` Nullable(Int32),
53+
`branchProtectionRequiresStatusChecks` Nullable(UInt8),
54+
`branchProtectionAllowsForcePush` Nullable(UInt8),
55+
`dependencyHealthScore` Nullable(UInt8),
56+
`dependencyHealthAvailable` Nullable(UInt8),
57+
`vulnerableDeps` Nullable(UInt64),
58+
`releaseCadenceScore` Nullable(UInt8),
59+
`releaseCadenceAvailable` Nullable(UInt8),
60+
`daysSinceLatest` Nullable(Int64),
61+
`daysBetweenRecent` Nullable(Int64),
62+
`commitActivityScore` Nullable(UInt8),
63+
`commitsLast6m` Nullable(UInt64),
64+
`lastCommitAt` Nullable(DateTime64(3)),
65+
`issueResolutionScore` Nullable(UInt8),
66+
`issueResolutionAvailable` Nullable(UInt8),
67+
`closed12m` Nullable(UInt64),
68+
`opened12m` Nullable(UInt64),
69+
`medianCloseS` Nullable(Float64),
70+
`prMergeScore` Nullable(UInt8),
71+
`prMergeAvailable` Nullable(UInt8),
72+
`merged12m` Nullable(UInt64),
73+
`closedUnmerged12m` Nullable(UInt64),
74+
`medianMergeS` Nullable(Float64)
75+
76+
ENGINE MergeTree
77+
ENGINE_SORTING_KEY repoUrl

‎services/libs/tinybird/datasources/project_insights_health_breakdown_ds.datasource‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
DESCRIPTION >
2+
- `project_insights_health_breakdown_ds` contains the materialized project-level Health Score v2
3+
signal breakdown (IN-1212), populated by `project_insights_health_breakdown_copy.pipe` on a
4+
schedule.
5+
- Precomputed so the public `project_insights_health_breakdown.pipe` endpoint is a cheap filtered
6+
SELECT rather than a live per-request rollup — see that pipe's DESCRIPTION and
7+
`project_insights_health_breakdown_copy.pipe`'s DESCRIPTION for the full per-column aggregation
8+
methodology (score columns avg, count columns max, median-seconds columns avg, boolean/flag
9+
columns max, `isGerrit`/`isExcluded` min).
10+
- `projectId` is the project id (`insightsProjects.id`).
11+
- `slug` is the project's URL-friendly identifier — the primary filter key for the endpoint.
12+
- Maintainer signals: `busFactorScore` (avg), `busFactorAvailable` (max), `busFactorCount` (max),
13+
`orgDiversityScore` (avg), `orgDiversityAvailable` (max), `orgCount` (max),
14+
`responsivenessScore` (avg), `responsivenessAvailable` (max), `medianPrResponseS` (avg),
15+
`medianIssueResponseS` (avg), `isGerrit` (min), `isExcluded` (min).
16+
- Security signals: `openVulnScore` (avg), `openCriticals`/`openHighs`/`openModerates` (max),
17+
`scorecardScorePts` (avg), `scorecardAvailable` (max), `scorecardScore` (avg, cast from the raw
18+
String), `securityPracticesScore` (avg), `securityPracticesAvailable` (max),
19+
`securityPolicyEnabled`/`branchProtectionEnabled`/`branchProtectionRequiredReviews`/
20+
`branchProtectionRequiresStatusChecks`/`branchProtectionAllowsForcePush` (max),
21+
`dependencyHealthScore` (avg), `dependencyHealthAvailable` (max), `vulnerableDeps` (max).
22+
- Development signals: `releaseCadenceScore` (avg), `releaseCadenceAvailable` (max),
23+
`daysSinceLatest`/`daysBetweenRecent` (max), `commitActivityScore` (avg), `commitsLast6m` (max),
24+
`lastCommitAt` (max), `issueResolutionScore` (avg), `issueResolutionAvailable` (max),
25+
`closed12m`/`opened12m` (max), `medianCloseS` (avg), `prMergeScore` (avg), `prMergeAvailable`
26+
(max), `merged12m`/`closedUnmerged12m` (max), `medianMergeS` (avg).
27+
- All columns except `projectId`/`slug` are Nullable: a project with zero enabled/non-excluded
28+
repos, or repos with no Health Score v2 signal data, rolls up to NULL rather than a fabricated 0.
29+
30+
SCHEMA >
31+
`projectId` String,
32+
`slug` String,
33+
`busFactorScore` Nullable(Float64),
34+
`busFactorAvailable` Nullable(UInt8),
35+
`busFactorCount` Nullable(UInt64),
36+
`orgDiversityScore` Nullable(Float64),
37+
`orgDiversityAvailable` Nullable(UInt8),
38+
`orgCount` Nullable(UInt64),
39+
`responsivenessScore` Nullable(Float64),
40+
`responsivenessAvailable` Nullable(UInt8),
41+
`medianPrResponseS` Nullable(Float64),
42+
`medianIssueResponseS` Nullable(Float64),
43+
`isGerrit` Nullable(UInt8),
44+
`isExcluded` Nullable(UInt8),
45+
`openVulnScore` Nullable(Float64),
46+
`openCriticals` Nullable(UInt64),
47+
`openHighs` Nullable(UInt64),
48+
`openModerates` Nullable(UInt64),
49+
`scorecardScorePts` Nullable(Float64),
50+
`scorecardAvailable` Nullable(UInt8),
51+
`scorecardScore` Nullable(Float64),
52+
`securityPracticesScore` Nullable(Float64),
53+
`securityPracticesAvailable` Nullable(UInt8),
54+
`securityPolicyEnabled` Nullable(UInt8),
55+
`branchProtectionEnabled` Nullable(UInt8),
56+
`branchProtectionRequiredReviews` Nullable(Int32),
57+
`branchProtectionRequiresStatusChecks` Nullable(UInt8),
58+
`branchProtectionAllowsForcePush` Nullable(UInt8),
59+
`dependencyHealthScore` Nullable(Float64),
60+
`dependencyHealthAvailable` Nullable(UInt8),
61+
`vulnerableDeps` Nullable(UInt64),
62+
`releaseCadenceScore` Nullable(Float64),
63+
`releaseCadenceAvailable` Nullable(UInt8),
64+
`daysSinceLatest` Nullable(Int64),
65+
`daysBetweenRecent` Nullable(Int64),
66+
`commitActivityScore` Nullable(Float64),
67+
`commitsLast6m` Nullable(UInt64),
68+
`lastCommitAt` Nullable(DateTime64(3)),
69+
`issueResolutionScore` Nullable(Float64),
70+
`issueResolutionAvailable` Nullable(UInt8),
71+
`closed12m` Nullable(UInt64),
72+
`opened12m` Nullable(UInt64),
73+
`medianCloseS` Nullable(Float64),
74+
`prMergeScore` Nullable(Float64),
75+
`prMergeAvailable` Nullable(UInt8),
76+
`merged12m` Nullable(UInt64),
77+
`closedUnmerged12m` Nullable(UInt64),
78+
`medianMergeS` Nullable(Float64)
79+
80+
ENGINE MergeTree
81+
ENGINE_SORTING_KEY slug, projectId

‎services/libs/tinybird/pipes/health_score_v2_development.pipe‎

Lines changed: 47 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,45 @@ SQL >
1919
coveredWeight / 25.0 < 0.4,
2020
NULL,
2121
toNullable(toUInt8(round(least(25.0, rawScore * (25.0 / coveredWeight)))))
22-
) AS developmentActivityScoreV2
22+
) AS developmentActivityScoreV2,
23+
releaseCadenceScore,
24+
releaseCadenceAvailable,
25+
daysSinceLatest,
26+
daysBetweenRecent,
27+
commitActivityScore,
28+
commitsLast6m,
29+
lastCommitAt,
30+
issueResolutionScore,
31+
issueResolutionAvailable,
32+
closed12m,
33+
opened12m,
34+
medianCloseS,
35+
prMergeScore,
36+
prMergeAvailable,
37+
merged12m,
38+
closedUnmerged12m,
39+
medianMergeS
2340
FROM
2441
(
2542
SELECT
2643
repoUrl,
44+
releaseCadenceScore,
45+
releaseCadenceAvailable,
46+
daysSinceLatest,
47+
daysBetweenRecent,
48+
commitActivityScore,
49+
commitsLast6m,
50+
lastCommitAt,
51+
issueResolutionScore,
52+
issueResolutionAvailable,
53+
closed12m,
54+
opened12m,
55+
medianCloseS,
56+
prMergeScore,
57+
prMergeAvailable,
58+
merged12m,
59+
closedUnmerged12m,
60+
medianMergeS,
2761
(
2862
releaseCadenceAvailable * releaseCadenceScore
2963
+ commitActivityScore
@@ -40,7 +74,10 @@ SQL >
4074
(
4175
SELECT
4276
rp.repoUrl AS repoUrl,
77+
rp.lastCommitAt AS lastCommitAt,
4378
(rs.repoUrl != '') AS releaseCadenceAvailable,
79+
rs.daysSinceLatest AS daysSinceLatest,
80+
rs.daysBetweenRecent AS daysBetweenRecent,
4481
multiIf(
4582
rs.daysSinceLatest < 90 AND rs.daysBetweenRecent < 90,
4683
8,
@@ -52,6 +89,7 @@ SQL >
5289
2,
5390
0
5491
) AS releaseCadenceScore,
92+
c.commitsLast6m AS commitsLast6m,
5593
-- commit activity is never `blocked`: it has its own spec-mandated fallback to
5694
-- repos.lastCommitAt, so it always contributes its full 5pt weight to coverage
5795
multiIf(
@@ -74,7 +112,10 @@ SQL >
74112
0
75113
) AS commitActivityScore,
76114
(ist.repoUrl != '') AS issueResolutionAvailable,
77-
(
115+
ist.closed12m AS closed12m,
116+
ist.opened12m AS opened12m,
117+
ist.medianCloseS AS medianCloseS,
118+
toUInt8(
78119
multiIf(
79120
ist.opened12m = 0,
80121
0,
@@ -87,7 +128,10 @@ SQL >
87128
+ multiIf(ist.medianCloseS < 604800, 3, ist.medianCloseS < 2592000, 2, 0)
88129
) AS issueResolutionScore,
89130
(prs.repoUrl != '') AS prMergeAvailable,
90-
(
131+
prs.merged12m AS merged12m,
132+
prs.closedUnmerged12m AS closedUnmerged12m,
133+
prs.medianMergeS AS medianMergeS,
134+
toUInt8(
91135
multiIf(
92136
(prs.merged12m + prs.closedUnmerged12m) = 0,
93137
0,

0 commit comments

Comments
 (0)