perf: use GraphQL batching for repo extras and contributor detail fetching

[arun-gupta]

Background

Follow-on to #8 (GraphQL batching for user/org repo fetching). The same bottleneck exists in two more pipeline steps: get_features_data() in get_repo_extras.py and get_contributor_data() in get_contributors.py. Both make one sequential REST call per repository or contributor — at UCSC scale that is tens of thousands of individual requests.

Why README and contributor data matter for affiliation scoring

The score-based classifier (SBC) uses three signals:

Signal	Without README/contributors	With README/contributors
repository_score	full_name + description + homepage only	+ full README text — university mentions, course names, faculty/lab names
organization_score	org name/description keywords	same
contributor_score	0 — no data	@ucsc.edu email domains, bios, company fields

In a test run on UCSC (24,919 repos):

• Without README/contributor data: 64 repos pass the 0.7 affiliation threshold
• Expected with README/contributor data: significantly more — repos with no university keywords in name/description but @ucsc.edu contributor emails or "UC Santa Cruz" in the README jump from score 0 → high

Score distribution without README/contributor data (UCSC, 24,919 repos):

mean:  0.039
std:   0.109
> 0.7: 64 repos
> 0.5: 192 repos
> 0.3: 727 repos

Current bottleneck

get_features_data() — get_repo_extras.py

• GET /repos/{owner}/{repo}/readme × N repos (one call per repo)
• GET /repos/{owner}/{repo}/releases × N repos
• GET /repos/{owner}/{repo}/community/profile × N repos
• At 24,919 repos: ~75,000 sequential REST calls for the three main features

get_contributor_data() — get_contributors.py

• GET /repos/{owner}/{repo}/contributors × N repos (get logins)
• GET /users/{login} × M contributors (get bio/email/company/location)
• At 24,919 repos with avg ~5 contributors: ~125,000+ sequential REST calls for contributor details alone

Proposed GraphQL optimisations

README + release downloads: batch 50 repos per query

query BatchRepoExtras {
  r0: repository(owner: "ucsd", name: "triton") {
    object(expression: "HEAD:README.md") { ... on Blob { text } }
    releases(first: 20) {
      nodes { releaseAssets(first: 10) { nodes { downloadCount } } }
    }
  }
  r1: repository(owner: "ucsc", name: "some-repo") { ... }
  ...
}

• Reduces ~50,000 REST calls → ~500 GraphQL requests (~100× fewer)
• Case-sensitivity caveat: GraphQL object(expression: "HEAD:README.md") is case-sensitive. Use REST fallback for repos where GraphQL returns null (handles readme.md, Readme.md, etc.)

Contributor details: batch 50 users per query (same pattern as #8)

query BatchContributorDetails {
  u0: user(login: "alice") { name bio location company email }
  u1: user(login: "bob")   { name bio location company email }
  ...
}

• Reduces ~100,000+ REST calls → ~2,000 GraphQL requests (~50× fewer)
• Exact same batching pattern already implemented for user/org repo fetching in perf: replace sequential REST calls with GraphQL batching for user and org repo fetching #8

Contributor list: keep as REST

• GET /repos/{owner}/{repo}/contributors lists git commit authors
• GraphQL mentionableUsers / collaborators are not equivalent
• No change here

Impact summary

Step	Current REST calls (UCSC)	With GraphQL batching	Speedup
README + releases	~50,000	~500	~100×
Contributor details	~125,000+	~2,000	~50×
Contributor list	~25,000	~25,000 (unchanged)	—
Total	~200,000	~27,500	~7× overall

Implementation plan

1. Add _build_repo_extras_batch_query(repos) to repo_scraping_utils.py — batches 50 (owner, name) pairs per query, fetches README text + release asset download counts
2. Add _build_contributor_details_batch_query(logins) — batches 50 contributor logins per query (same pattern as existing user batching)
3. Update get_features_data() in get_repo_extras.py to use GraphQL batch path with REST fallback for README case-sensitivity misses
4. Update get_contributor_data() in get_contributors.py to use GraphQL batch path for contributor detail fetching; keep REST for contributor list

Notes

• No new dependencies — requests already used; GraphQL helper graphql_request() added in perf: replace sequential REST calls with GraphQL batching for user and org repo fetching #8
• Backwards-compatible: same DB schema, same downstream behaviour
• REST fallback retained throughout for partial failures

Related

• perf: replace sequential REST calls with GraphQL batching for user and org repo fetching #8 — GraphQL batching for user/org repo fetching (already merged/in-progress)
• repo-pulse integration: epic: repofinder ↔ repo-pulse integration arun-gupta/repo-pulse#525