feat(tide): shard large queries to avoid GitHub resource limits - #821
Draft
Prucek wants to merge 4 commits into
Draft
feat(tide): shard large queries to avoid GitHub resource limits#821Prucek wants to merge 4 commits into
Prucek wants to merge 4 commits into
Conversation
✅ Deploy Preview for k8s-prow ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
kubernetes-prow
Bot
added
cncf-cla: yes
Contributor
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: Prucek The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
kubernetes-prow
Bot
added
the
area/tide
…limits The openshift org has 521 repos, producing a ~20KB search query that consistently triggers GitHub's "Resource limits for this query exceeded" error, returning 0 results. This means PRs in that org never receive Tide status updates via the search path. Split queries with more than 100 repo: terms into multiple shards (e.g. "openshift#0", "openshift#1", ...), each containing up to 100 repos. The non-repo prefix (is:pr state:open sort:updated-asc etc) is preserved in each shard. Shards run concurrently alongside other org queries. The per-org latestPR checkpoint is advanced to the minimum across all shards to ensure no PRs are skipped. Only the status controller is affected; the sync controller uses different query construction (orgRepoQueryStrings) which is unchanged.
Prucek
force-pushed
the
tide-shard-large-queries
branch
from
August 5, 2026 09:45
1b60edf to
44a66b4
Compare
kubernetes-prow
Bot
added
size/XL
Contributor
|
@Prucek: The following tests failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Prucek
marked this pull request as draft
August 5, 2026 12:48
kubernetes-prow
Bot
added
the
do-not-merge/work-in-progress
Tide runs one GitHub GraphQL search per configured TideQuery per org shard. At scale (~480 shards) the unbounded goroutine fan-out overwhelms ghproxy, causing widespread client timeouts. Add a new config field max_query_concurrency (default 0 = unlimited) that limits concurrent search queries via errgroup.SetLimit in both the sync and status controllers. Also promote the "failed to query" log from Debug to Error so query outages are visible at default log level.
Add six new Prometheus metrics to give operators visibility into Tide's GitHub query layer: - tide_query_duration_seconds (histogram, controller × result) - tide_query_prs_returned (histogram, controller) - tide_query_errors_total (counter, controller × query_id × org_shard × error_class) - tide_query_partial_results_total (counter, controller × query_id × org_shard) - tide_sync_query_shards (gauge, controller × result) - tide_pool_completeness_ratio (gauge, controller) Also add classifyQueryError to break errors into actionable classes (context_deadline, client_timeout, resource_limits, server_error, etc.) instead of a binary success/error signal.
Temporary default until the config field is set in production. This commit should be dropped before merging upstream.
Prucek
force-pushed
the
tide-shard-large-queries
branch
from
August 7, 2026 09:46
44a66b4 to
2151f08
Compare
kubernetes-prow
Bot
added
size/L
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
or-queries into smaller batchesShardQueryandShardQueriesfunctions toconfig.TideQuerythat partition queries based onorgsandreposfields🤖 Generated with the help of AI assistance