Forward RAYCI_PYPI_MIRROR_URL into build containers - #498
Open
elliot-barn wants to merge 1 commit into
Open
Conversation
CI is moving to resolving Python packages through a caching mirror in front of PyPI, because the intermittent 502s from files.pythonhosted.org (pypi/support#11895) do not clear on retry: the client stays pinned to the same bad edge, so a cache that fetches each package version once is the fix rather than more attempts. The mirror URL differs per fleet and each one is only reachable from inside its own VPC, so it cannot be a constant in the repos that run here. It has to arrive as agent environment, and the docker plugin only passes through what this list names, so a variable set on the agent otherwise stops at the container boundary. Name only: no fleet-specific value lands in this repo, and where the variable is unset nothing changes -- consumers fall back to their own default, and from there to public PyPI. Signed-off-by: Ray CI Test <rayci@ray.io>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request adds the RAYCI_PYPI_MIRROR_URL environment variable to the list of forwarded Buildkite environment variables (buildkiteEnvs) in raycicmd/bk_pipeline.go. This allows forwarding the package index URL used by CI to resolve dependencies per fleet. There are no review comments, and we have no feedback to provide.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
RAYCI_PYPI_MIRROR_URLto the env allowlist the docker plugin receives, so a value set on the agent reaches the build container. Name only — no value, no per-pipeline logic.Why
Ray CI is moving to resolving Python packages through a caching mirror in front of PyPI. The intermittent 502s from
files.pythonhosted.org(pypi/support#11895) do not clear on retry — measured on premerge 72083, pip with--retries=50over ~8 minutes still failed withtoo many 502 error responses, while 47 of 62 job-level retries in the same build passed, because in-process retries reuse the connection and anycast keeps them on the same bad edge. A cache that fetches each(package, version)once is the fix; more attempts are not.The mirror URL differs per fleet, and each deployment is only reachable from inside its own VPC, so it can't be a constant in the repos that run here. It has to arrive as agent environment — and
dockerPluginEnvListonly forwards what this list names, so today a variable set on the agent stops at the container boundary.Impact
None where the variable is unset, which is everywhere until a fleet opts in: consumers fall back to their own default and from there to public PyPI.
go test ./raycicmd/...passes.