[ci] Publish marin-* lib wheels nightly, on tagged releases, and to a local vendor dir#4612
Merged
[ci] Publish marin-* lib wheels nightly, on tagged releases, and to a local vendor dir#4612
Conversation
Adds scripts/python_libs_package.py and .github/workflows/marin-libs-wheels.yaml
to build the seven marin-* lib wheels (marin, marin-iris, marin-fray,
marin-haliax, marin-levanter, marin-rigging, marin-zephyr) and publish them
as one GitHub Release per package, mirroring the dupekit-wheels pattern.
Three modes share the same job:
schedule (06:00 UTC daily) -> nightly: <base>.dev<YYYYMMDD>
tag marin-<pkg>-<YYYYMMDD>
+ rolling marin-<pkg>-latest pointer
+ auto-prune marin-<pkg>-<date> > 30 days old
push tag marin-libs-vX.Y.Z -> stable: <X.Y.Z>
tag marin-<pkg>-vX.Y.Z
+ rolling marin-<pkg>-stable pointer
workflow_dispatch / PR -> manual: <base>+manual.<sha>, no publish on PR
The build script patches each package's version field plus every sibling
marin-* dependency string to ==<resolved_version> so a nightly produces a
self-consistent set of wheels (consumer cannot accidentally mix a fresh
marin-levanter with a stale marin-haliax). Mutations are reverted via a
context manager that captures pre-mutation file contents exactly once per
path, leaving the working tree clean even on build failure.
Consumers add the rolling pointer URLs to their find-links, e.g.:
find-links = [
"https://github.com/marin-community/marin/releases/expanded_assets/marin-haliax-latest",
...
]
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f91628878f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…loops
vendor mode builds the seven marin-* wheels with a second-precision
timestamp version (1.4.dev<YYYYMMDDHHMMSS>) and copies them into a
user-supplied directory, replacing any prior marin-*.whl in that target.
Skips the GitHub Release publish/prune entirely. The timestamp guarantees
a freshly-built wheel beats whatever nightly was published earlier the
same day, so `uv sync` in the consumer always picks up the local copy
without cache games.
Iteration loop:
cd submodules/marin && git worktree add ../marin-fix some-fix-branch
cd ../marin-fix && # edit lib/levanter/...
uv run python scripts/python_libs_package.py --mode vendor \
--vendor ../../tiny-tpu/vendor
cd ../../tiny-tpu && uv sync && uv run python launch.py
Existing nightly/stable/manual modes are unchanged.
rjpower
changed the title
…align package versions to 0.99 Three review fixes from PR 4612: 1. Fix nightly date drift across midnight UTC (claude bot, real bug). The resolve job emits an empty version for nightly mode and both build and publish jobs called resolve_version() independently, computing the date via datetime.now() each time. A run that started at 23:59 UTC could stamp wheels with dev<YYYYMMDD> while the publish step a few minutes later created tags like marin-<pkg>-<YYYYMMDD+1>. Fix: build_wheels() now persists dist/BUILD_INFO.json with the resolved version. main() reads it on --skip-build / --publish-only paths instead of re-resolving. Workflow upload-artifact ships BUILD_INFO.json alongside the wheels so the publish job picks it up. 2. Drop dated nightly releases entirely (codex + claude bots, P2). Was keeping marin-<pkg>-<YYYYMMDD> tags + a 30-day pruner with a hard --limit 1000 that would silently miss old tags after ~5 months. Simpler: nightly mode now overwrites the rolling marin-<pkg>-latest tag in place and keeps no history. Reproducibility comes from stable tags (marin-<pkg>-vX.Y.Z + marin-<pkg>-stable). Drops prune_old_nightlies() and the --skip-prune flag entirely. 3. Align all seven packages to version 0.99 (claude bot nit). Was a mix of 0.1.0 (5 packages), 1.2 (levanter), and 1.4 (haliax). Bumping all 7 to 0.99 makes _highest_base_version() return a consistent number and lets the bot's "users see marin-iris==1.4.dev<date>" concern go away. Drops the now-unsatisfiable marin-haliax>=1.4.dev450 constraint from levanter's deps; the script's _rewrite_sibling_pins resolves it to ==<resolved> at build time anyway. Manual mode is now build-only (no publish, useful for inspecting wheels from a workflow_dispatch run). Verified locally: nightly build produces 0.99.dev20260410 wheels with cross-pins resolved to the same version, BUILD_INFO.json round-trips correctly under --publish-only --skip-publish, and the working tree stays clean after build.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds the seven marin-* lib wheels and publishes one GitHub Release per package, mirroring the dupekit-wheels pattern.
Schedule cron emits marin--YYYYMMDD plus rolling marin--latest; pushing tag marin-libs-vX.Y.Z emits marin--vX.Y.Z plus marin--stable. Build script patches each version field plus every sibling marin-* dependency to ==<resolved_version> so a nightly is internally coherent.
Nightlies older than 30 days are auto-pruned. Fourth mode --mode vendor builds with a second-precision timestamp and drops wheels into a user-supplied directory for fast worktree-driven local iteration against an experiment repo. Part of #4271.