Factor executor code+history from marin-community/marin by ryan-williams · Pull Request #3 · marin-community/thalas

ryan-williams · 2025-08-29T03:32:37Z

Update: superseded by #4

✅ Use filter-marin-executor.sh to extract executor code
- Run git-filter-repo on a Marin clone:
  - Create filtered history (just executor and related files)
  - In PR-merge-commit subjects, rewrite #XXX to marin-community/marin#XXX
- Create empty initial commit f27eaad
- Rebase filtered history on it
✅ Add commits:
- 3b2ccdd:
  - Add project files whose history wasn't worth replaying (e.g. README.md)
  - Rename src/{marin,thalas}/, update imports
- 812031d: skip a macOS-failing test
✅ Push empty initial commit (f27eaad) to main
⏳ This PR: rewritten history + subsequent changes → main

See gist for scripts and more info.

# Conflicts: # tests/test_executor.py

# Conflicts: # .github/workflows/quickstart-test.yaml

…imPajama build (marin-community/marin#447) * wip * reduce the number of "shard groups" to something manageable, makes SlimPajama build * lint

* Local Fix * Local Fix

…arin#1364) * Initial Gemstone Scraping * Not Quite Working, But getting there * Only run evals of the cooled down WSD models (Too Many otherwise lol) * Final Version * International Corpus of English is Private * This just can't be dry run due to config loading

…running again (marin-community/marin#1442) * Refactor executor to run steps directly * working on apply autoscaler patch to ray 2.45 * fix deps for uv * tests? * wip * ok it runs! * fix build * stupid ray * oops * sigh * missed the lock * grr * grr2 * more uv run * no space?!? * tpu tests at least? * wtf du * more du * am faf * again * this? * grr * pr comments

…1479) * actually schedule statusactor on the headnode * Update marin/utilities/ray_utils.py Co-authored-by: William Held <Wbh230@nyu.edu> --------- Co-authored-by: William Held <Wbh230@nyu.edu>

…nity/marin#1490) * avoid race condition in executor * oops

…ty/marin#1453) * Use network_endpoints for expected TPU workers * Refactor executor to run steps directly * working on apply autoscaler patch to ray 2.45 * fix deps for uv * tests? * wip * ok it runs! * fix build * Refine TPU monitor * stupid ray * oops * sigh * missed the lock * grr * grr2 * more uv run * no space?!? * tpu tests at least? * wtf du * more du * am faf * again * this? * grr * background thread for monitor * simple main to try it out * auto-create * update docker and some deps * Add entrypoint resource options and TPU support to ray run * east 5, update clusterwqa * src layout? * src layout * tweaks to east5a cluster, enw image * make dockerfile work with uv and src layout * uv is causing problems * make uv want to install cpu torch (i think?) * increase max concurrent connections for setting up TPU nodes to 64 * wip * update cluster configs with latest image, hopefully faster tpu setup * wip * uv lock * wip * wip * remove tpu monitor for now * lock * no locked * sigh * i hate python * i hate python * lock * mkdocs * sigh * hrm, seems not ideal

src/thalas/utilities/json_encoder.py

dlwh

lgtm. we hsould fix licenses but I'm having codex do a thing for Marin and we can port here

dlwh · 2025-08-29T18:51:48Z

LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License


given we're forking a chunk of Marin we should probably be sure to put Stanford on the copyright of Marin and make sure it's credited here.

Good point, did my best to mimic/adapt Marin's copyright and AUTHORS.md in #4 (specifically 112e5f0). lmk (here or there) if that seems OK?

src/thalas/utilities/json_encoder.py

tests/test_executor.py

eric-czech

LGTM other than figuring out what to do with #3 (comment) and #3 (comment)

ryan-williams

I've improved the history-rewriting in #4, which I propose we shift focus to. I've linked this PR's 3 open threads over there, happy to continue discussing them here or there.

I responded on the JSON ser/de thread here/below; I partly made a new PR because force-pushing here would orphan that thread, which has good discussion.

src/thalas/utilities/json_encoder.py

alxmrs

I started reviewing this yesterday and then got side tracked. Here's what I've noticed so far and I'll continue the review now.

.pre-commit-config.yaml

README.md

alxmrs · 2025-09-10T17:08:00Z

README.md

+```bash
+git clone https://github.com/Open-Athena/thalas.git
+cd thalas
+uv venv --python 3.11


TIL. I always use uv sync + the .python-version file -- this is nice to test multiple versions.

Interesting, yea I'm fumbling through how best to manage multiple venvs in a project, in uv world. I've used Pyenv for many years, which uses .python-version in a way that conflicts with uv. Open to suggestions.

I'm surprised to hear that .python-version conflicts with uv! I use that feature all the time -- it seems to work for me. Maybe I misunderstand how you're using it? IIRC, when you uv init a new project it generates .python-version for you.

alxmrs · 2025-09-10T17:09:25Z

README.md

+uv venv --python 3.11
+source .venv/bin/activate
+uv pip install -e .[dev]
+pre-commit install


optional: you could use uvx to install the pre-commit hooks (without specifically having to install pre-commit locally):

Suggested change

pre-commit install

uvx pre-commit install

Interesting, afaict it's better to have the version pinned/managed in pyproject.toml, otherwise we're either not pinning, or have to specify version during each invocation?

src/thalas/execution/executor.py

pyproject.toml

src/thalas/execution/executor.py

alxmrs

LGTM. It's nice we can see all these sources in one PR.

ryan-williams

I believe I've addressed all comments over in #4, will close this now, but feel free to respond to existing threads here if it's easier and further discussion is warranted.

README.md

ryan-williams · 2025-09-15T15:16:51Z

README.md

+```bash
+git clone https://github.com/Open-Athena/thalas.git
+cd thalas
+uv venv --python 3.11


Interesting, yea I'm fumbling through how best to manage multiple venvs in a project, in uv world. I've used Pyenv for many years, which uses .python-version in a way that conflicts with uv. Open to suggestions.

pyproject.toml

ryan-williams · 2025-09-15T15:25:43Z

README.md

+uv venv --python 3.11
+source .venv/bin/activate
+uv pip install -e .[dev]
+pre-commit install


Interesting, afaict it's better to have the version pinned/managed in pyproject.toml, otherwise we're either not pinning, or have to specify version during each invocation?

.pre-commit-config.yaml

ryan-williams · 2025-09-15T16:02:51Z

LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License


Good point, did my best to mimic/adapt Marin's copyright and AUTHORS.md in #4 (specifically 112e5f0). lmk (here or there) if that seems OK?

ryan-williams

Addressing a couple more comments

src/thalas/execution/executor.py

tests/test_executor.py

abhinavg4 and others added 30 commits October 6, 2024 23:14

Merge remote-tracking branch 'origin/force_run' into force_run

2eccfcb

# Conflicts: # tests/test_executor.py

Make it work with custom types

b3ce1f1

Remove force_ran

c98453d

Adding force run (marin-community/marin#385)

48ba79d

Change extension of executor info files

4ad3b45

Adding warning

57d6559

Comments from David and Percy

d0c970c

Merge remote-tracking branch 'origin/main' into training_quickstart

b293908

# Conflicts: # .github/workflows/quickstart-test.yaml

Adding experiment files (marin-community/marin#375)

5f3f53c

use whoami instead of USER

4b0c537

formatting

686bb6c

fix tests, revamped Ray benchmarker

64e93d4

keep full username

5574935

lint fix

bf42a39

Merge branch 'main' into pliang-fix-tests

bdeeb45

small change to tests, revamped script to benchmark Ray (marin-commun…

c793bc1

…ity/marin#406)

factor out datasets and models, run baselines

35f388e

create a PR to train baselines (marin-community/marin#423)

790ff5c

Adding force run failed

dcb5b23

Misc tweaks/fixes (marin-community/marin#443)

f9288ea

Reduce the number of "shard groups" to something manageable, makes Sl…

47f7355

…imPajama build (marin-community/marin#447) * wip * reduce the number of "shard groups" to something manageable, makes SlimPajama build * lint

general cleanup of experiments

b9c8a32

general cleanup of experiments (marin-community/marin#450)

67e343e

render experiment page

f00c0dc

update

042e588

use material UI, make descriptions work

be697b8

format

c53711c

set defaults

ed61739

address PR comments

9263207

make ExperimentPage (marin-community/marin#454)

4a84b58

Helw150 and others added 10 commits July 1, 2025 12:07

Local Fix (marin-community/marin#1411)

532afe7

* Local Fix * Local Fix

Comment out uv run due to OOM (marin-community/marin#1458)

ea2197b

actually schedule statusactor on the headnode (marin-community/marin#…

a5fea0b

…1479) * actually schedule statusactor on the headnode * Update marin/utilities/ray_utils.py Co-authored-by: William Held <Wbh230@nyu.edu> --------- Co-authored-by: William Held <Wbh230@nyu.edu>

write_infos before steps are launched (for crashes, etc) (marin-commu…

dced9ad

…nity/marin#1490) * avoid race condition in executor * oops

Refactor executor to run steps directly (marin-community/marin#1406)

4f3b09d

Rename to thalas, add other project files

119ddf8

skip test_status_actor on macOS

bb40edf

ryan-williams force-pushed the rw/ps2.3 branch from 812031d to bb40edf Compare August 29, 2025 03:34

This was referenced Aug 29, 2025

Configure project #1

Closed

Initial Marin Executor extraction #2

Closed

ryan-williams requested review from Copilot and eric-czech and removed request for Copilot August 29, 2025 03:46

eric-czech reviewed Aug 29, 2025

View reviewed changes

src/thalas/utilities/json_encoder.py Show resolved Hide resolved

dlwh approved these changes Aug 29, 2025

View reviewed changes

eric-czech reviewed Sep 2, 2025

View reviewed changes

tests/test_executor.py Show resolved Hide resolved

eric-czech reviewed Sep 2, 2025

View reviewed changes

ryan-williams mentioned this pull request Sep 8, 2025

Factor executor code+history from marin-community/marin #4

Merged

ryan-williams commented Sep 8, 2025

View reviewed changes

src/thalas/utilities/json_encoder.py Show resolved Hide resolved

alxmrs reviewed Sep 11, 2025

View reviewed changes

alxmrs approved these changes Sep 11, 2025

View reviewed changes

eric-czech mentioned this pull request Sep 11, 2025

Support array and dtype json encoding without jax dependency #5

Open

ryan-williams mentioned this pull request Sep 15, 2025

Replace fsspec with obstore or obspec #7

Open

ryan-williams commented Sep 15, 2025

View reviewed changes

src/thalas/execution/executor.py Show resolved Hide resolved

src/thalas/execution/executor.py Show resolved Hide resolved

tests/test_executor.py Show resolved Hide resolved

ryan-williams closed this Sep 15, 2025

ryan-williams mentioned this pull request Sep 15, 2025

Multi-job support (using job start/end hooks), optional CloudWatch logging, demo workflows, module/repo rename Open-Athena/ec2-gha#2

Closed

Conversation

ryan-williams commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Update: superseded by #4

Uh oh!

Uh oh!

dlwh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

eric-czech left a comment

Choose a reason for hiding this comment

Uh oh!

ryan-williams left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alxmrs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alxmrs left a comment

Choose a reason for hiding this comment

Uh oh!

ryan-williams left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ryan-williams left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

14 participants

ryan-williams commented Aug 29, 2025 •

edited

Loading

ryan-williams left a comment •

edited

Loading