Skip to content

Navigation Menu

Appearance settings

View all features
- BY COMPANY SIZE
  Enterprises
  Small and medium teams
  Startups
  Nonprofits
- BY USE CASE
  App Modernization
  DevSecOps
  DevOps
  CI/CD
  View all use cases
- BY INDUSTRY
  Healthcare
  Financial services
  Manufacturing
  Government
  View all industries
View all solutions
- EXPLORE BY TOPIC
  AI
  Software Development
  DevOps
  Security
  View all topics
- EXPLORE BY TYPE
  Customer stories
  Events & webinars
  Ebooks & reports
  Business insights
  GitHub Skills
- SUPPORT & SERVICES
  Documentation
  Customer support
  Community forum
  Trust center
  Partners
View all resources
- COMMUNITY
  GitHub SponsorsFund open source developers
- PROGRAMS
  Security Lab
  Maintainer Community
  Accelerator
  GitHub Stars
  Archive Program
- REPOSITORIES
  Topics
  Trending
  Collections
- ENTERPRISE SOLUTIONS
  Enterprise platformAI-powered developer platform
- AVAILABLE ADD-ONS
  GitHub Advanced SecurityEnterprise-grade security features
  Copilot for BusinessEnterprise-grade AI features
  Premium SupportEnterprise-grade 24/7 support
Pricing

Search code, repositories, users, issues, pull requests...

Search

Clear

Search syntax tips

Provide feedback

We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Saved searches

Use saved searches to filter your results more quickly

Name

Query

To see all available qualifiers, see our documentation.

Appearance settings

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

Dismiss alert

marin-community / marin Public

Notifications You must be signed in to change notification settings
Fork 110
Star 873

Code
Issues 391
Pull requests 116
Actions
Projects
Security and quality
Insights

Additional navigation options

Code
Issues
Pull requests
Actions
Projects
Security and quality
Insights

Bootstrap `datakit` - consolidate and cleanup downloads#4142

Merged

ravwojdyla merged 56 commits intomainmarin-community/marin:mainfrom

rav-datakit-normmarin-community/marin:rav-datakit-normCopy head branch name to clipboard

Mar 26, 2026

Conversation Commits56 (56)Checks Files changed

Merged

Bootstrap `datakit` - consolidate and cleanup downloads#4142
ravwojdyla merged 56 commits intomainfrom
rav-datakit-norm

Commits

Commits on Mar 26, 2026

Stage datakit design doc
ravwojdyla
committed
Add datakit download, normalize, and tokenize modules

ravwojdyla
and
claude
committed
StepSpec: auto-prefix relative override_output_path with marin_prefix

ravwojdyla
and
claude
committed
Convert datakit/download to package and move HF download modules

ravwojdyla
and
claude
committed
Move specialized downloaders to datakit/download/ with StepSpec factories

ravwojdyla
and
claude
committed
Convert marin.download.* to backward-compat re-export shims

ravwojdyla
and
claude
committed
Extract pretraining download definitions into datakit/download/pretraining.py

ravwojdyla
and
claude
committed
Fix mock targets in download tests to use canonical module paths

ravwojdyla
and
claude
committed
Migrate all imports from marin.download.* to marin.datakit.download.*

ravwojdyla
and
claude
committed
Migrate imports to canonical paths and simplify download functions

ravwojdyla
and
claude
committed
Remove unused config dataclasses from download functions

ravwojdyla
and
claude
committed
Update datakit design doc: use Parquet instead of Vortex

ravwojdyla
and
claude
committed
Remove global HfFileSystem() instance from stream_remove_columns

ravwojdyla
and
claude
committed
Inline pretraining downloads into simple.py, delete pretraining.py

ravwojdyla
and
claude
committed
Delete old marin.download/ shim directory

ravwojdyla
and
claude
committed
Remove unused stackexchange data files

ravwojdyla
and
claude
committed
Revert "Remove unused stackexchange data files"
ravwojdyla
committed
Move upload_gcs_to_hf from datakit/download/ to utilities/

ravwojdyla
and
claude
committed
Convert ar5iv into a package with its JSON data file alongside

ravwojdyla
and
claude
committed
Delete unused filesystem transfer module

ravwojdyla
and
claude
committed
Move ar5iv logic from __init__.py to download.py within the package

ravwojdyla
and
claude
committed
Delete unused stream_remove_columns module and its test

ravwojdyla
and
claude
committed
Remove unused dclm_hq_step function

ravwojdyla
and
claude
committed
Simplify nemotron_cc_step to download_nemotron_cc_step

ravwojdyla
and
claude
committed
Rename nemotron_cc.py to nemotron_v1.py

ravwojdyla
and
claude
committed
Extract nemotron_v2 download definitions into datakit/download/nemotron_v2.py

ravwojdyla
and
claude
committed
Rename nemotron step functions for consistency

ravwojdyla
and
claude
committed
Remove unnecessary __all__ from uncheatable_eval module

ravwojdyla
and
claude
committed
Remove unused wikipedia_step function

ravwojdyla
and
claude
committed
Flatten wikipedia download to plain parameters, remove draccus CLI

ravwojdyla
and
claude
committed
Remove unused draccus CLI from huggingface download module

ravwojdyla
and
claude
committed
Remove backward-compat aliases from datakit/download/__init__.py

ravwojdyla
and
claude
committed
Remove output_path_prefix from download_hf_step

ravwojdyla
and
claude
committed
Remove unused datakit/tokenize.py module

ravwojdyla
and
claude
committed
Remove unused datakit/normalize.py module

ravwojdyla
and
claude
committed
Move tests/download/ to tests/datakit/download/

ravwojdyla
and
claude
committed
Restore tests/datakit/__init__.py and test_datakit.py

ravwojdyla
and
claude
committed
Replace nemotron downloads dict with nemotron_cc_download variable

ravwojdyla
and
claude
committed
Replace nemotron_cc_download global with a function

ravwojdyla
and
claude
committed
Fix typo in nemotron_v1 comment

ravwojdyla
and
claude
committed
Rename huggingface.py to huggingface_utils.py, update all imports

ravwojdyla
and
claude
committed
Add override_output_path to nemotron_v2 datasets

ravwojdyla
and
claude
committed
Revert huggingface_utils.py rename back to huggingface.py

ravwojdyla
and
claude
committed
Delete unused dclm_hq download and transform modules

ravwojdyla
and
claude
committed
Extract dolmino download into datakit/download/dolmino.py

ravwojdyla
and
claude
committed
Extract dolma download into datakit/download/dolma.py

ravwojdyla
and
claude
committed
Move DOLMA_OLMO_MIXTURE_WEIGHTS back to experiment file

ravwojdyla
and
claude
committed
Add download_wikipedia_step with override pointing at existing data

ravwojdyla
and
claude
committed
Simplify download_wikipedia_step and remove revision param

ravwojdyla
and
claude
committed
Wire download_wikipedia_step into exp934 as StepSpec dependency

ravwojdyla
and
claude
committed
Fix Wikipedia download override path to wikipedia-9273e1

ravwojdyla
and
claude
committed
Restore revision parameter in download_wikipedia

ravwojdyla
and
claude
committed
Make revision required in download_wikipedia_step

ravwojdyla
and
claude
committed
Wire ar5iv_step into exp934 as StepSpec dependency

ravwojdyla
and
claude
committed
Remove unused download entries from simple.py

ravwojdyla
and
claude
committed
Address PR review comments

ravwojdyla
and
claude
committed

Footer

© 2026 GitHub, Inc.

Footer navigation

Terms
Privacy
Security
Status
Community
Docs
Contact

You can’t perform that action at this time.