Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -306,17 +306,17 @@ A wine cellar is a storage facility, but it's also a facility for serving.
ml-cellar clone {your_ml_registry}
```

- Materialize all files.
- "Materialize" means downloading the actual file contents from Git LFS storage.
- Download all files.
- This command download the actual file contents from Git LFS storage.
Comment thread
scepter914 marked this conversation as resolved.
Outdated

```sh
ml-cellar materialize-all
ml-cellar download-all
```

- Materialize a specific directory or file.
- Download a specific directory or file.

```sh
ml-cellar materialize {path to a directory or a file}
ml-cellar download {path to a directory or a file}
```

## Reference
Expand Down
2 changes: 1 addition & 1 deletion src/bin/ml-cellar/clone.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ use std::process::{Command, Stdio};
/// This function clones a Git repository with `GIT_LFS_SKIP_SMUDGE=1` environment variable set,
/// which means Git LFS pointers are cloned instead of the actual large files.
/// This is useful for creating a lightweight clone of a model registry where you can
/// selectively materialize only the files you need later.
/// selectively download (materialize) only the files you need later.
///
/// # Arguments
///
Expand Down
83 changes: 83 additions & 0 deletions src/bin/ml-cellar/download.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
use std::path::PathBuf;
use std::process::{Command, Stdio};

/// Materializes (downloads) all Git LFS files in the repository.
/// This function executes `git lfs pull` to download the actual contents of all
/// Git LFS files. Use this when you need to work with all models and large files locally.
/// After cloning a repository with `GIT_LFS_SKIP_SMUDGE=1`, this command converts
/// all LFS pointers to their actual file contents.
///
/// # Panics
///
/// Exits the process with code 1 if:
/// - Git or Git LFS is not installed
/// - The `git lfs pull` command fails
///
pub fn download_all() {
let status = Command::new("git")
.arg("lfs")
.arg("pull")
.stdin(Stdio::null())
.stdout(Stdio::null())
.stderr(Stdio::null())
.status();

match status {
Ok(s) if s.success() => {
log::info!("git lfs pull done");
}
Ok(s) => {
log::error!("git lfs pull failed: {s}\nPlease check by git lfs pull command manually.");
std::process::exit(1);
}
Err(e) => {
log::error!(
"failed to execute git: {e}\nPlease check by git lfs pull command manually."
);
std::process::exit(1);
}
}
}

/// Materializes (downloads) a specific Git LFS file or directory.
/// This function executes `git lfs pull -I <path>` to selectively download
/// the actual contents of Git LFS files matching the specified path pattern.
/// Use this for selective downloading when you only need specific models or files.
///
/// # Arguments
///
/// - `path` - Path to a file or directory to materialize. Can include wildcards.
///
/// # Panics
///
/// Exits the process with code 1 if:
/// - Git or Git LFS is not installed
/// - The `git lfs pull -I` command fails
///
pub fn download(path: PathBuf) {
let status = Command::new("git")
.arg("lfs")
.arg("pull")
.arg("-I")
.arg(path.to_str().unwrap())
.stdin(Stdio::null())
.stdout(Stdio::null())
.stderr(Stdio::null())
.status();

match status {
Ok(s) if s.success() => {
log::info!("git lfs pull -I {} done", path.display());
}
Ok(s) => {
log::error!("git lfs pull failed: {s}\nPlease check by git lfs pull command manually.");
std::process::exit(1);
}
Err(e) => {
log::error!(
"failed to execute git: {e}\nPlease check by git lfs pull command manually."
);
std::process::exit(1);
}
}
}
18 changes: 9 additions & 9 deletions src/bin/ml-cellar/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ use std::path::PathBuf;
mod check;
mod clone;
mod docs;
mod download;
mod init;
mod materialize;
mod push;
mod rack;

Expand Down Expand Up @@ -58,16 +58,16 @@ enum Commands {
/// Path to the artifacts
path: PathBuf,
},
/// Materialize (download) all files managed by Git LFS
/// Download (materialize) all files managed by Git LFS
///
/// Downloads the actual contents of all Git LFS files in the repository.
/// This is useful when you need to work with all models locally.
MaterializeAll {},
/// Materialize (download) a specific file or directory managed by Git LFS
DownloadAll {},
/// Download (materialize) a specific file or directory managed by Git LFS
///
/// Downloads the actual contents of the specified Git LFS file or all files in a directory.
/// Use this for selective downloading when you only need specific models.
Materialize {
Download {
/// Path to the file
path: PathBuf,
},
Expand Down Expand Up @@ -157,11 +157,11 @@ fn main() {
Commands::Check { path } => {
check::check(path);
}
Commands::MaterializeAll {} => {
materialize::materialize_all();
Commands::DownloadAll {} => {
download::download_all();
}
Commands::Materialize { path } => {
materialize::materialize(path);
Commands::Download { path } => {
download::download(path);
}
Commands::Push { repository, commit } => {
push::push(&repository, &commit);
Expand Down
6 changes: 3 additions & 3 deletions src/bin/ml-cellar/materialize.rs
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
use std::path::PathBuf;
use std::process::{Command, Stdio};

/// Materializes (downloads) all Git LFS files in the repository.
/// Downloads (materialize) all Git LFS files in the repository.
/// This function executes `git lfs pull` to download the actual contents of all
/// Git LFS files. Use this when you need to work with all models and large files locally.
/// After cloning a repository with `GIT_LFS_SKIP_SMUDGE=1`, this command converts
Expand Down Expand Up @@ -39,14 +39,14 @@ pub fn materialize_all() {
}
}

/// Materializes (downloads) a specific Git LFS file or directory.
/// Downloads (materializes) a specific Git LFS file or directory.
/// This function executes `git lfs pull -I <path>` to selectively download
/// the actual contents of Git LFS files matching the specified path pattern.
/// Use this for selective downloading when you only need specific models or files.
///
/// # Arguments
///
/// - `path` - Path to a file or directory to materialize. Can include wildcards.
/// - `path` - Path to a file or directory to download. Can include wildcards.
///
/// # Panics
///
Expand Down