Merge branch 'mlcommons:dev' into dev

Author: amd-arsuresh

script/get-generic-python-lib/README.md

@@ -1,5 +1,5 @@
 # README for get-generic-python-lib
-This README is automatically generated. Add custom content in [info.md](info.md). Please follow the [script execution document](https://docs.mlcommons.org/mlcflow/targets/script/execution-flow/) to understand more about the MLC script execution.
+This README is automatically generated. Create and add custom content in info.md. Please follow the [script execution document](https://docs.mlcommons.org/mlcflow/targets/script/execution-flow/) to understand more about the MLC script execution.
 
 `mlcflow` stores all local data under `$HOME/MLC` by default. So, if there is space constraint on the home directory and you have more space on say `/mnt/$USER`, you can do
 ```
@@ -51,10 +51,12 @@ mlcr get generic-python-lib
 | `--output` | Output from the script passed using the env key `MLC_OUTPUT` |  | `` |
 | `--outdirname` | The directory to store the script output |  | `cache directory ($HOME/MLC/repos/local/cache/<>) if the script is cacheable or else the current directory` |
 | `--outbasename` | The output file/folder name |  | `` |
+| `--search_folder_path` | The folder path where executables of a given script need to be searched. Search is done recursively upto 4 levels. |  | `` |
 | `--name` |  |  | `` |
 | `--extra_cache_tags` | Extra cache tags to be added to the cached entry when the script results are saved |  | `` |
 | `--skip_compile` | Skip compilation |  | `False` |
 | `--skip_run` | Skip run |  | `False` |
+| `--skip_sudo` | Skip SUDO detection |  | `False` |
 | `--accept_license` | Accept the required license requirement to run the script |  | `False` |
 | `--skip_system_deps` | Skip installing any system dependencies |  | `False` |
 | `--git_ssh` | Use SSH for git repos |  | `False` |
@@ -89,6 +91,7 @@ mlcr get generic-python-lib
 - `find_links_url.#` _(# can be substituted dynamically)_
 - `google-api-python-client`
 - `google-auth-oauthlib`
+- `google-genai`
 - `google-generativeai`
 - `groq`
 - `huggingface_hub`

script/get-generic-python-lib/meta.yaml

@@ -171,6 +171,11 @@ variations:
       MLC_GENERIC_PYTHON_PACKAGE_NAME: google-generativeai
     new_env_keys:
     - MLC_GOOGLE_GENERATIVEAI_VERSION
+  google-genai:
+    env:
+      MLC_GENERIC_PYTHON_PACKAGE_NAME: google-genai
+    new_env_keys:
+    - MLC_GOOGLE_GENAI_VERSION
   groq:
     env:
       MLC_GENERIC_PYTHON_PACKAGE_NAME: groq

script/get-huggingface-cli/customize.py

@@ -18,7 +18,7 @@ def preprocess(i):
         hf_cli = _get_hf_cli()
         env['MLC_HF_LOGIN_CMD'] = (
             f"git config --global credential.helper store && "
-            f"{hf_cli} login --token {env['MLC_HF_TOKEN']} --add-to-git-credential\n"
+            f"{hf_cli} auth login --token {env['MLC_HF_TOKEN']} --add-to-git-credential\n"
         )
     elif is_true(str(env.get('MLC_HF_DO_LOGIN'))):
         env['MLC_HF_LOGIN_CMD'] = f"""git config --global credential.helper store && huggingface-cli login

script/get-preprocessed-dataset-nuscenes/README.md

@@ -1,5 +1,5 @@
 # README for get-preprocessed-dataset-nuscenes
-This README is automatically generated. Add custom content in [info.md](info.md). Please follow the [script execution document](https://docs.mlcommons.org/mlcflow/targets/script/execution-flow/) to understand more about the MLC script execution.
+This README is automatically generated. Create and add custom content in info.md. Please follow the [script execution document](https://docs.mlcommons.org/mlcflow/targets/script/execution-flow/) to understand more about the MLC script execution.
 
 `mlcflow` stores all local data under `$HOME/MLC` by default. So, if there is space constraint on the home directory and you have more space on say `/mnt/$USER`, you can do
 ```
@@ -45,10 +45,12 @@ No script specific inputs
 | `--output` | Output from the script passed using the env key `MLC_OUTPUT` |  | `` |
 | `--outdirname` | The directory to store the script output |  | `cache directory ($HOME/MLC/repos/local/cache/<>) if the script is cacheable or else the current directory` |
 | `--outbasename` | The output file/folder name |  | `` |
+| `--search_folder_path` | The folder path where executables of a given script need to be searched. Search is done recursively upto 4 levels. |  | `` |
 | `--name` |  |  | `` |
 | `--extra_cache_tags` | Extra cache tags to be added to the cached entry when the script results are saved |  | `` |
 | `--skip_compile` | Skip compilation |  | `False` |
 | `--skip_run` | Skip run |  | `False` |
+| `--skip_sudo` | Skip SUDO detection |  | `False` |
 | `--accept_license` | Accept the required license requirement to run the script |  | `False` |
 | `--skip_system_deps` | Skip installing any system dependencies |  | `False` |
 | `--git_ssh` | Use SSH for git repos |  | `False` |
@@ -72,8 +74,8 @@ No script specific inputs
 
 ### Download-tool
 
-- `r2-downloader`
-- `rclone` (default)
+- `r2-downloader` (default)
+- `rclone`
 
 ### Run-mode
 

script/get-preprocessed-dataset-nuscenes/meta.yaml

@@ -28,8 +28,8 @@ variations:
         tags: _rclone
       dae_ac:
         tags: _rclone
-    default: true
   r2-downloader:
+    default: true
     env:
       MLC_DOWNLOAD_TOOL: r2-downloader
     add_deps_recursive:
@@ -155,6 +155,7 @@ variations:
   validation,mlc,r2-downloader:
     env:
       MLC_DOWNLOAD_URL: https://nuscenes.mlcommons-storage.org/metadata/nuscenes-dataset.uri
+      MLC_DOWNLOAD_FILENAME: nuscenes_dataset
   calibration,mlc,r2-downloader:
       MLC_DOWNLOAD_URL: https://nuscenes.mlcommons-storage.org/metadata/nuscenes-dataset.uri
 tests:
@@ -164,6 +165,5 @@ tests:
       - validation,prebuilt,rclone,mlc,dry-run
       - calibration,prebuilt,rclone,mlc,dry-run
       - validation,prebuilt,r2-downloader,mlc,dry-run
-
       - calibration,prebuilt,r2-downloader,mlc,dry-run