Skip to content

Migrate gsutil usage to gcloud storage #4308

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gericdong merged 9 commits into from
Dec 24, 2025
Merged

Migrate gsutil usage to gcloud storage #4308

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
2a8dfd2
Migrate gsutil usage to gcloud storage
googlyrahman Oct 16, 2025
0fb7624
changes for 4308
gurusai-voleti Dec 12, 2025
e2a83ab
added = in command
gurusai-voleti Dec 12, 2025
3a2c468
Apply automated linter fixes
gurusai-voleti Dec 16, 2025
eecc519
Merge pull request #41 from gurusai-voleti/br_4308
googlyrahman Dec 17, 2025
1d77a0f
removed model_garden folder changes
gurusai-voleti Dec 22, 2025
3292f95
Merge pull request #74 from gurusai-voleti/br_4308
gurusai-voleti Dec 22, 2025
750f261
removed model_garden folder changes
gurusai-voleti Dec 22, 2025
656a45e
Merge pull request #76 from gurusai-voleti/br_4308
gurusai-voleti Dec 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 5 additions & 10 deletions notebooks/community/ml_ops/stage1/get_started_with_data_labeling.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -491,8 +491,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_URI"
]
"! gcloud storage buckets create --location=$REGION $BUCKET_URI" ]
},
{
"cell_type": "markdown",
Expand All @@ -511,8 +510,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_URI"
]
"! gcloud storage ls --all-versions --long $BUCKET_URI" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -715,8 +713,7 @@
"print(IMPORT_FILE)\n",
"\n",
"# printing content of uploaded file\n",
"! gsutil cat $IMPORT_FILE"
]
"! gcloud storage cat $IMPORT_FILE" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -838,8 +835,7 @@
"outputs": [],
"source": [
"# create placeholder file for instructions for data labeling\n",
"! echo \"this is instruction\" >> instruction.txt | gsutil cp instruction.txt $BUCKET_URI"
]
"! echo \"this is instruction\" >> instruction.txt | gcloud storage cp instruction.txt $BUCKET_URI" ]
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This command is inefficient and the use of the pipe | is likely incorrect. It creates a local file instruction.txt and then attempts to upload it, but the pipe | sends the (empty) stdout of the echo >> command to gcloud storage cp, which is not the intended behavior. A more idiomatic and efficient approach is to pipe the output of echo directly to gcloud storage cp using - for stdin. This avoids creating an unnecessary local file.

! echo "this is instruction" | gcloud storage cp - "$BUCKET_URI/instruction.txt"

Copy link
Contributor

@gurusai-voleti gurusai-voleti Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Resolved] fix not needed

},
{
"cell_type": "code",
Expand Down Expand Up @@ -995,8 +991,7 @@
"\n",
"# Delete the bucket created\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
" ! gsutil rm -r $BUCKET_URI"
]
" ! gcloud storage rm --recursive $BUCKET_URI" ]
}
],
"metadata": {
Expand Down
12 changes: 4 additions & 8 deletions notebooks/community/ml_ops/stage2/get_started_with_cmek_training.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -449,8 +449,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_URI"
]
"! gcloud storage buckets create --location=$REGION $BUCKET_URI" ]
},
{
"cell_type": "markdown",
Expand All @@ -469,8 +468,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_URI"
]
"! gcloud storage ls --all-versions --long $BUCKET_URI" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -926,8 +924,7 @@
},
"outputs": [],
"source": [
"test_item = !gsutil cat $IMPORT_FILE | head -n1\n",
"test_item, test_label = str(test_item[0]).split(\",\")\n",
"test_item = !gcloud storage cat $IMPORT_FILE | head -n1\n", "test_item, test_label = str(test_item[0]).split(\",\")\n",
"\n",
"print(test_item, test_label)"
]
Expand Down Expand Up @@ -1015,8 +1012,7 @@
"delete_bucket = False\n",
"\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
" ! gsutil rm -r $BUCKET_URI"
]
" ! gcloud storage rm --recursive $BUCKET_URI" ]
},
{
"cell_type": "markdown",
Expand Down
26 changes: 10 additions & 16 deletions notebooks/community/model_garden/model_garden_jax_paligemma_finetuning.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,11 +163,9 @@
"if BUCKET_URI is None or BUCKET_URI.strip() == \"\" or BUCKET_URI == \"gs://\":\n",
" BUCKET_URI = f\"gs://{PROJECT_ID}-tmp-{now}-{str(uuid.uuid4())[:4]}\"\n",
" BUCKET_NAME = \"/\".join(BUCKET_URI.split(\"/\")[:3])\n",
" ! gsutil mb -l {REGION} {BUCKET_URI}\n",
"else:\n",
" ! gcloud storage buckets create --location={REGION} {BUCKET_URI}\n", "else:\n",
" assert BUCKET_URI.startswith(\"gs://\"), \"BUCKET_URI must start with `gs://`.\"\n",
" shell_output = ! gsutil ls -Lb {BUCKET_NAME} | grep \"Location constraint:\" | sed \"s/Location constraint://\"\n",
" bucket_region = shell_output[0].strip().lower()\n",
" shell_output = ! gcloud storage ls --full --buckets {BUCKET_NAME} | grep \"Location constraint:\" | sed \"s/Location constraint://\"\n", " bucket_region = shell_output[0].strip().lower()\n",
" if bucket_region != REGION:\n",
" raise ValueError(\n",
" \"Bucket region %s is different from notebook region %s\"\n",
Expand All @@ -191,8 +189,9 @@
"\n",
"\n",
"# Provision permissions to the SERVICE_ACCOUNT with the GCS bucket\n",
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.admin $BUCKET_NAME\n",
"\n",
"# Note: Migrating scripts using gsutil iam ch is more complex than get or set. You need to replace the single iam ch command with a series of gcloud storage bucket add-iam-policy-binding and/or gcloud storage bucket remove-iam-policy-binding commands, or replicate the read-modify-write loop.\n",
"# Note: gsutil iam ch does not support modifying IAM policies that contain conditions. gcloud storage commands do support conditions.\n",
"! gcloud storage buckets add-iam-policy-binding $BUCKET_NAME --member=\"serviceAccount:{SERVICE_ACCOUNT}\" --role=\"roles/storage.admin\"\n", "\n",
"! gcloud config set project $PROJECT_ID\n",
"! gcloud projects add-iam-policy-binding --no-user-output-enabled {PROJECT_ID} --member=serviceAccount:{SERVICE_ACCOUNT} --role=\"roles/storage.admin\"\n",
"! gcloud projects add-iam-policy-binding --no-user-output-enabled {PROJECT_ID} --member=serviceAccount:{SERVICE_ACCOUNT} --role=\"roles/aiplatform.user\"\n",
Expand Down Expand Up @@ -237,11 +236,9 @@
" MODEL_BUCKET,\n",
")\n",
"\n",
"! gsutil -m cp -R $VERTEX_AI_MODEL_GARDEN_PALIGEMMA/* $MODEL_BUCKET\n",
"\n",
"! gcloud storage cp --recursive $VERTEX_AI_MODEL_GARDEN_PALIGEMMA/* $MODEL_BUCKET\n", "\n",
"assert (\n",
" os.system(f\"gsutil ls {MODEL_BUCKET}\") == 0\n",
"), f\"MODEL_BUCKET does not exist: {MODEL_BUCKET}.\"\n",
" os.system(f\"gcloud storage ls {MODEL_BUCKET}\") == 0\n", "), f\"MODEL_BUCKET does not exist: {MODEL_BUCKET}.\"\n",
"model_path_prefix = MODEL_BUCKET"
]
},
Expand Down Expand Up @@ -461,8 +458,7 @@
"temp_dir = tempfile.TemporaryDirectory()\n",
"local_metrics_path = os.path.join(temp_dir.name, metrics_file_name)\n",
"\n",
"! gsutil cp $metrics_path $local_metrics_path\n",
"\n",
"! gcloud storage cp $metrics_path $local_metrics_path\n", "\n",
"steps = []\n",
"training_losses = []\n",
"with open(local_metrics_path, \"r\") as f:\n",
Expand Down Expand Up @@ -507,8 +503,7 @@
"\n",
"local_last_checkpoint_path = os.path.join(temp_dir.name, last_checkpoint_file_name)\n",
"\n",
"! gsutil cp $last_checkpoint_path $local_last_checkpoint_path\n",
"\n",
"! gcloud storage cp $last_checkpoint_path $local_last_checkpoint_path\n", "\n",
"with open(local_last_checkpoint_path, \"r\") as f:\n",
" final_checkpoint_name = \"checkpoint.bv-\" + f.read()\n",
" checkpoint_path = os.path.join(finetune_output_dir, final_checkpoint_name)\n",
Expand Down Expand Up @@ -693,8 +688,7 @@
"\n",
"delete_bucket = False # @param {type:\"boolean\"}\n",
"if delete_bucket:\n",
" ! gsutil -m rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
Expand Down
18 changes: 6 additions & 12 deletions notebooks/community/sdk/sdk_automl_image_classification_online.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -473,8 +473,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location=$REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
Expand All @@ -493,8 +492,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -602,12 +600,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -821,8 +817,7 @@
},
"outputs": [],
"source": [
"test_item = !gsutil cat $IMPORT_FILE | head -n1\n",
"if len(str(test_item[0]).split(\",\")) == 3:\n",
"test_item = !gcloud storage cat $IMPORT_FILE | head -n1\n", "if len(str(test_item[0]).split(\",\")) == 3:\n",
" _, test_item, test_label = str(test_item[0]).split(\",\")\n",
"else:\n",
" test_item, test_label = str(test_item[0]).split(\",\")\n",
Expand Down Expand Up @@ -990,8 +985,7 @@
" print(e)\n",
"\n",
" if \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
Expand Down
18 changes: 6 additions & 12 deletions notebooks/community/sdk/sdk_automl_tabular_binary_classification_online.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -473,8 +473,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location $REGION $BUCKET_NAME" ]
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

For consistency with other gcloud commands in this pull request and for better readability, it's recommended to use an equals sign for assigning values to flags. While gcloud often accepts a space, using = is more explicit and less prone to parsing errors.

! gcloud storage buckets create --location=$REGION $BUCKET_NAME

Copy link
Contributor

@gurusai-voleti gurusai-voleti Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Resolved] fixed inline with others

},
{
"cell_type": "markdown",
Expand All @@ -493,8 +492,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -597,14 +595,11 @@
},
"outputs": [],
"source": [
"count = ! gsutil cat $IMPORT_FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $IMPORT_FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $IMPORT_FILE | head\n",
"\n",
"heading = ! gsutil cat $IMPORT_FILE | head -n1\n",
"label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"! gcloud storage cat $IMPORT_FILE | head\n", "\n",
"heading = ! gcloud storage cat $IMPORT_FILE | head -n1\n", "label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"print(\"Label Column Name\", label_column)\n",
"if label_column is None:\n",
" raise Exception(\"label column missing\")"
Expand Down Expand Up @@ -996,8 +991,7 @@
" print(e)\n",
"\n",
" if \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
Expand Down
12 changes: 4 additions & 8 deletions notebooks/community/vertex_endpoints/find_ideal_machine_type/find_ideal_machine_type.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -355,8 +355,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -p $PROJECT_ID -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --project=$PROJECT_ID --location=$REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
Expand All @@ -375,8 +374,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -596,8 +594,7 @@
},
"outputs": [],
"source": [
"!sudo gsutil cp -r ./bert_sentence_embedding/00001/* $GCS_URI/1/"
]
"!sudo gcloud storage cp --recursive ./bert_sentence_embedding/00001/* $GCS_URI/1/" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -1045,8 +1042,7 @@
},
"outputs": [],
"source": [
"!gsutil rm -r $GCS_URI/*"
]
"!gcloud storage rm --recursive $GCS_URI/*" ]
},
{
"cell_type": "markdown",
Expand Down
6 changes: 2 additions & 4 deletions notebooks/official/custom/custom-tabular-bq-managed-dataset.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -291,8 +291,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $LOCATION -p $PROJECT_ID $BUCKET_URI"
]
"! gcloud storage buckets create --location=$LOCATION --project=$PROJECT_ID $BUCKET_URI" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -911,8 +910,7 @@
"delete_bucket = True\n",
"\n",
"if delete_bucket:\n",
" ! gsutil rm -r $BUCKET_URI"
]
" ! gcloud storage rm --recursive $BUCKET_URI" ]
}
],
"metadata": {
Expand Down
6 changes: 2 additions & 4 deletions notebooks/official/custom/custom_training_container_and_model_registry.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -283,8 +283,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $LOCATION -p $PROJECT_ID $BUCKET_URI"
]
"! gcloud storage buckets create --location=$LOCATION --project=$PROJECT_ID $BUCKET_URI" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -932,8 +931,7 @@
"# Delete the Cloud Storage bucket\n",
"delete_bucket = True\n",
"if delete_bucket:\n",
" ! gsutil -m rm -r $BUCKET_URI\n",
"\n",
" ! gcloud storage rm --recursive $BUCKET_URI\n", "\n",
"# Delete application directory\n",
"!rm -rf $APPLICATION_DIR"
]
Expand Down
6 changes: 2 additions & 4 deletions notebooks/official/model_registry/get_started_with_model_registry.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -303,8 +303,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $LOCATION $BUCKET_URI"
]
"! gcloud storage buckets create --location=$LOCATION $BUCKET_URI" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -996,8 +995,7 @@
"delete_bucket = False\n",
"\n",
"if delete_bucket:\n",
" ! gsutil rm -rf {BUCKET_URI}\n",
"\n",
" ! gcloud storage rm --recursive --continue-on-error {BUCKET_URI}\n", "\n",
"# Undeploy the models and delete the endpoint resources\n",
"try:\n",
" endpoint.undeploy_all()\n",
Expand Down
11 changes: 4 additions & 7 deletions notebooks/official/reduction_server/pytorch_distributed_training_reduction_server.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -291,8 +291,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l {LOCATION} -p {PROJECT_ID} {BUCKET_URI}"
]
"! gcloud storage buckets create --location={LOCATION} --project={PROJECT_ID} {BUCKET_URI}" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -720,9 +719,8 @@
"source": [
"! cd {PYTHON_PACKAGE_APPLICATION_DIR} && python3 setup.py sdist --formats=gztar\n",
"\n",
"! gsutil cp {source_package_file_name} {python_package_gcs_uri}\n",
"\n",
"! gsutil ls -l {python_package_gcs_uri}"
"! gcloud storage cp {source_package_file_name} {python_package_gcs_uri}\n", "\n",
"! gcloud storage ls --long {python_package_gcs_uri}"
]
},
{
Expand Down Expand Up @@ -933,8 +931,7 @@
" print(e)\n",
"\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
" ! gsutil rm -r $BUCKET_URI"
]
" ! gcloud storage rm --recursive $BUCKET_URI" ]
}
],
"metadata": {
Expand Down
Loading