Skip to content

Migrate gsutil usage to gcloud storage #4321

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gericdong merged 6 commits into from
Dec 25, 2025
Merged

Migrate gsutil usage to gcloud storage #4321

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
102d0a2
Migrate gsutil usage to gcloud storage
googlyrahman Oct 16, 2025
a332732
Changes for 4321
bhandarivijay-png Dec 15, 2025
ab13c61
fix the linter issue for br 4321
bhandarivijay-png Dec 17, 2025
36d5ecc
Merge pull request #58 from bhandarivijay-png/br_4321
gurusai-voleti Dec 22, 2025
2bafa65
removed changes for model garden
bhandarivijay-png Dec 22, 2025
6ddc8a2
Merge pull request #104 from bhandarivijay-png/br_4321
gurusai-voleti Dec 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 6 additions & 12 deletions ...books/community/gapic/custom/showcase_custom_text_classification_online_exported_ds.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -426,8 +426,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location=$REGION $BUCKET_NAME" ]
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The gcloud storage buckets create command requires the bucket name to be specified as a URI (e.g., gs://my-bucket). It's crucial to ensure that $BUCKET_NAME includes the gs:// prefix. If $BUCKET_NAME does not include the gs:// prefix, the command will fail.

Consider adding the gs://$BUCKET_NAME to ensure the bucket name is correctly formatted.

! gcloud storage buckets create --location=$REGION gs://$BUCKET_NAME

Copy link
Contributor

@bhandarivijay-png bhandarivijay-png Dec 12, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gsutil mb -l $REGION $BUCKET_NAME
Creating gs://vijay-bucket11/...

gcloud storage buckets create --location=$REGION $BUCKET_NAME
Creating gs://vijay-bucket9999/...
command is correct..
gcloud storage buckets create --location=$REGION gs://$BUCKET_NAME
ERROR: (gcloud.storage.buckets.create) "gcloud storage buckets create" only accepts bucket URLs.

Hence Gemini incorrect ..

},
{
"cell_type": "markdown",
Expand All @@ -446,8 +445,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -991,12 +989,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -1153,8 +1149,7 @@
"source": [
"jsonl_index = result.exported_files[0]\n",
"\n",
"! gsutil cat $jsonl_index | head"
]
"! gcloud storage cat $jsonl_index | head" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -1286,8 +1281,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
Expand Down
19 changes: 6 additions & 13 deletions ...books/community/migration/UJ7 legacy AutoML Natural Language Text Entity Extraction.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -326,8 +326,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION gs://$BUCKET_NAME"
]
"! gcloud storage buckets create --location=$REGION gs://$BUCKET_NAME" ]
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The gcloud storage buckets create command requires the bucket name to be specified as a URI (e.g., gs://my-bucket). It's crucial to ensure that $BUCKET_NAME includes the gs:// prefix. If $BUCKET_NAME does not include the gs:// prefix, the command will fail.

Consider adding the gs:// prefix to $BUCKET_NAME to ensure the bucket name is correctly formatted.

! gcloud storage buckets create --location=$REGION gs://$BUCKET_NAME

Copy link
Contributor

@bhandarivijay-png bhandarivijay-png Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bhandarivijay@bhandarivijay:$BUCKET_NAME=my-bucket232
bhandarivijay@bhandarivijay:$ REGION=asia-southeast1
bhandarivijay@bhandarivijay:$ ! gcloud storage buckets create --location=$REGION gs://$BUCKET_NAME
Creating gs://my-bucket232/...
command is correct

},
{
"cell_type": "markdown",
Expand All @@ -346,8 +345,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al gs://$BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long gs://$BUCKET_NAME" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -476,8 +474,7 @@
},
"outputs": [],
"source": [
"! gsutil cat $IMPORT_FILE | head -n 10"
]
"! gcloud storage cat $IMPORT_FILE | head -n 10" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -1132,8 +1129,7 @@
" data = {\"id\": 0, \"text_snippet\": {\"content\": test_item}}\n",
" f.write(json.dumps(data) + \"\\n\")\n",
"\n",
"! gsutil cat $gcs_input_uri"
]
"! gcloud storage cat $gcs_input_uri" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -1280,9 +1276,7 @@
"source": [
"destination_uri = output_config[\"gcs_destination\"][\"output_uri_prefix\"][:-1]\n",
"\n",
"! gsutil ls $destination_uri/*\n",
"! gsutil cat $destination_uri/prediction*/*.jsonl"
]
"! gcloud storage ls $destination_uri/*\n", "! gcloud storage cat $destination_uri/prediction*/*.jsonl" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -1614,8 +1608,7 @@
"\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r gs://$BUCKET_NAME"
]
" ! gcloud storage rm --recursive gs://$BUCKET_NAME" ]
}
],
"metadata": {
Expand Down
12 changes: 4 additions & 8 deletions notebooks/community/ml_ops/stage2/get_started_vertex_tensorboard.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -511,8 +511,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_URI"
]
"! gcloud storage buckets create --location $REGION $BUCKET_URI" ]
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The gcloud storage buckets create command requires the bucket name to be specified as a URI (e.g., gs://my-bucket). It's crucial to ensure that $BUCKET_URI includes the gs:// prefix. If $BUCKET_URI does not include the gs:// prefix, the command will fail.

Consider adding the gs:// prefix to $BUCKET_URI to ensure the bucket name is correctly formatted.

! gcloud storage buckets create --location $REGION gs://$BUCKET_URI

},
{
"cell_type": "markdown",
Expand All @@ -531,8 +530,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_URI"
]
"! gcloud storage ls --all-versions --long $BUCKET_URI" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -1147,8 +1145,7 @@
"! rm -f custom.tar custom.tar.gz\n",
"! tar cvf custom.tar custom\n",
"! gzip custom.tar\n",
"! gsutil cp custom.tar.gz $BUCKET_URI/trainer_example.tar.gz"
]
"! gcloud storage cp custom.tar.gz $BUCKET_URI/trainer_example.tar.gz" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -1357,8 +1354,7 @@
"delete_bucket = False\n",
"\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
" ! gsutil rm -r $BUCKET_URI"
]
" ! gcloud storage rm --recursive $BUCKET_URI" ]
}
],
"metadata": {
Expand Down
12 changes: 4 additions & 8 deletions notebooks/community/ml_ops/stage2/get_started_with_tfhub_models.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -448,8 +448,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_URI"
]
"! gcloud storage buckets create --location=$REGION $BUCKET_URI" ]
},
{
"cell_type": "markdown",
Expand All @@ -468,8 +467,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_URI"
]
"! gcloud storage ls --all-versions --long $BUCKET_URI" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -1132,8 +1130,7 @@
"! rm -f custom.tar custom.tar.gz\n",
"! tar cvf custom.tar custom\n",
"! gzip custom.tar\n",
"! gsutil cp custom.tar.gz $BUCKET_URI/trainer_cifar10.tar.gz"
]
"! gcloud storage cp custom.tar.gz $BUCKET_URI/trainer_cifar10.tar.gz" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -1381,8 +1378,7 @@
"delete_bucket = False\n",
"\n",
"if delete_bucket and \"BUCKET_URI\" in globals():\n",
" ! gsutil rm -r $BUCKET_URI"
]
" ! gcloud storage rm --recursive $BUCKET_URI" ]
}
],
"metadata": {
Expand Down
9 changes: 3 additions & 6 deletions notebooks/community/ml_ops/stage5/get_started_with_autoscaling.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -437,8 +437,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_URI"
]
"! gcloud storage buckets create --location $REGION $BUCKET_URI" ]
},
{
"cell_type": "markdown",
Expand All @@ -457,8 +456,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_URI"
]
"! gcloud storage ls --all-versions --long $BUCKET_URI" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -1357,8 +1355,7 @@
"delete_bucket = True\n",
"\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
" ! gsutil rm -r $BUCKET_URI"
]
" ! gcloud storage rm --recursive $BUCKET_URI" ]
}
],
"metadata": {
Expand Down
27 changes: 9 additions & 18 deletions notebooks/community/ml_ops/stage6/get_started_with_xgboost_model_online.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -484,8 +484,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_URI"
]
"! gcloud storage buckets create --location=$REGION $BUCKET_URI" ]
},
{
"cell_type": "markdown",
Expand All @@ -504,8 +503,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_URI"
]
"! gcloud storage ls --all-versions --long $BUCKET_URI" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -717,8 +715,7 @@
"with open(\"instance.yaml\", \"w\") as f:\n",
" f.write(yaml)\n",
"\n",
"! gsutil cp instance.yaml {BUCKET_URI}/instance.yaml"
]
"! gcloud storage cp instance.yaml {BUCKET_URI}/instance.yaml" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -1274,30 +1271,25 @@
" + \"/evaluation_metrics\"\n",
" )\n",
" if tf.io.gfile.exists(EXECUTE_OUTPUT):\n",
" ! gsutil cat $EXECUTE_OUTPUT\n",
" return EXECUTE_OUTPUT\n",
" ! gcloud storage cat $EXECUTE_OUTPUT\n", " return EXECUTE_OUTPUT\n",
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This command is using gcloud storage cat to read the contents of a file, but it's missing a newline character at the end. This could cause issues when parsing the output, especially if the file is large or if the parsing logic relies on complete lines.

Consider adding a newline character (\n) to the end of the gcloud storage cat command to ensure that the output is properly formatted.

" elif tf.io.gfile.exists(GCP_RESOURCES):\n",
" ! gsutil cat $GCP_RESOURCES\n",
" return GCP_RESOURCES\n",
" ! gcloud storage cat $GCP_RESOURCES\n", " return GCP_RESOURCES\n",
" elif tf.io.gfile.exists(EVAL_METRICS):\n",
" ! gsutil cat $EVAL_METRICS\n",
" return EVAL_METRICS\n",
" ! gcloud storage cat $EVAL_METRICS\n", " return EVAL_METRICS\n",
"\n",
" return None\n",
"\n",
"\n",
"print(\"get-vertex-model\")\n",
"artifacts = print_pipeline_output(pipeline, \"get-vertex-model\")\n",
"print(\"\\n\\n\")\n",
"output = !gsutil cat $artifacts\n",
"output = json.loads(output[0])\n",
"output = !gcloud storage cat $artifacts\n", "output = json.loads(output[0])\n",
"model_id = output[\"artifacts\"][\"model\"][\"artifacts\"][0][\"metadata\"][\"resourceName\"]\n",
"print(\"\\n\\n\")\n",
"print(\"endpoint-create\")\n",
"artifacts = print_pipeline_output(pipeline, \"endpoint-create\")\n",
"print(\"\\n\\n\")\n",
"output = !gsutil cat $artifacts\n",
"output = json.loads(output[0])\n",
"output = !gcloud storage cat $artifacts\n", "output = json.loads(output[0])\n",
"endpoint_id = output[\"artifacts\"][\"endpoint\"][\"artifacts\"][0][\"metadata\"][\n",
" \"resourceName\"\n",
"]\n",
Expand Down Expand Up @@ -1404,8 +1396,7 @@
"delete_bucket = True\n",
"\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
" ! gsutil rm -rf {BUCKET_URI}"
]
" ! gcloud storage rm --recursive --continue-on-error {BUCKET_URI}" ]
}
],
"metadata": {
Expand Down
16 changes: 6 additions & 10 deletions notebooks/community/model_garden/model_garden_gemma_deployment_on_vertex.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,11 +152,9 @@
"if BUCKET_URI is None or BUCKET_URI.strip() == \"\" or BUCKET_URI == \"gs://\":\n",
" BUCKET_URI = f\"gs://{PROJECT_ID}-tmp-{now}-{str(uuid.uuid4())[:4]}\"\n",
" BUCKET_NAME = \"/\".join(BUCKET_URI.split(\"/\")[:3])\n",
" ! gsutil mb -l {REGION} {BUCKET_URI}\n",
"else:\n",
" ! gcloud storage buckets create --location={REGION} {BUCKET_URI}\n", "else:\n",
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The gcloud storage buckets create command requires the bucket name to be specified as a URI (e.g., gs://my-bucket). It's crucial to ensure that $BUCKET_URI includes the gs:// prefix. If $BUCKET_URI does not include the gs:// prefix, the command will fail.

Consider adding the gs:// prefix to $BUCKET_URI to ensure the bucket name is correctly formatted.

! gcloud storage buckets create --location={REGION} {BUCKET_URI}

" assert BUCKET_URI.startswith(\"gs://\"), \"BUCKET_URI must start with `gs://`.\"\n",
" shell_output = ! gsutil ls -Lb {BUCKET_NAME} | grep \"Location constraint:\" | sed \"s/Location constraint://\"\n",
" bucket_region = shell_output[0].strip().lower()\n",
" shell_output = ! gcloud storage ls --full --buckets {BUCKET_NAME} | grep \"Location constraint:\" | sed \"s/Location constraint://\"\n", " bucket_region = shell_output[0].strip().lower()\n",
" if bucket_region != REGION:\n",
" raise ValueError(\n",
" \"Bucket region %s is different from notebook region %s\"\n",
Expand All @@ -180,8 +178,8 @@
"\n",
"\n",
"# Provision permissions to the SERVICE_ACCOUNT with the GCS bucket\n",
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.admin $BUCKET_NAME\n",
"\n",
"# Note: Migrating scripts using gsutil iam ch is more complex than get or set. You need to replace the single iam ch command with a series of gcloud storage bucket add-iam-policy-binding and/or gcloud storage bucket remove-iam-policy-binding commands, or replicate the read-modify-write loop.\n",
"! gcloud storage buckets add-iam-policy-binding $BUCKET_NAME --member=serviceAccount:{SERVICE_ACCOUNT} --role=roles/storage.admin\n", "\n",
"! gcloud config set project $PROJECT_ID\n",
"! gcloud projects add-iam-policy-binding --no-user-output-enabled {PROJECT_ID} --member=serviceAccount:{SERVICE_ACCOUNT} --role=\"roles/storage.admin\"\n",
"! gcloud projects add-iam-policy-binding --no-user-output-enabled {PROJECT_ID} --member=serviceAccount:{SERVICE_ACCOUNT} --role=\"roles/aiplatform.user\"\n",
Expand Down Expand Up @@ -229,8 +227,7 @@
"\n",
" ! mkdir -p ./gemma\n",
" ! curl -X GET \"{signed_url}\" | tar -xzvf - -C ./gemma/\n",
" ! gsutil -m cp -R ./gemma/* {MODEL_BUCKET}\n",
"\n",
" ! gcloud storage cp --recursive ./gemma/* {MODEL_BUCKET}\n", "\n",
" model_path_prefix = MODEL_BUCKET\n",
" HF_TOKEN = \"\"\n",
"else:\n",
Expand Down Expand Up @@ -1007,8 +1004,7 @@
"\n",
"delete_bucket = False # @param {type:\"boolean\"}\n",
"if delete_bucket:\n",
" ! gsutil -m rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
Expand Down
10 changes: 4 additions & 6 deletions notebooks/community/model_garden/model_garden_integration_with_agent.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,11 +177,10 @@
"if BUCKET_URI is None or BUCKET_URI.strip() == \"\" or BUCKET_URI == \"gs://\":\n",
" BUCKET_URI = f\"gs://{PROJECT_ID}-tmp-{now}-{str(uuid.uuid4())[:4]}\"\n",
" BUCKET_NAME = \"/\".join(BUCKET_URI.split(\"/\")[:3])\n",
" ! gsutil mb -l {REGION} {BUCKET_URI}\n",
"else:\n",
" ! gcloud storage buckets create --location={REGION} {BUCKET_URI}\n", "else:\n",
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The gcloud storage buckets create command requires the bucket name to be specified as a URI (e.g., gs://my-bucket). It's crucial to ensure that $BUCKET_URI includes the gs:// prefix. If $BUCKET_URI does not include the gs:// prefix, the command will fail.

Consider adding the gs:// prefix to $BUCKET_URI to ensure the bucket name is correctly formatted.

! gcloud storage buckets create --location={REGION} {BUCKET_URI}

" assert BUCKET_URI.startswith(\"gs://\"), \"BUCKET_URI must start with `gs://`.\"\n",
" shell_output = ! gsutil ls -Lb {BUCKET_NAME} | grep \"Location constraint:\" | sed \"s/Location constraint://\"\n",
" bucket_region = shell_output[0].strip().lower()\n",
" # Note: The format of the full listing output is different. gcloud storage uses a title case for keys and will not display a field if its value is \"None\".\n",
" shell_output = ! gcloud storage ls --full --buckets {BUCKET_NAME} | grep \"Location constraint:\" | sed \"s/Location constraint://\"\n", " bucket_region = shell_output[0].strip().lower()\n",
" if bucket_region != REGION:\n",
" raise ValueError(\n",
" \"Bucket region %s is different from notebook region %s\"\n",
Expand All @@ -202,8 +201,7 @@
"\n",
"\n",
"# Provision permissions to the SERVICE_ACCOUNT with the GCS bucket\n",
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.admin $BUCKET_NAME\n",
"\n",
"! gcloud storage buckets add-iam-policy-binding $BUCKET_NAME --member=serviceAccount:{SERVICE_ACCOUNT} --role=roles/storage.admin\n", "\n",
"! gcloud config set project $PROJECT_ID\n",
"! gcloud projects add-iam-policy-binding --no-user-output-enabled {PROJECT_ID} --member=serviceAccount:{SERVICE_ACCOUNT} --role=\"roles/storage.admin\"\n",
"! gcloud projects add-iam-policy-binding --no-user-output-enabled {PROJECT_ID} --member=serviceAccount:{SERVICE_ACCOUNT} --role=\"roles/aiplatform.user\"\n",
Expand Down
Loading
Loading