Skip to content

Migrate gsutil usage to gcloud storage #4315

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gericdong merged 6 commits into from
Dec 24, 2025
Merged

Migrate gsutil usage to gcloud storage #4315

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
c24fac0
Migrate gsutil usage to gcloud storage
googlyrahman Oct 16, 2025
1115222
changes for 4315
gurusai-voleti Dec 12, 2025
5506a3c
Apply automated linter fixes
gurusai-voleti Dec 16, 2025
ccdb33e
Merge pull request #48 from gurusai-voleti/br_4315
googlyrahman Dec 17, 2025
c1378af
removed model_garden folder changes
gurusai-voleti Dec 22, 2025
ff92ebe
Merge pull request #86 from gurusai-voleti/br_4315
gurusai-voleti Dec 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 6 additions & 12 deletions ...ity-content/prediction_featurestore_integration/prediction_featurestore_integration.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -293,8 +293,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
]
"! gcloud storage buckets create --location=$REGION --project=$PROJECT_ID $BUCKET_URI" ]
},
{
"cell_type": "code",
Expand All @@ -304,8 +303,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_URI"
]
"! gcloud storage ls --all-versions --long $BUCKET_URI" ]
},
{
"cell_type": "markdown",
Expand All @@ -324,8 +322,7 @@
},
"outputs": [],
"source": [
"! gsutil cp -r gs://mco-mm/churn/* $BUCKET_URI"
]
"! gcloud storage cp --recursive gs://mco-mm/churn/* $BUCKET_URI" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -1063,8 +1060,7 @@
"outputs": [],
"source": [
"# Remove if the file already exists\n",
"!gsutil rm $BUCKET_URI/prediction_featuresstore_fetch_config.yaml"
]
"!gcloud storage rm $BUCKET_URI/prediction_featuresstore_fetch_config.yaml" ]
},
{
"cell_type": "code",
Expand All @@ -1074,8 +1070,7 @@
},
"outputs": [],
"source": [
"!gsutil cp prediction_featuresstore_fetch_config.yaml $BUCKET_URI"
]
"!gcloud storage cp prediction_featuresstore_fetch_config.yaml $BUCKET_URI" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -1475,8 +1470,7 @@
"outputs": [],
"source": [
"# Delete bucket\n",
"!gsutil -m rm -r $BUCKET_URI"
]
"!gcloud storage rm --recursive $BUCKET_URI" ]
}
],
"metadata": {
Expand Down
9 changes: 3 additions & 6 deletions notebooks/community/ml_ops/stage3/get_started_with_model_registry.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -441,8 +441,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_URI"
]
"! gcloud storage buckets create --location=$REGION $BUCKET_URI" ]
},
{
"cell_type": "markdown",
Expand All @@ -461,8 +460,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_URI"
]
"! gcloud storage ls --all-versions --long $BUCKET_URI" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -1176,8 +1174,7 @@
" print(e)\n",
"\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
" ! gsutil rm -rf {BUCKET_URI}"
]
" ! gcloud storage rm --recursive --continue-on-error {BUCKET_URI}" ]
}
],
"metadata": {
Expand Down
6 changes: 2 additions & 4 deletions notebooks/community/model_garden/model_garden_openai_api_llama3_1.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -253,8 +253,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l {LOCATION} -p {PROJECT_ID} {BUCKET_URI}"
]
"! gcloud storage buckets create --location={LOCATION} --project={PROJECT_ID} {BUCKET_URI}" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -1237,8 +1236,7 @@
" rag.delete_corpus(name=rag_corpus.name)\n",
"\n",
"if delete_bucket:\n",
" ! gsutil -m rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
Expand Down
14 changes: 6 additions & 8 deletions notebooks/community/model_garden/model_garden_pytorch_mixtral_deployment.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,11 +174,10 @@
"if BUCKET_URI is None or BUCKET_URI.strip() == \"\" or BUCKET_URI == \"gs://\":\n",
" BUCKET_URI = f\"gs://{PROJECT_ID}-tmp-{now}-{str(uuid.uuid4())[:4]}\"\n",
" BUCKET_NAME = \"/\".join(BUCKET_URI.split(\"/\")[:3])\n",
" ! gsutil mb -l {REGION} {BUCKET_URI}\n",
"else:\n",
" ! gcloud storage buckets create --location {REGION} {BUCKET_URI}\n", "else:\n",
" assert BUCKET_URI.startswith(\"gs://\"), \"BUCKET_URI must start with `gs://`.\"\n",
" shell_output = ! gsutil ls -Lb {BUCKET_NAME} | grep \"Location constraint:\" | sed \"s/Location constraint://\"\n",
" bucket_region = shell_output[0].strip().lower()\n",
" # Note: The format of the full listing output is different. gcloud storage uses a title case for keys and will not display a field if its value is \"None\".\n",
" shell_output = ! gcloud storage ls --full --buckets {BUCKET_NAME} | grep \"Location constraint:\" | sed \"s/Location constraint://\"\n", " bucket_region = shell_output[0].strip().lower()\n",
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The grep command is case-sensitive and is looking for Location constraint:, but the note on the previous line correctly states that gcloud storage uses title case for keys. The output from gcloud storage ls --full --buckets will be Location Constraint:, causing this grep command to fail and break the logic for retrieving the bucket region.

    shell_output = ! gcloud storage ls --full --buckets {BUCKET_NAME} | grep "Location Constraint:" | sed "s/Location Constraint://"

Copy link
Contributor

@gurusai-voleti gurusai-voleti Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Resolved] fixed

" if bucket_region != REGION:\n",
" raise ValueError(\n",
" \"Bucket region %s is different from notebook region %s\"\n",
Expand All @@ -202,8 +201,8 @@
"\n",
"\n",
"# Provision permissions to the SERVICE_ACCOUNT with the GCS bucket\n",
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.admin $BUCKET_NAME\n",
"\n",
"# Note: Migrating scripts using gsutil iam ch is more complex than get or set. You need to replace the single iam ch command with a series of gcloud storage bucket add-iam-policy-binding and/or gcloud storage bucket remove-iam-policy-binding commands, or replicate the read-modify-write loop.\n",
"! gcloud storage buckets add-iam-policy-binding $BUCKET_NAME --member=serviceAccount:{SERVICE_ACCOUNT} --role=roles/storage.admin\n", "\n",
"! gcloud config set project $PROJECT_ID\n",
"! gcloud projects add-iam-policy-binding --no-user-output-enabled {PROJECT_ID} --member=serviceAccount:{SERVICE_ACCOUNT} --role=\"roles/storage.admin\"\n",
"! gcloud projects add-iam-policy-binding --no-user-output-enabled {PROJECT_ID} --member=serviceAccount:{SERVICE_ACCOUNT} --role=\"roles/aiplatform.user\""
Expand Down Expand Up @@ -826,8 +825,7 @@
"\n",
"delete_bucket = False # @param {type:\"boolean\"}\n",
"if delete_bucket:\n",
" ! gsutil -m rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
Expand Down
15 changes: 5 additions & 10 deletions ...ex_endpoints/tf_hub_obj_detection/deploy_tfhub_object_detection_on_vertex_endpoints.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -445,8 +445,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -p $PROJECT_ID -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --project=$PROJECT_ID --location=$REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
Expand All @@ -465,8 +464,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -950,8 +948,7 @@
},
"outputs": [],
"source": [
"!gsutil cp -r $VERTEX_MODEL_PATH $BUCKET_NAME/obj_detection_model_vertex"
]
"!gcloud storage cp --recursive $VERTEX_MODEL_PATH $BUCKET_NAME/obj_detection_model_vertex" ]
},
{
"cell_type": "code",
Expand All @@ -961,8 +958,7 @@
},
"outputs": [],
"source": [
"!gsutil ls $BUCKET_NAME"
]
"!gcloud storage ls $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -1297,8 +1293,7 @@
"--quiet\n",
"\n",
"# Delete Cloud Storage objects that were created\n",
"#! gsutil -m rm -r $BUCKET_NAME"
]
"#! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
Expand Down
11 changes: 3 additions & 8 deletions notebooks/official/feature_store_legacy/sdk-feature-store-pandas.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -303,8 +303,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l {LOCATION} -p {PROJECT_ID} {BUCKET_URI}"
]
"! gcloud storage buckets create --location={LOCATION} --project={PROJECT_ID} {BUCKET_URI}" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -564,10 +563,7 @@
"# set the local file names\n",
"USERS_AVRO_FN = \"users.avro\"\n",
"MOVIES_AVRO_FN = \"movies.avro\"\n",
"# copy the files using gsutil\n",
"! gsutil cp $GCS_USERS_AVRO_URI $USERS_AVRO_FN\n",
"! gsutil cp $GCS_MOVIES_AVRO_URI $MOVIES_AVRO_FN"
]
"# copy the files using gcloud storage\n", "! gcloud storage cp $GCS_USERS_AVRO_URI $USERS_AVRO_FN\n", "! gcloud storage cp $GCS_MOVIES_AVRO_URI $MOVIES_AVRO_FN" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -1127,8 +1123,7 @@
"# Delete Cloud Storage objects that were created\n",
"delete_bucket = False # Set True for deletion\n",
"if delete_bucket:\n",
" ! gsutil -m rm -r $BUCKET_URI"
]
" ! gcloud storage rm --recursive $BUCKET_URI" ]
}
],
"metadata": {
Expand Down
27 changes: 10 additions & 17 deletions notebooks/official/pipelines/custom_model_training_and_batch_prediction.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -296,8 +296,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l {LOCATION} -p {PROJECT_ID} {BUCKET_URI}"
]
"! gcloud storage buckets create --location={LOCATION} --project={PROJECT_ID} {BUCKET_URI}" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -369,10 +368,10 @@
},
"outputs": [],
"source": [
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
"# Note: Migrating scripts using gsutil iam ch is more complex than get or set. You need to replace the single iam ch command with a series of gcloud storage bucket add-iam-policy-binding and/or gcloud storage bucket remove-iam-policy-binding commands, or replicate the read-modify-write loop.\n! gcloud storage buckets add-iam-policy-binding $BUCKET_URI --member=serviceAccount:{SERVICE_ACCOUNT} --role=roles/storage.objectCreator\n",
"\n",
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
]
# Note: Migrating scripts using gsutil iam ch is more complex than get or set. You need to replace the single iam ch command with a series of gcloud storage bucket add-iam-policy-binding and/or gcloud storage bucket remove-iam-policy-binding commands, or replicate the read-modify-write loop.
! gcloud storage buckets add-iam-policy-binding $BUCKET_URI --member=serviceAccount:{SERVICE_ACCOUNT} --role=roles/storage.objectViewer ]
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

This change introduces a JSON syntax error into the notebook file. The note and the gcloud command are not properly quoted as strings within the source array of the cell. This will cause the notebook to fail to load. Each line within the source array must be a valid JSON string.

        "# Note: Migrating scripts using gsutil iam ch is more complex than get or set. You need to replace the single iam ch command with a series of gcloud storage bucket add-iam-policy-binding and/or gcloud storage bucket remove-iam-policy-binding commands, or replicate the read-modify-write loop.",
        "! gcloud storage buckets add-iam-policy-binding $BUCKET_URI --member=serviceAccount:{SERVICE_ACCOUNT} --role=roles/storage.objectViewer"

Copy link
Contributor

@gurusai-voleti gurusai-voleti Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Resolved] removed note and formatted command

},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -1068,30 +1067,25 @@
" + \"/evaluation_metrics\"\n",
" )\n",
" if tf.io.gfile.exists(EXECUTE_OUTPUT):\n",
" ! gsutil cat $EXECUTE_OUTPUT\n",
" return EXECUTE_OUTPUT\n",
" ! gcloud storage cat $EXECUTE_OUTPUT\n", " return EXECUTE_OUTPUT\n",
" elif tf.io.gfile.exists(GCP_RESOURCES):\n",
" ! gsutil cat $GCP_RESOURCES\n",
" return GCP_RESOURCES\n",
" ! gcloud storage cat $GCP_RESOURCES\n", " return GCP_RESOURCES\n",
" elif tf.io.gfile.exists(EVAL_METRICS):\n",
" ! gsutil cat $EVAL_METRICS\n",
" return EVAL_METRICS\n",
" ! gcloud storage cat $EVAL_METRICS\n", " return EVAL_METRICS\n",
"\n",
" return None\n",
"\n",
"\n",
"print(\"model-upload\")\n",
"artifacts = print_pipeline_output(job, \"model-upload\")\n",
"print(\"\\n\")\n",
"output = !gsutil cat $artifacts\n",
"print(output)\n",
"output = !gcloud storage cat $artifacts\n", "print(output)\n",
"output = json.loads(output[0])\n",
"model_id = output[\"artifacts\"][\"model\"][\"artifacts\"][0][\"metadata\"][\"resourceName\"]\n",
"print(\"model-batch-predict\")\n",
"artifacts = print_pipeline_output(job, \"model-batch-predict\")\n",
"print(\"\\n\")\n",
"output = !gsutil cat $artifacts\n",
"output = json.loads(output[0])\n",
"output = !gcloud storage cat $artifacts\n", "output = json.loads(output[0])\n",
"batch_job_id = output[\"artifacts\"][\"batchpredictionjob\"][\"artifacts\"][0][\"metadata\"][\n",
" \"resourceName\"\n",
"]"
Expand Down Expand Up @@ -1133,8 +1127,7 @@
"# Delete the Cloud Storage bucket\n",
"delete_bucket = False # Set True for deletion\n",
"if delete_bucket:\n",
" ! gsutil rm -r $BUCKET_URI\n",
"\n",
" ! gcloud storage rm --recursive $BUCKET_URI\n", "\n",
"# Remove the locally generated files\n",
"! rm custom_model_training_spec.yaml\n",
"! rm -rf custom"
Expand Down
9 changes: 3 additions & 6 deletions notebooks/official/prediction/get_started_with_vertex_private_endpoints.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -307,8 +307,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l {LOCATION} -p {PROJECT_ID} {BUCKET_URI}"
]
"! gcloud storage buckets create --location={LOCATION} --project={PROJECT_ID} {BUCKET_URI}" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -975,8 +974,7 @@
},
"outputs": [],
"source": [
"! gsutil cp gs://cloud-ml-data/img/flower_photos/daisy/100080576_f52e8ee070_n.jpg test.jpg"
]
"! gcloud storage cp gs://cloud-ml-data/img/flower_photos/daisy/100080576_f52e8ee070_n.jpg test.jpg" ]
},
{
"cell_type": "code",
Expand Down Expand Up @@ -1171,8 +1169,7 @@
" print(e)\n",
"\n",
"if delete_bucket:\n",
" ! gsutil rm -rf {BUCKET_URI}\n",
"\n",
" ! gcloud storage rm --recursive --continue-on-error {BUCKET_URI}\n", "\n",
"if delete_generated_files:\n",
" ! rm -rf \"test.jpg\" \"instances.json\""
]
Expand Down
6 changes: 2 additions & 4 deletions notebooks/official/tabular_workflows/wide_and_deep_on_vertex_pipelines.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -297,8 +297,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $LOCATION -p $PROJECT_ID $BUCKET_URI"
]
"! gcloud storage buckets create --location $LOCATION --project $PROJECT_ID $BUCKET_URI" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -957,8 +956,7 @@
"# Delete bucket\n",
"delete_bucket = False\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
" ! gsutil -m rm -r $BUCKET_URI"
]
" ! gcloud storage rm --recursive $BUCKET_URI" ]
}
],
"metadata": {
Expand Down
12 changes: 4 additions & 8 deletions notebooks/official/training/multi_node_ddp_gloo_vertex_training_with_custom_container.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -282,8 +282,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $LOCATION -p $PROJECT_ID $BUCKET_URI"
]
"! gcloud storage buckets create --location=$LOCATION --project=$PROJECT_ID $BUCKET_URI" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -1203,8 +1202,7 @@
},
"outputs": [],
"source": [
"! gsutil ls $gcs_output_uri_prefix"
]
"! gcloud storage ls $gcs_output_uri_prefix" ]
},
{
"cell_type": "markdown",
Expand Down Expand Up @@ -1242,16 +1240,14 @@
"# Set this to true only if you'd like to delete your artifact repository\n",
"delete_artifact_repository = False\n",
"\n",
"! gsutil rm -rf $gcs_output_uri_prefix\n",
"\n",
"! gcloud storage rm --recursive --continue-on-error $gcs_output_uri_prefix\n", "\n",
"! rm -rf ./trainer\n",
"\n",
"if delete_artifact_repository:\n",
" !gcloud artifacts repositories delete {PRIVATE_REPO} --location={LOCATION} --quiet\n",
"\n",
"if delete_bucket:\n",
" ! gsutil rm -r $BUCKET_URI\n",
"\n",
" ! gcloud storage rm --recursive $BUCKET_URI\n", "\n",
"if delete_tensorboard:\n",
" tensorboard.delete()"
]
Expand Down
Loading