-
Notifications
You must be signed in to change notification settings - Fork 59
Open
Labels
Milestone
Description
Replace on-the-fly defined GCP buckets with a single persistent bucket that can have narrower IAM grants applied to it, avoiding the gsutil mb/gsutil rb commands during SQL import.
We currently create here
splice/cluster/pulumi/canton-network/bigquery-cloudsql.sh
Lines 159 to 167 in ec95f03
| # create temporary bucket | |
| echo "Creating temporary bucket $TMP_BUCKET" | |
| gsutil mb --pap enforced -p "$PRIVATE_NETWORK_PROJECT" \ | |
| -l "$COMPUTE_REGION" "gs://$TMP_BUCKET" | |
| # grant DB service account access to the bucket | |
| echo "Granting CloudSQL DB access to $TMP_BUCKET" | |
| gsutil iam ch "serviceAccount:$SERVICE_ACCOUNT_EMAIL:roles/storage.objectAdmin" \ | |
| "gs://$TMP_BUCKET" |
and delete here
splice/cluster/pulumi/canton-network/bigquery-cloudsql.sh
Lines 152 to 154 in ec95f03
| echo 'Cleaning up temporary GCS object and bucket' | |
| gsutil rm "$GCS_URI" || true | |
| gsutil rb "gs://$TMP_BUCKET" || true |
Instead, we can
- define a
gcp.storage.Bucketin the Pulumi (bigQuery.ts) - passing its location in as an argument to the script:
splice/cluster/pulumi/canton-network/src/bigQuery.ts
Lines 351 to 353 in ec95f03
| const scriptArgs = pulumi.interpolate`\ | |
| --private-network-project="${privateNetwork.project}" \ | |
| --compute-region="${cloudsdkComputeRegion()}" \ |
- and in cleanup, revoke the IAM grant to the
$SERVICE_ACCOUNT_EMAIL.