Skip to content
This repository was archived by the owner on Oct 16, 2025. It is now read-only.
This repository was archived by the owner on Oct 16, 2025. It is now read-only.

Correct arguments for gcloud dataflow jobs run #73

@kjothen

Description

@kjothen

Hi - I get an error with the last line in deploy-data-tokeninzation-solution.sh.

gcloud config set project <project_id>
sh deploy-data-tokeninzation-solution.sh

...

gcloud dataflow jobs run ${jobId} --gcs-location gs://dataflow-templates/latest/Stream_DLP_GCS_Text_to_BigQuery --parameters --region=us-central1,inputFilePattern=gs://${DATA_STORAGE_BUCKET}/CCRecords_1564602825.csv,dlpProjectId=${PROJECT_ID},deidentifyTemplateName=${DEID_TEMPLATE_NAME},inspectTemplateName=${INSPECT_TEMPLATE_NAME},datasetName=${BQ_DATASET_NAME},batchSize=500
ERROR: (gcloud.dataflow.jobs.run) argument --parameters: expected one argument
Usage: gcloud dataflow jobs run JOB_NAME --gcs-location=GCS_LOCATION [optional flags]
  optional flags may be  --additional-experiments | --dataflow-kms-key |
                         --disable-public-ips | --enable-streaming-engine |
                         --help | --max-workers | --network | --num-workers |
                         --parameters | --region | --service-account-email |
                         --staging-location | --subnetwork |
                         --worker-machine-type | --worker-region |
                         --worker-zone | --zone

If I switch the args around, pulling region out of parameters, then it works:

gcloud dataflow jobs run ${jobId} --gcs-location gs://dataflow-templates/latest/Stream_DLP_GCS_Text_to_BigQuery --region=us-central1 --parameters=inputFilePattern=gs://${DATA_STORAGE_BUCKET}/CCRecords_1564602825.csv,dlpProjectId=${PROJECT_ID},deidentifyTemplateName=${DEID_TEMPLATE_NAME},inspectTemplateName=${INSPECT_TEMPLATE_NAME},datasetName=${BQ_DATASET_NAME},batchSize=500

createTime: '2021-08-11T11:14:10.713985Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: 2021-08-11_04_14_08-3628396078995258394
location: us-central1
name: demo-dlp-deid-pipeline-20210811-111101
projectId: <project-id>
startTime: '2021-08-11T11:14:10.713985Z'
type: JOB_TYPE_STREAMING

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions