-
Notifications
You must be signed in to change notification settings - Fork 148
Open
Description
Users like to have a way to create cloud build triggers in the Terraform folder or with gcloud commands. I found tensorflow transform is using Apache beam runner in the following cloud build file.
# Compile the pipeline.
- name: '$_CICD_IMAGE_URI'
entrypoint: 'python'
args: ['build/utils.py',
'--mode', 'compile-pipeline',
'--pipeline-name', '$_PIPELINE_NAME'
]
dir: 'mlops-with-vertex-ai'
env:
- 'PROJECT=$_PROJECT'
- 'REGION=$_REGION'
- 'MODEL_DISPLAY_NAME=$_MODEL_DISPLAY_NAME'
- 'DATASET_DISPLAY_NAME=$_DATASET_DISPLAY_NAME'
- 'GCS_LOCATION=$_GCS_LOCATION'
- 'TFX_IMAGE_URI=$_TFX_IMAGE_URI'
- 'BEAM_RUNNER=$_BEAM_RUNNER'
- 'TRAINING_RUNNER=$_TRAINING_RUNNER'
id: 'Compile Pipeline'
waitFor: ['Local Test E2E Pipeline']
So far I gather an example from a notebook's section. But I can't find how to create the cloud build trigger.
- Run the training pipeline using Vertex Pipelines
Set the pipeline configurations for the Vertex AI run
os.environ["DATASET_DISPLAY_NAME"] = DATASET_DISPLAY_NAME
os.environ["MODEL_DISPLAY_NAME"] = MODEL_DISPLAY_NAME
os.environ["PIPELINE_NAME"] = PIPELINE_NAME
os.environ["PROJECT"] = PROJECT
os.environ["REGION"] = REGION
os.environ["GCS_LOCATION"] = f"gs://{BUCKET}/{DATASET_DISPLAY_NAME}"
os.environ["TRAIN_LIMIT"] = "85000"
os.environ["TEST_LIMIT"] = "15000"
os.environ["BEAM_RUNNER"] = "DataflowRunner"
os.environ["TRAINING_RUNNER"] = "vertex"
os.environ["TFX_IMAGE_URI"] = f"gcr.io/{PROJECT}/{DATASET_DISPLAY_NAME}:{VERSION}"
os.environ["ENABLE_CACHE"] = "1"
Metadata
Metadata
Assignees
Labels
No labels