Skip to content

Missing gcloud command or terraform code to create cloud build triggers #19

@hilliao

Description

@hilliao

Users like to have a way to create cloud build triggers in the Terraform folder or with gcloud commands. I found tensorflow transform is using Apache beam runner in the following cloud build file.

# Compile the pipeline.
- name: '$_CICD_IMAGE_URI'
  entrypoint: 'python'
  args: ['build/utils.py',
          '--mode', 'compile-pipeline',
          '--pipeline-name', '$_PIPELINE_NAME'
          ]
  dir: 'mlops-with-vertex-ai'
  env: 
  - 'PROJECT=$_PROJECT'  
  - 'REGION=$_REGION'
  - 'MODEL_DISPLAY_NAME=$_MODEL_DISPLAY_NAME'
  - 'DATASET_DISPLAY_NAME=$_DATASET_DISPLAY_NAME'  
  - 'GCS_LOCATION=$_GCS_LOCATION' 
  - 'TFX_IMAGE_URI=$_TFX_IMAGE_URI' 
  - 'BEAM_RUNNER=$_BEAM_RUNNER'
  - 'TRAINING_RUNNER=$_TRAINING_RUNNER'
  id: 'Compile Pipeline'
  waitFor: ['Local Test E2E Pipeline']

So far I gather an example from a notebook's section. But I can't find how to create the cloud build trigger.

  1. Run the training pipeline using Vertex Pipelines
    Set the pipeline configurations for the Vertex AI run
    os.environ["DATASET_DISPLAY_NAME"] = DATASET_DISPLAY_NAME
    os.environ["MODEL_DISPLAY_NAME"] = MODEL_DISPLAY_NAME
    os.environ["PIPELINE_NAME"] = PIPELINE_NAME
    os.environ["PROJECT"] = PROJECT
    os.environ["REGION"] = REGION
    os.environ["GCS_LOCATION"] = f"gs://{BUCKET}/{DATASET_DISPLAY_NAME}"
    os.environ["TRAIN_LIMIT"] = "85000"
    os.environ["TEST_LIMIT"] = "15000"
    os.environ["BEAM_RUNNER"] = "DataflowRunner"
    os.environ["TRAINING_RUNNER"] = "vertex"
    os.environ["TFX_IMAGE_URI"] = f"gcr.io/{PROJECT}/{DATASET_DISPLAY_NAME}:{VERSION}"
    os.environ["ENABLE_CACHE"] = "1"

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions