Skip to content

Commit b0da645

Browse files
authored
Enable deploy of model registered by previous build (#207)
Allows you to skip the first two stages of the pipeline and deploy a model registered by a previous build by setting the variable MODEL_BUILD_ID to the previous build's id when queuing the pipeline. This speeds up the process of fixing issues with the scoring or deployment configuration.
1 parent 166c654 commit b0da645

File tree

5 files changed

+18
-11
lines changed

5 files changed

+18
-11
lines changed

.pipelines/diabetes_regression-ci-build-train.yml

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ pool:
2121
stages:
2222
- stage: 'Model_CI'
2323
displayName: 'Model CI'
24+
condition: not(variables['MODEL_BUILD_ID'])
2425
jobs:
2526
- job: "Model_CI_Pipeline"
2627
displayName: "Model CI Pipeline"
@@ -42,6 +43,7 @@ stages:
4243

4344
- stage: 'Trigger_AML_Pipeline'
4445
displayName: 'Train model'
46+
condition: not(variables['MODEL_BUILD_ID'])
4547
jobs:
4648
- job: "Get_Pipeline_ID"
4749
condition: and(succeeded(), eq(coalesce(variables['auto-trigger-training'], 'true'), 'true'))
@@ -90,7 +92,7 @@ stages:
9092
- stage: 'Deploy_ACI'
9193
displayName: 'Deploy to ACI'
9294
dependsOn: Trigger_AML_Pipeline
93-
condition: and(succeeded(), variables['ACI_DEPLOYMENT_NAME'])
95+
condition: and(or(succeeded(), variables['MODEL_BUILD_ID']), variables['ACI_DEPLOYMENT_NAME'])
9496
jobs:
9597
- job: "Deploy_ACI"
9698
displayName: "Deploy to ACI"
@@ -157,7 +159,7 @@ stages:
157159
- stage: 'Deploy_Webapp'
158160
displayName: 'Deploy to Webapp'
159161
dependsOn: Trigger_AML_Pipeline
160-
condition: and(succeeded(), variables['WEBAPP_DEPLOYMENT_NAME'])
162+
condition: and(or(succeeded(), variables['MODEL_BUILD_ID']), variables['WEBAPP_DEPLOYMENT_NAME'])
161163
jobs:
162164
- job: "Deploy_Webapp"
163165
displayName: "Deploy to Webapp"

.pipelines/diabetes_regression-template-get-model-version.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ steps:
77
inlineScript: |
88
set -e # fail on error
99
export SUBSCRIPTION_ID=$(az account show --query id -o tsv)
10-
python -m ml_service.pipelines.diabetes_regression_verify_train_pipeline --build_id $(Build.BuildId) --output_model_version_file "model_version.txt"
10+
python -m ml_service.pipelines.diabetes_regression_verify_train_pipeline --build_id $(modelbuildid) --output_model_version_file "model_version.txt"
1111
# Output model version to Azure DevOps job
1212
MODEL_VERSION="$(cat model_version.txt)"
1313
echo "##vso[task.setvariable variable=MODEL_VERSION]$MODEL_VERSION"

.pipelines/diabetes_regression-variables.yml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,3 +51,8 @@ variables:
5151
# value: "true"
5252
# - name: ALLOW_RUN_CANCEL
5353
# value: "true"
54+
55+
# For debugging deployment issues. Specify a build id with the MODEL_BUILD_ID pipeline variable at queue time
56+
# to skip training and deploy a model registered by a previous build.
57+
- name: modelbuildid
58+
value: $[coalesce(variables['MODEL_BUILD_ID'], variables['Build.BuildId'])]

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ You can find the details of the code and scripts in the repository [here](/docs/
6666
- [Azure ML Python SDK Quickstart](https://docs.microsoft.com/en-us/azure/machine-learning/service/quickstart-create-workspace-with-python)
6767
- [Azure DevOps](https://docs.microsoft.com/en-us/azure/devops/?view=vsts)
6868

69-
# Contributing
69+
## Contributing
7070

7171
This project welcomes contributions and suggestions. Most contributions require you to agree to a
7272
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us

docs/getting_started.md

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,6 @@ This repository contains a template and demonstrates how to apply it to a sample
1717

1818
If the desire is to adopt this template for your project and to use it with your machine learning code, it is recommended to go through this guide as it is first. This ensures everything is working on your environment. After the sample is working, follow the [bootstrap instructions](../bootstrap/README.md) to convert the ***diabetes_regression*** sample into your project starting point.
1919

20-
2120
## Create a Variable Group for your Pipeline
2221

2322
We make use of a variable group inside Azure DevOps to store variables and their
@@ -173,7 +172,6 @@ and check out the published training pipeline in the **mlops-AML-WS** workspace
173172

174173
Great, you now have the build pipeline set up which automatically triggers every time there's a change in the master branch.
175174

176-
177175
* The first stage of the pipeline, **Model CI**, performs linting, unit testing, build and publishes an **ML Training Pipeline** in an **ML Workspace**.
178176

179177
**Note:** The build pipeline also supports building and publishing ML
@@ -187,22 +185,24 @@ with R on Azure ML Compute. You will also need to uncomment (i.e. include) the
187185
to train a model with R on Databricks. You will need
188186
to manually create a Databricks cluster and attach it to the ML Workspace as a
189187
compute (Values DB_CLUSTER_ID and DATABRICKS_COMPUTE_NAME variables should be
190-
specified). Example ML pipelines using R have a single step to train a model. They don't demonstrate how to evaluate and register a model. The evaluation and registering techniques are shown only in the Python implementation.
188+
specified). Example ML pipelines using R have a single step to train a model. They don't demonstrate how to evaluate and register a model. The evaluation and registering techniques are shown only in the Python implementation.
191189

192190
* The second stage of the pipeline, **Train model**, triggers the run of the ML Training Pipeline. The training pipeline will train, evaluate, and register a new model. The actual computation is performed in an [Azure Machine Learning Compute cluster](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute). In Azure DevOps, this stage runs an agentless job that waits for the completion of the Azure ML job, allowing the pipeline to wait for training completion for hours or even days without using agent resources.
193191

194192
**Note:** If the model evaluation determines that the new model does not perform better than the previous one then the new model will not be registered and the pipeline will be cancelled.
195193

196194
* The third stage of the pipeline, **Deploy to ACI**, deploys the model to the QA environment in [Azure Container Instances](https://azure.microsoft.com/en-us/services/container-instances/). It then runs a *smoke test* to validate the deployment, i.e. sends a sample query to the scoring web service and verifies that it returns a response in the expected format.
197195

198-
The pipeline uses a Docker container on the Azure Pipelines agents to accomplish the pipeline steps. The image of the container ***mcr.microsoft.com/mlops/python:latest*** is built with this [Dockerfile](./environment_setup/Dockerfile) and it has all necessary dependencies installed for the purposes of this repository. This image serves as an example of using a custom Docker image that provides a pre-baked environment. This environment is guaranteed to be the same on any building agent, VM or local machine. In your project you will want to build your own Docker image that only contains the dependencies and tools required for your use case. This image will be more likely smaller and therefore faster, and it will be totally maintained by your team.
196+
The pipeline uses a Docker container on the Azure Pipelines agents to accomplish the pipeline steps. The image of the container ***mcr.microsoft.com/mlops/python:latest*** is built with this [Dockerfile](./environment_setup/Dockerfile) and it has all necessary dependencies installed for the purposes of this repository. This image serves as an example of using a custom Docker image that provides a pre-baked environment. This environment is guaranteed to be the same on any building agent, VM or local machine. In your project you will want to build your own Docker image that only contains the dependencies and tools required for your use case. This image will be more likely smaller and therefore faster, and it will be totally maintained by your team.
199197

200198
Wait until the pipeline finishes and verify that there is a new model in the **ML Workspace**:
201199

202200
![trained model](./images/trained-model.png)
203201

204202
To disable the automatic trigger of the training pipeline, change the `auto-trigger-training` variable as listed in the `.pipelines\diabetes_regression-ci-build-train.yml` pipeline to `false`. This can also be overridden at runtime execution of the pipeline.
205203

204+
To skip model training and registration, and deploy a model successfully registered by a previous build (for testing changes to the score file or inference configuration), add the variable `MODEL_BUILD_ID` when the pipeline is queued, and set the value to the id of the previous build.
205+
206206
### Deploy the Model to Azure Kubernetes Service
207207

208208
The final stage is to deploy the model to the production environment running on
@@ -238,7 +238,7 @@ In the Variables tab, edit your variable group (`devopsforai-aml-vg`). In the va
238238
| ---------------------- | ---------------------- |
239239
| WEBAPP_DEPLOYMENT_NAME | _name of your web app_ |
240240

241-
Set **WEBAPP_DEPLOYMENT_NAME** to the name of your Azure Web App. This app must exist before you can deploy the model to it.
241+
Set **WEBAPP_DEPLOYMENT_NAME** to the name of your Azure Web App. This app must exist before you can deploy the model to it.
242242

243243
Delete the **ACI_DEPLOYMENT_NAME** variable.
244244

@@ -254,10 +254,10 @@ Make sure your webapp has the credentials to pull the image from the Azure Conta
254254

255255
![build](./images/multi-stage-webapp.png)
256256

257-
# Next steps
257+
## Next steps
258258

259259
* You may wish to follow the [bootstrap instructions](../bootstrap/README.md) to create a starting point for your project use case.
260-
* Use the [Convert ML experimental code to production code](https://docs.microsoft.com/azure/machine-learning/tutorial-convert-ml-experiment-to-production#use-your-own-model-with-mlopspython-code-template) tutorial which explains how to bring your machine learning code on top of this template.
260+
* Use the [Convert ML experimental code to production code](https://docs.microsoft.com/azure/machine-learning/tutorial-convert-ml-experiment-to-production#use-your-own-model-with-mlopspython-code-template) tutorial which explains how to bring your machine learning code on top of this template.
261261
* The provided pipeline definition YAML file is a sample starting point, which you should tailor to your processes and environment.
262262
* You should edit the pipeline definition to remove unused stages. For example, if you are deploying to ACI and AKS, you should delete the unused `Deploy_Webapp` stage.
263263
* You may wish to enable [manual approvals](https://docs.microsoft.com/en-us/azure/devops/pipelines/process/approvals) before the deployment stages.

0 commit comments

Comments
 (0)