Skip to content

Commit c416f67

Browse files
kukushkingkukushking
authored
feat: model monitoring updates (#362)
* feat: align monitoring input parameters * update CDK * add input example * model quality monitoring fixes * bias and explainability fixes * kms key defaults * update readme * update tests * changelog * kms redact
1 parent e920f16 commit c416f67

13 files changed

Lines changed: 269 additions & 282 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1010
### **Added**
1111

1212
### **Changed**
13+
- updated `sagemaker-model-monitor` module and tested 4 types of monitoring end-to-end
1314

1415
## v2.0.0
1516

‎examples/manifests/sagemaker-model-monitoring-modules.yaml‎

Lines changed: 66 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -1,41 +1,71 @@
1-
# This is an example manifest group.
2-
# Replace the parameters with the parameters for your model below prior the deployment.
3-
name: monitor
4-
path: git::https://github.com/awslabs/aiops-modules.git//modules/sagemaker/sagemaker-model-monitoring?ref=release/2.0.0&depth=1
1+
name: monitor2
2+
path: modules/sagemaker/sagemaker-model-monitoring
53
parameters:
64
- name: sagemaker_project_id
7-
value: project-1
5+
value: SF-DEMO-xgb-churn-pred-model-monitor-2025-10-23-19-11-16
86
- name: sagemaker_project_name
9-
value: project-1
10-
- name: model_package_arn
11-
value: arn:aws:sagemaker:<region>:<account>:model-package/<package-name>/1
7+
value: SF-DEMO-xgb-churn-pred-model-monitor-2025-10-23-19-11-16
128
- name: model_bucket_arn
13-
value: arn:aws:s3:::<model-bucket-name>>
14-
- name: data-quality-checkstep-output-prefix
15-
value: <model-prefix>/dataqualitycheckstep
16-
- name: data-quality-output-prefix
17-
value: monitor/dataqualityoutput
9+
value: arn:aws:s3:::sagemaker-us-east-2-<REDACTED>
10+
# - name: kms_key_id
11+
# value: <REDACTED>
1812
- name: endpoint_name
19-
valueFrom:
20-
moduleMetadata:
21-
group: endpoints
22-
name: endpoint
23-
key: EndpointName
24-
- name: security_group_id
25-
valueFrom:
26-
moduleMetadata:
27-
group: endpoints
28-
name: endpoint
29-
key: SecurityGroupId
30-
- name: kms_key_id
31-
valueFrom:
32-
moduleMetadata:
33-
group: endpoints
34-
name: endpoint
35-
key: KmsKeyId
36-
- name: subnet_ids
37-
valueFrom:
38-
moduleMetadata:
39-
group: networking
40-
name: networking
41-
key: PrivateSubnetIds
13+
value: DEMO-xgb-churn-pred-model-monitor-2025-10-23-19-11-16
14+
# - name: security_group_id
15+
# valueFrom:
16+
# moduleMetadata:
17+
# group: networking
18+
# name: networking
19+
# key: SecurityGroupId
20+
# - name: subnet_ids
21+
# valueFrom:
22+
# moduleMetadata:
23+
# group: networking
24+
# name: networking
25+
# key: PrivateSubnetIds
26+
# Data quality monitor properties
27+
- name: enable_data_quality_monitor
28+
value: true
29+
- name: data-quality-schedule-expression
30+
value: cron(0 * ? * * *)
31+
- name: data_quality_baseline_s3_uri
32+
value: s3://sagemaker-us-east-2-<REDACTED>/SF-DEMO-xgb-churn-pred-model-monitor/baseline
33+
- name: data_quality_output_s3_uri
34+
value: s3://sagemaker-us-east-2-<REDACTED>/SF-DEMO-xgb-churn-pred-model-monitor/output
35+
# Model quality monitor properties
36+
- name: enable_model_quality_monitor
37+
value: true
38+
- name: model-quality-schedule-expression
39+
value: cron(0 * ? * * *)
40+
- name: model_quality_baseline_s3_uri
41+
value: s3://sagemaker-us-east-2-<REDACTED>/SF-DEMO-xgb-churn-pred-model-monitor/mq-baseline
42+
- name: model_quality_output_s3_uri
43+
value: s3://sagemaker-us-east-2-<REDACTED>/SF-DEMO-xgb-churn-pred-model-monitor/mq-output
44+
- name: model_quality_ground_truth_s3_uri
45+
value: s3://sagemaker-us-east-2-<REDACTED>/SF-DEMO-xgb-churn-pred-model-monitor/ground_truth
46+
- name: model_quality_inference_attribute
47+
value: 0
48+
- name: model_quality_probability_attribute
49+
value: 0
50+
- name: model_quality_probability_threshold_attribute
51+
value: 0.8
52+
# Model bias monitor properties
53+
- name: enable_model_bias_monitor
54+
value: true
55+
- name: model_bias_schedule_expression
56+
value: cron(0 * ? * * *)
57+
- name: model_bias_baseline_s3_uri
58+
value: s3://sagemaker-us-east-2-<REDACTED>/SF-DEMO-xgb-churn-pred-model-monitor/baseline
59+
- name: model_bias_output_s3_uri
60+
value: s3://sagemaker-us-east-2-<REDACTED>/SF-DEMO-xgb-churn-pred-model-monitor/output
61+
- name: model_bias_ground_truth_s3_uri
62+
value: s3://sagemaker-us-east-2-<REDACTED>/SF-DEMO-xgb-churn-pred-model-monitor/ground_truth
63+
# Model explainability monitor properties
64+
- name: enable_model_explainability_monitor
65+
value: true
66+
- name: model_explainability_schedule_expression
67+
value: cron(0 * ? * * *)
68+
- name: model_explainability_baseline_s3_uri
69+
value: s3://sagemaker-us-east-2-<REDACTED>/SF-DEMO-xgb-churn-pred-model-monitor/baseline
70+
- name: model_explainability_output_s3_uri
71+
value: s3://sagemaker-us-east-2-<REDACTED>/SF-DEMO-xgb-churn-pred-model-monitor/output

‎modules/sagemaker/sagemaker-model-monitoring/README.md‎

Lines changed: 86 additions & 121 deletions
Original file line numberDiff line numberDiff line change
@@ -2,14 +2,16 @@
22

33
## Description
44

5-
This module creates SageMaker Model Monitoring jobs for (optionally) data quality, model quality,
6-
model bias, and model explainability. It requires a deployed model endpoint and the proper check steps
7-
for each monitoring job:
5+
This module creates SageMaker Model Monitoring jobs and monitoring schedules for (optionally)
6+
data quality, model quality, model bias, and model explainability. It requires a deployed model
7+
endpoint with data capture enabled.
88

9-
* Data Quality: [QualityCheck step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-quality-check)
10-
* Model Quality: [QualityCheck step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-quality-check)
11-
* Model Bias: [ClarifyCheck step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-clarify-check)
12-
* Model Explainability: [ClarifyCheck step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-clarify-check)
9+
Available monitoring types:
10+
11+
* Data Quality
12+
* Model Quality
13+
* Model Bias
14+
* Model Explainability
1315

1416
Note that updating parameters will require replacing resources. Deployments may be delayed until any
1517
running monitoring jobs complete (and the resources can be destroyed).
@@ -24,83 +26,79 @@ running monitoring jobs complete (and the resources can be destroyed).
2426

2527
#### Required
2628

27-
- `endpoint-name`: The name of the endpoint used to run the monitoring job.
28-
- `security-group-id`: The VPC security group IDs, should provide access to the given `subnet-ids`.
29-
- `subnet-ids`: The ID of the subnets in the VPC to which you want to connect your training job or model.
30-
- `model-package-arn`: Model package ARN
31-
- `model-bucket-arn`: S3 bucket ARN for model artifacts
32-
- `kms-key-id`: The KMS key used to encrypted storage and output.
29+
- `endpoint_name`: The name of the endpoint used to run the monitoring job. NOTE: The endpoint must have data capture enabled. Data capture location must be in the bucket provided by `model_bucket_arn` parameter below.
30+
- `model_bucket_arn`: S3 bucket ARN for model, data capture, and monitoring artifacts. Used to provide IAM permissions for monitoring jobs. NOTE: All following S3 URIs must be under this bucket.
3331

3432
One or more of:
3533

36-
- `enable-data-quality-monitor`: True to enable the data quality monitoring job.
37-
- `enable-model-quality-monitor`: True to enable the model quality monitoring job.
38-
- `enable-model-bias-monitor`: True to enable the model bias monitoring job.
39-
- `enable-model-explainability-monitor`: True to enable the model explainability monitoring job.
34+
- `enable_data_quality_monitor`: True to enable the data quality monitoring job.
35+
- `enable_model_quality_monitor`: True to enable the model quality monitoring job.
36+
- `enable_model_bias_monitor`: True to enable the model bias monitoring job.
37+
- `enable_model_explainability_monitor`: True to enable the model explainability monitoring job.
4038

4139
#### Optional
4240

43-
- `sagemaker-project-id`: SageMaker project id
44-
- `sagemaker-project-name`: SageMaker project name
41+
- `security_group_id`: The VPC security group IDs, should provide access to the given `subnet_ids`.
42+
- `subnet_ids`: The ID of the subnets in the VPC to which you want to connect your training job or model.
43+
- `kms_key_id`: The KMS key used to encrypted storage and output.
44+
- `sagemaker_project_id`: SageMaker project id
45+
- `sagemaker_project_name`: SageMaker project name
46+
- `tags`: Dictionary of tags to apply to resources
4547

4648
### Per-job Parameters
4749

48-
Each job has prefixed parameters, some of which are required. E.g. if
49-
`enable-data-quality-monitor` is set to `true`, then the
50-
`data-quality-checkstep-output-prefix` parameter is required.
51-
52-
#### Required
53-
54-
- `*-checkstep-output-prefix`: The S3 prefix in `model-artifacts-bucket-arn` which contains the output from the corresponding [Check step in the SageMaker Pipeline](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#build-and-manage-steps-types).
55-
- `*-output-prefix`: The S3 prefix in `model-artifacts-bucket-arn` to contain the output of the monitoring job.
56-
57-
#### Optional
58-
59-
- `*-instance-count`: The number of ML compute instances to use in the model monitoring job.
60-
- `*-instance-type`: The ML compute instance type for the processing job.
61-
- `*-instance-volume-size-in-gb`: The size of the ML storage volume, in gigabytes, that you want to provision.
62-
- `*-max-runtime-in-seconds`: The maximum length of time, in seconds, the monitoring job can run before it is stopped.
63-
- `*-schedule-expression`: A cron expression that describes details about the monitoring schedule.
64-
65-
#### Job-specific Parameters
66-
67-
See below for the specific input parameters for each model monitoring
68-
job. These parameters are all required.
69-
70-
##### Data Quality Monitoring Job Parameters
71-
72-
N/A
73-
74-
##### Model Quality Monitoring Job Parameters
75-
76-
###### Required
77-
78-
- `model-quality-problem-type`: The machine learning problem type of the model that the monitoring job monitors.
79-
80-
###### Optional
81-
82-
- `model-quality-inference-attribute`: The attribute of the input data that represents the ground truth label.
83-
- `model-quality-probability-attribute`: In a classification problem, the attribute that represents the class probability.
84-
- `model-quality-probability-threshold-attribute`: The threshold for the class probability to be evaluated as a positive result.
85-
86-
##### Model Bias Monitoring Job Parameters
87-
88-
###### Optional
89-
90-
- `model-bias-checkstep-analysis-config-prefix`: The S3 prefix in `model-artifacts-bucket-arn` which contains the output from the [Clarify Check Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-clarify-check) used for model bias.
91-
- `model-bias-features-attribute`: The attributes of the input data that are the input features.
92-
- `model-bias-inference-attribute`: The attribute of the input data that represents the ground truth label.
93-
- `model-bias-probability-attribute`: In a classification problem, the attribute that represents the class probability.
94-
- `model-bias-probability-threshold-attribute`: The threshold for the class probability to be evaluated as a positive result.
95-
96-
##### Model Explainability Monitoring Job Parameters
97-
98-
###### Optional
99-
100-
- `model-explainability-checkstep-analysis-config-prefix`: The S3 prefix in `model-artifacts-bucket-arn` which contains the output from the [Clarify Check Step](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-clarify-check) used for model explainability.
101-
- `model-explainability-features-attribute`: The attributes of the input data that are the input features.
102-
- `model-explainability-inference-attribute`: The attribute of the input data that represents the ground truth label.
103-
- `model-explainability-probability-attribute`: In a classification problem, the attribute that represents the class probability.
50+
#### Data Quality Monitoring Job Parameters
51+
52+
- `data_quality_baseline_s3_uri`: S3 URI for baseline data quality statistics
53+
- `data_quality_output_s3_uri`: S3 URI for data quality monitoring output
54+
- `data_quality_instance_count`: Number of ML compute instances (default: 1)
55+
- `data_quality_instance_type`: ML compute instance type (default: "ml.m5.large")
56+
- `data_quality_instance_volume_size_in_gb`: Size of ML storage volume in GB (default: 20)
57+
- `data_quality_max_runtime_in_seconds`: Maximum runtime in seconds (default: 3600)
58+
- `data_quality_schedule_expression`: Cron expression for monitoring schedule (default: "cron(0 * ? * * *)")
59+
60+
#### Model Quality Monitoring Job Parameters
61+
62+
- `model_quality_baseline_s3_uri`: S3 URI for baseline model quality statistics
63+
- `model_quality_output_s3_uri`: S3 URI for model quality monitoring output
64+
- `model_quality_ground_truth_s3_uri`: S3 URI for ground truth data
65+
- `model_quality_instance_count`: Number of ML compute instances (default: 1)
66+
- `model_quality_instance_type`: ML compute instance type (default: "ml.m5.large")
67+
- `model_quality_instance_volume_size_in_gb`: Size of ML storage volume in GB (default: 20)
68+
- `model_quality_max_runtime_in_seconds`: Maximum runtime in seconds (default: 1800)
69+
- `model_quality_problem_type`: Machine learning problem type (default: "Regression")
70+
- `model_quality_inference_attribute`: Attribute representing the ground truth label
71+
- `model_quality_probability_attribute`: Attribute representing class probability
72+
- `model_quality_probability_threshold_attribute`: Threshold for class probability evaluation
73+
- `model_quality_schedule_expression`: Cron expression for monitoring schedule (default: "cron(0 * ? * * *)")
74+
75+
#### Model Bias Monitoring Job Parameters
76+
77+
- `model_bias_baseline_s3_uri`: S3 URI for baseline model bias statistics
78+
- `model_bias_output_s3_uri`: S3 URI for model bias monitoring output
79+
- `model_bias_ground_truth_s3_uri`: S3 URI for ground truth data
80+
- `model_bias_instance_count`: Number of ML compute instances (default: 1)
81+
- `model_bias_instance_type`: ML compute instance type (default: "ml.m5.large")
82+
- `model_bias_instance_volume_size_in_gb`: Size of ML storage volume in GB (default: 20)
83+
- `model_bias_max_runtime_in_seconds`: Maximum runtime in seconds (default: 1800)
84+
- `model_bias_features_attribute`: Attributes of input data that are input features
85+
- `model_bias_inference_attribute`: Attribute representing the ground truth label
86+
- `model_bias_probability_attribute`: Attribute representing class probability
87+
- `model_bias_probability_threshold_attribute`: Threshold for class probability evaluation
88+
- `model_bias_schedule_expression`: Cron expression for monitoring schedule (default: "cron(0 * ? * * *)")
89+
90+
#### Model Explainability Monitoring Job Parameters
91+
92+
- `model_explainability_baseline_s3_uri`: S3 URI for baseline model explainability statistics
93+
- `model_explainability_output_s3_uri`: S3 URI for model explainability monitoring output
94+
- `model_explainability_instance_count`: Number of ML compute instances (default: 1)
95+
- `model_explainability_instance_type`: ML compute instance type (default: "ml.m5.large")
96+
- `model_explainability_instance_volume_size_in_gb`: Size of ML storage volume in GB (default: 20)
97+
- `model_explainability_max_runtime_in_seconds`: Maximum runtime in seconds (default: 1800)
98+
- `model_explainability_features_attribute`: Attributes of input data that are input features
99+
- `model_explainability_inference_attribute`: Attribute representing the ground truth label
100+
- `model_explainability_probability_attribute`: Attribute representing class probability
101+
- `model_explainability_schedule_expression`: Cron expression for monitoring schedule (default: "cron(0 * ? * * *)")
104102

105103
### Sample manifest declaration
106104

@@ -109,53 +107,20 @@ name: monitoring
109107
path: modules/sagemaker/sagemaker-model-monitoring
110108
parameters:
111109
- name: sagemaker_project_id
112-
value: dummy123
110+
value: SF-DEMO-xgb-churn-pred-model-monitor-2025-10-23-19-11-16
113111
- name: sagemaker_project_name
114-
value: dummy123
115-
- name: model_package_arn
116-
value: arn:aws:sagemaker:<region>:<account>:model-package/<package_name>/1
112+
value: SF-DEMO-xgb-churn-pred-model-monitor-2025-10-23-19-11-16
117113
- name: model_bucket_arn
118-
value: arn:aws:s3:::<bucket name>
119-
- name: enable-data-quality-monitor
120-
value: True
121-
- name: data-quality-checkstep-output-prefix
122-
value: model-training-run-1234/dataqualitycheckstep
123-
- name: data-quality-output-prefix
124-
value: model-training-run-1234/monitor/dataqualityoutput
114+
value: arn:aws:s3:::sagemaker-us-east-2-<REDACTED>
125115
- name: endpoint_name
126-
valueFrom:
127-
moduleMetadata:
128-
group: endpoints
129-
name: endpoint
130-
key: EndpointName
131-
- name: security_group_id
132-
valueFrom:
133-
moduleMetadata:
134-
group: endpoints
135-
name: endpoint
136-
key: SecurityGroupId
137-
- name: kms_key_id
138-
valueFrom:
139-
moduleMetadata:
140-
group: endpoints
141-
name: endpoint
142-
key: KmsKeyId
143-
- name: subnet_ids
144-
valueFrom:
145-
moduleMetadata:
146-
group: networking
147-
name: networking
148-
key: PrivateSubnetIds
149-
```
150-
151-
### Module Metadata Outputs
116+
value: DEMO-xgb-churn-pred-model-monitor-2025-10-23-19-11-16
117+
- name: enable_data_quality_monitor
118+
value: true
119+
- name: data-quality-schedule-expression
120+
value: cron(0 * ? * * *)
121+
- name: data_quality_baseline_s3_uri
122+
value: s3://sagemaker-us-east-2-<REDACTED>/SF-DEMO-xgb-churn-pred-model-monitor/baseline
123+
- name: data_quality_output_s3_uri
124+
value: s3://sagemaker-us-east-2-<REDACTED>/SF-DEMO-xgb-churn-pred-model-monitor/output
152125

153-
- `ModelExecutionRoleArn`: SageMaker Model Execution IAM role ARN
154-
- `ModelName`: SageMaker Model name
155-
- `ModelPackageArn`: SageMaker Model package ARN
156-
- `EndpointName`: SageMaker Endpoint name
157-
- `EndpointUrl`: SageMaker Endpoint Url
158-
159-
#### Output Example
160-
161-
N/A
126+
```

‎modules/sagemaker/sagemaker-model-monitoring/deployspec.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ deploy:
55
commands:
66
- env
77
# Install whatever additional build libraries
8-
- npm install -g aws-cdk@2.126.0
8+
- npm install -g aws-cdk@2.1031.0
99
- pip install -r requirements.txt
1010
build:
1111
commands:
@@ -17,7 +17,7 @@ destroy:
1717
install:
1818
commands:
1919
# Install whatever additional build libraries
20-
- npm install -g aws-cdk@2.126.0
20+
- npm install -g aws-cdk@2.1031.0
2121
- pip install -r requirements.txt
2222
build:
2323
commands:

‎modules/sagemaker/sagemaker-model-monitoring/requirements.txt‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
1-
aws-cdk-lib==2.126.0
2-
cdk-nag==2.28.27
3-
sagemaker==2.218.0
4-
pydantic~=2.5.3
5-
pydantic-settings~=2.0.3
1+
aws-cdk-lib==2.221.0
2+
cdk-nag==2.37.55
3+
sagemaker==2.253.1
4+
pydantic~=2.12.3
5+
pydantic-settings~=2.11.0

0 commit comments

Comments
 (0)