Skip to content

Commit 6eb5f0e

Browse files
alyssacgoinsalyssacgoins
committed
website: document KFP MLflow plugin
Signed-off-by: Alyssa Goins <agoins@redhat.com>
1 parent f9e5102 commit 6eb5f0e

3 files changed

Lines changed: 216 additions & 0 deletions

File tree

‎content/en/docs/components/pipelines/operator-guides/mlflow-plugin.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
+++
2+
title = "MLflow Plugin Configuration"
3+
weight = 4
4+
+++
5+
6+
## KFP MLflow Plugin
7+
8+
Kubeflow Pipelines supports an MLflow plugin that enables automatic experiment tracking.
9+
When enabled, the plugin registers each KFP run with MLflow, allowing users to view and analyze experiments, and their pipeline runs in the MLflow UI.
10+
11+
## Prerequisites
12+
13+
- Admin access to a Kubernetes cluster
14+
- Kubeflow Pipelines installed on your cluster
15+
- kubectl configured to access your cluster
16+
17+
## Deploying MLflow
18+
19+
You need an MLflow instance running on your Kubernetes cluster. One recommended option is the [OpenDataHub MLflow Operator](https://github.com/opendatahub-io/mlflow-operator#mlflow-operator), which simplifies MLflow deployment and management on Kubernetes.
20+
21+
To deploy using the MLflow Operator:
22+
23+
1. Follow the operator [deployment instructions](https://github.com/opendatahub-io/mlflow-operator#to-deploy-on-the-cluster)
24+
2. Note the MLflow service endpoint and port (typically `https://<service-name>:8443`)
25+
3. If using TLS, retrieve the CA certificate for your MLflow deployment
26+
27+
## Security and TLS
28+
29+
The KFP MLflow plugin uses TLS to secure communication with the MLflow server. Configure the following TLS settings in your [API server configuration](#configuring-the-kfp-mlflow-plugin):
30+
31+
- **caBundlePath**: Path to the CA certificate bundle used to verify the MLflow server's certificate
32+
- **insecureSkipVerify**: Set to `false` (default) to enforce certificate verification
33+
34+
For production deployments, always use valid TLS certificates and keep `insecureSkipVerify` set to `false`.
35+
36+
## MLflow Experiments
37+
38+
When the plugin is enabled, pipeline runs are logged to MLflow experiments:
39+
40+
- **Default experiment**: If no experiment is specified, runs are logged to an experiment named "default" (created automatically if it doesn't exist)
41+
- **Custom experiments**: Users can specify a custom experiment name when submitting a run. KFP will create the experiment if it doesn't already exist
42+
- **RBAC**: For enhanced security, you can enforce Kubernetes RBAC for MLflow requests using the [Kubeflow MLflow extension](https://github.com/kubeflow/mlflow-integration#mlflow-kubeflow-integration)
43+
44+
## MLflow Workspaces
45+
46+
MLflow workspaces provide an optional organizational layer and permission framework, similar to Kubernetes namespaces. A workspace can contain multiple experiments, but each experiment belongs to only one workspace.
47+
48+
**Configuration:**
49+
- When `authType` is set to `kubernetes`, workspaces are **enabled by default**
50+
- For other authentication types, workspaces are **disabled by default**
51+
- You can explicitly control this behavior by setting `workspacesEnabled` to `true` or `false` in the plugin configuration
52+
53+
**Requirements:**
54+
To use MLflow workspaces, you must deploy the [Kubeflow MLflow extension](https://github.com/kubeflow/mlflow-integration#mlflow-kubeflow-integration), which adds Kubernetes-native authentication and multi-tenancy support to MLflow.
55+
56+
## Configuring the KFP MLflow Plugin
57+
58+
To enable the MLflow plugin, add the following configuration to your KFP API server `config.json` file:
59+
60+
```json
61+
{
62+
"plugins": {
63+
"mlflow": {
64+
"endpoint": "<schema>://<mlflow-service>:<mlflow-port>",
65+
"timeout": 30,
66+
"tls": {
67+
"insecureSkipVerify": <boolean>,
68+
"caBundlePath": "<path-to-ca-bundle>"
69+
},
70+
"settings": {}
71+
}
72+
}
73+
}
74+
```
75+
76+
### Configuration Values
77+
78+
Replace the placeholder values with your deployment-specific values:
79+
80+
- **endpoint**: The full URL of your MLflow server (e.g., `https://mlflow-service.mlflow.svc.cluster.local:8443`)
81+
- **timeout**: Timeout in seconds for MLflow API calls (default: `30`)
82+
- **tls**: TLS configuration for MLflow communication
83+
- **insecureSkipVerify**: Set to `true` to skip TLS certificate verification (not recommended for production)
84+
- **caBundlePath**: Path to the CA certificate for your MLflow server. If MLflow uses a custom/internal CA, create a ConfigMap containing the CA certificate, mount it in the API server pod (for example, at /kfp/certs/ca.crt), and set plugins.mlflow.tls.caBundlePath to that mounted path.
85+
- **settings**: See the [user guide](/docs/components/pipelines/user-guides/integrations/mlflow-plugin/#configuring-plugin-settings) for a complete list of MLflow plugin settings
86+
87+
### Restart the API Server
88+
89+
After updating the configuration, restart the KFP API server to apply the changes:
90+
91+
```bash
92+
kubectl rollout restart deployment/ml-pipeline -n kubeflow
93+
```
94+
95+
Verify the plugin is enabled by checking the API server logs:
96+
97+
```bash
98+
kubectl logs -n kubeflow deployment/ml-pipeline | grep mlflow
99+
```
100+
101+
You should see messages indicating the MLflow plugin has been initialized successfully.

‎content/en/docs/components/pipelines/user-guides/integrations/_index.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
+++
2+
title = "Integrations"
3+
description = "Kubeflow Pipelines supports plugins with external services."
4+
weight = 5
5+
+++

‎content/en/docs/components/pipelines/user-guides/integrations/mlflow-plugin.md‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
+++
2+
title = "MLflow Plugin for KFP"
3+
description = "Deploy MLflow on your cluster and configure the KFP MLflow plugin to use it."
4+
weight = 1
5+
+++
6+
7+
## MLflow & Experiment Tracking
8+
9+
[MLflow](https://mlflow.org/) is an open-source platform for managing the end-to-end machine learning lifecycle.
10+
It provides tools for experiment tracking, model versioning, and deployment.
11+
The MLflow UI is a powerful interface for organizing, querying, and visualizing experiments and their corresponding runs, including run parameters and scalar metric artifacts.
12+
13+
14+
## Integration Overview
15+
16+
The KFP MLflow integration enables automatic experiment tracking for your Kubeflow Pipelines. When enabled, each pipeline run is automatically registered in MLflow, allowing you to:
17+
18+
- Track pipeline runs as MLflow experiments
19+
- View pipeline parameters and metrics in the MLflow UI
20+
- Organize related runs under custom experiments
21+
- Maintain a complete audit trail of your ML workflows
22+
23+
This integration bridges the gap between pipeline orchestration and experiment tracking, giving you a unified view of your machine learning workflows.
24+
All tracked data is viewable in the MLflow UI, organized by experiments. You can specify a custom experiment name for each run, or use the default experiment configured by your administrator.
25+
This page provides detailed instructions on how to set up the MLflow plugin for Kubeflow Pipelines at the user level, once it has been deployed and configured at the [admin level](/docs/components/pipelines/operator-guides/mlflow-plugin/).
26+
27+
## Prerequisites
28+
29+
Before using the MLflow plugin, ensure your cluster administrator has:
30+
- Deployed MLflow on your cluster
31+
- Enabled and configured the KFP MLflow plugin
32+
33+
See the [operator guide](/docs/components/pipelines/operator-guides/mlflow-plugin/) for setup instructions.
34+
35+
## User Configuration
36+
37+
To use the MLflow plugin with your pipelines, configure the plugin in your KFP API `config.json` file:
38+
39+
```json
40+
{
41+
"plugins": {
42+
"mlflow": {
43+
"endpoint": "<schema>://<mlflow-service>:<mlflow-port>",
44+
"timeout": 30,
45+
"tls": {
46+
"insecureSkipVerify": <boolean>,
47+
"caBundlePath": "<path-to-ca-bundle>"
48+
},
49+
"settings": {}
50+
}
51+
}
52+
}
53+
```
54+
55+
## Configuring Plugin Settings
56+
See the [operator guide](/docs/components/pipelines/operator-guides/mlflow-plugin/#configuring-the-kfp-mlflow-plugin) for instructions on configuring the `endpoint`, `timeout` and `TLS` fields.
57+
The following `settings` fields are available for further configuration:
58+
59+
#### authType
60+
Authentication method for the MLflow server. Options:
61+
- `kubernetes` (default): Use Kubernetes-based authentication
62+
- `bearer`: Use bearer token authentication
63+
- `basic-auth`: Use username/password authentication
64+
- `none`: No authentication
65+
66+
#### credentialSecretRef
67+
When using `bearer` or `basic-auth` authentication, this field references a Kubernetes secret containing the credentials.
68+
69+
#### workspacesEnabled
70+
Enable MLflow workspaces for multi-tenant organization.
71+
When `authType` is `kubernetes`, this defaults to `true`. Otherwise, it defaults to `false`.
72+
Learn more about workspaces in MLflow in the [operator guide](/docs/components/pipelines/operator-guides/mlflow-plugin/#mlflow-workspaces).
73+
74+
#### defaultExperimentName
75+
Default experiment name when none is specified for a pipeline run. Default is `"default"`.
76+
77+
#### experimentDescription
78+
Default description for newly created experiments. Default is `"Created by Kubeflow Pipelines"`.
79+
80+
#### kfpBaseURL
81+
If set, this URL is added as a tag to MLflow runs for traceability back to KFP.
82+
83+
#### mlflowBaseURL
84+
Base URL for linking to the MLflow UI from pipeline runs.
85+
86+
#### mlflowUIPathPrefix
87+
Path prefix for constructing MLflow UI links.
88+
89+
#### injectUserEnvVars
90+
When set to `true`, injects MLflow environment variables into user containers, enabling a component's user code to interact with MLflow directly:
91+
92+
**General variables:**
93+
- `MLFLOW_RUN_ID`: The MLflow run ID for the pipeline
94+
- `MLFLOW_TRACKING_URI`: The MLflow endpoint URL
95+
- `MLFLOW_EXPERIMENT_ID`: The MLflow experiment ID
96+
- `MLFLOW_WORKSPACE`: The MLflow workspace name (if workspaces are enabled)
97+
98+
**Authentication variables:**
99+
- `MLFLOW_TRACKING_TOKEN`: Bearer token (when using bearer auth)
100+
- `MLFLOW_TRACKING_USERNAME` and `MLFLOW_TRACKING_PASSWORD`: Credentials (when using basic auth)
101+
102+
## Using the Plugin
103+
104+
Once configured, pipeline runs are automatically registered in MLflow. You can optionally specify a custom experiment name when submitting a run through the KFP UI or SDK. If no experiment is specified, runs are logged to the default experiment.
105+
106+
### Visualizing Pipeline Artifacts with MLflow
107+
Current plugin artifact support extends to scalar Metric artifacts, which are logged to MLflow runs and can be visualized in the MLflow console.
108+
109+
### Visualizing Pipeline Parameters with MLflow
110+
Pipeline parameters are automatically logged to MLflow runs, making it easy to track and analyze the configuration of your pipeline runs.

0 commit comments

Comments
 (0)