great_expectations on Azure Machine Learning

[oleflere]

Hi,

I would like to use great_expectations as part of ML pipelines. I tried it in an Azure Machine Learning environment, but when running the pipeline it seems that the great_expectations library is being downloaded, but not being installed. My environment is defined by:

from azureml.core.runconfig import RunConfiguration
from azureml.core.conda_dependencies import CondaDependencies
aml_run_config = RunConfiguration()
aml_run_config.target = compute_target
#aml_run_config.environment.docker.enabled = True
#aml_run_config.environment.docker.base_image = "mcr.microsoft.com/azureml/base:latest"
aml_run_config.environment.python.user_managed_dependencies = False
conda_dep = CondaDependencies()
conda_dep.add_conda_package("python=3.8")
conda_dep.add_conda_package("pandas")
conda_dep.add_conda_package("packaging")
conda_dep.add_conda_package('pip')
#conda_dep.add_pip_package("azureml-defaults")
conda_dep.add_pip_package("great_expectations")
aml_run_config.environment.python.conda_dependencies = conda_dep

Is there anybody who has already experience with great_expectations on Azure ML and who knows what is going wrong and how to solve it?

Kind regards,

Olivier Leflere