Skip to content

[BUG]: MlflowException when running the E2E example #3458

Open
@vladimir-sol

Description

@vladimir-sol

Contact Details [Optional]

No response

System Information

ZENML_LOCAL_VERSION: 0.74.0
ZENML_SERVER_VERSION: 0.74.0
ZENML_SERVER_DATABASE: sqlite
ZENML_SERVER_DEPLOYMENT_TYPE: local
ZENML_CONFIG_DIR: /Users/username/Library/Application Support/zenml
ZENML_LOCAL_STORE_DIR: /Users/username/Library/Application Support/zenml/local_stores
ZENML_SERVER_URL: http://127.0.0.1:8237
ZENML_ACTIVE_REPOSITORY_ROOT: /Users/username/ws/zenml/examples/zenml_batch_e2e
PYTHON_VERSION: 3.12.9
ENVIRONMENT: native
SYSTEM_INFO: {'os': 'mac', 'mac_version': '15.3.2'}
ACTIVE_WORKSPACE: default
ACTIVE_STACK: e2e_template_stack
ACTIVE_USER: default
TELEMETRY_STATUS: enabled
ANALYTICS_CLIENT_ID: 18fea8fe-c5a1-44bb-b3bb-14abafce4a97
ANALYTICS_USER_ID: fa65f2bb-b279-4e1d-b0e9-f66f0f68bd4c
ANALYTICS_SERVER_ID: 18fea8fe-c5a1-44bb-b3bb-14abafce4a97
INTEGRATIONS: ['airflow', 'aws', 'bitbucket', 'numpy', 'pandas', 'evidently', 'kaniko', 'kubeflow', 'kubernetes', 'mlflow',
'pigeon', 'pillow', 'pytorch', 's3', 'scipy', 'sklearn', 'slack', 'tekton']
PACKAGES: {'deprecated': '1.2.18', 'gitpython': '3.1.44', 'mako': '1.3.9', 'markdown': '3.7', 'markupsafe': '3.0.2', 'pyjwt':
'2.7.0', 'pymysql': '1.1.1', 'pyyaml': '6.0.2', 'sqlalchemy-utils': '0.41.2', 'send2trash': '1.8.3', 'accelerate': '1.4.0',
'aiobotocore': '2.21.1', 'aiohappyeyeballs': '2.6.1', 'aiohttp': '3.11.14', 'aioitertools': '0.12.0', 'aiosignal': '1.3.2',
'alembic': '1.8.1', 'annotated-types': '0.7.0', 'antlr4-python3-runtime': '4.9.3', 'anyio': '4.8.0', 'appdirs': '1.4.4', 'appnope':
'0.1.4', 'argon2-cffi': '23.1.0', 'argon2-cffi-bindings': '21.2.0', 'argparse': '1.4.0', 'arrow': '1.3.0', 'asttokens': '3.0.0',
'async-lru': '2.0.4', 'attrs': '23.2.0', 'aws-profile-manager': '0.7.3', 'babel': '2.17.0', 'bcrypt': '4.0.1', 'beautifulsoup4':
'4.13.3', 'bleach': '6.2.0', 'blinker': '1.9.0', 'boto3': '1.37.1', 'botocore': '1.37.1', 'cachetools': '5.5.1', 'certifi':
'2025.1.31', 'cffi': '1.17.1', 'charset-normalizer': '3.4.1', 'click': '8.1.3', 'cloudpickle': '2.2.1', 'colorama': '0.4.6',
'comm': '0.2.2', 'configparser': '7.1.0', 'contourpy': '1.3.1', 'copier': '9.6.0', 'cryptography': '44.0.1', 'cycler': '0.12.1',
'databricks-sdk': '0.47.0', 'datasets': '3.3.1', 'debugpy': '1.8.12', 'decorator': '5.2.1', 'defusedxml': '0.7.1', 'dill': '0.3.9',
'distro': '1.9.0', 'docker': '7.1.0', 'docstring-parser': '0.16', 'dunamai': '1.23.1', 'durationpy': '0.9', 'dynaconf': '3.2.10',
'evidently': '0.4.22', 'executing': '2.2.0', 'faker': '37.0.2', 'fastapi': '0.110.0', 'fastjsonschema': '2.21.1', 'filelock':
'3.17.0', 'flask': '3.1.0', 'fonttools': '4.56.0', 'fqdn': '1.5.1', 'frozenlist': '1.5.0', 'fsspec': '2025.3.0', 'funcy': '2.0',
'gitdb': '4.0.12', 'google-api-core': '2.24.2', 'google-auth': '2.38.0', 'google-cloud-core': '2.4.3', 'google-cloud-storage':
'2.19.0', 'google-crc32c': '1.7.0', 'google-pasta': '0.2.0', 'google-resumable-media': '2.7.2', 'googleapis-common-protos':
'1.69.2', 'graphene': '3.4.3', 'graphql-core': '3.2.6', 'graphql-relay': '3.2.0', 'gunicorn': '23.0.0', 'h11': '0.14.0',
'httpcore': '1.0.7', 'httptools': '0.6.4', 'httpx': '0.28.1', 'huggingface-hub': '0.29.1', 'idna': '3.10', 'imageio': '2.37.0',
'importlib-metadata': '6.11.0', 'ipinfo': '5.1.1', 'ipykernel': '6.29.5', 'ipython': '9.0.2', 'ipython-pygments-lexers': '1.1.1',
'ipywidgets': '8.1.5', 'isoduration': '20.11.0', 'iterative-telemetry': '0.0.10', 'itsdangerous': '2.2.0', 'jedi': '0.19.2',
'jinja2': '3.1.6', 'jinja2-ansible-filters': '1.3.2', 'jinja2-time': '0.2.0', 'jmespath': '1.0.1', 'joblib': '1.4.2', 'json5':
'0.10.0', 'jsonpointer': '3.0.0', 'jsonschema': '4.23.0', 'jsonschema-specifications': '2024.10.1', 'jupyter-client': '8.6.3',
'jupyter-core': '5.7.2', 'jupyter-events': '0.12.0', 'jupyter-lsp': '2.2.5', 'jupyter-server': '2.15.0',
'jupyter-server-terminals': '0.5.3', 'jupyterlab': '4.3.5', 'jupyterlab-pygments': '0.3.0', 'jupyterlab-server': '2.27.3',
'jupyterlab-widgets': '3.0.13', 'kfp': '2.12.1', 'kfp-kubernetes': '1.4.0', 'kfp-pipeline-spec': '0.6.0', 'kfp-server-api':
'2.4.0', 'kiwisolver': '1.4.8', 'kubernetes': '25.3.0', 'lazy-loader': '0.4', 'litestar': '2.15.1', 'litestar-htmx': '0.4.1',
'markdown-it-py': '3.0.0', 'matplotlib': '3.10.1', 'matplotlib-inline': '0.1.7', 'mdurl': '0.1.2', 'mistune': '3.1.2', 'mlflow':
'2.21.0', 'mlflow-skinny': '2.21.0', 'mock': '4.0.3', 'mpmath': '1.3.0', 'msgspec': '0.19.0', 'multidict': '6.2.0', 'multipart':
'1.2.1', 'multiprocess': '0.70.17', 'mypy-extensions': '1.0.0', 'narwhals': '1.31.0', 'nbclient': '0.10.2', 'nbconvert': '7.16.6',
'nbformat': '5.10.4', 'nest-asyncio': '1.6.0', 'networkx': '3.4.2', 'nltk': '3.9.1', 'notebook': '7.3.2', 'notebook-shim': '0.2.4',
'numpy': '1.26.4', 'oauthlib': '3.2.2', 'omegaconf': '2.3.0', 'opentelemetry-api': '1.31.1', 'opentelemetry-sdk': '1.31.1',
'opentelemetry-semantic-conventions': '0.52b1', 'orjson': '3.10.15', 'overrides': '7.7.0', 'packaging': '24.2', 'pandas': '2.2.3',
'pandocfilters': '1.5.1', 'parso': '0.8.4', 'passlib': '1.7.4', 'pathos': '0.3.3', 'pathspec': '0.12.1', 'patsy': '1.0.1',
'pexpect': '4.9.0', 'pillow': '11.1.0', 'pip': '25.0.1', 'platformdirs': '4.3.7', 'plotly': '6.0.1', 'plumbum': '1.9.0',
'polyfactory': '2.20.0', 'pox': '0.3.5', 'ppft': '1.7.6.9', 'prometheus-client': '0.21.1', 'prompt-toolkit': '3.0.50', 'propcache':
'0.3.0', 'proto-plus': '1.26.1', 'protobuf': '4.25.6', 'psutil': '7.0.0', 'ptyprocess': '0.7.0', 'pure-eval': '0.2.3', 'pyarrow':
'19.0.1', 'pyasn1': '0.6.1', 'pyasn1-modules': '0.4.1', 'pycparser': '2.22', 'pydantic': '2.8.2', 'pydantic-core': '2.20.1',
'pydantic-settings': '2.8.1', 'pygments': '2.19.1', 'pyparsing': '3.2.1', 'python-dateutil': '2.9.0.post0', 'python-dotenv':
'1.0.1', 'python-json-logger': '3.2.1', 'python-multipart': '0.0.20', 'python-rapidjson': '1.14', 'pytz': '2025.1',
'pyyaml-include': '1.4.1', 'pyzmq': '26.2.1', 'questionary': '2.1.0', 'referencing': '0.36.2', 'regex': '2024.11.6', 'requests':
'2.32.3', 'requests-file': '2.1.0', 'requests-oauthlib': '2.0.0', 'requests-toolbelt': '1.0.0', 'rfc3339-validator': '0.1.4',
'rfc3986-validator': '0.1.1', 'rich': '13.9.4', 'rich-click': '1.8.8', 'rpds-py': '0.22.3', 'rsa': '4.9', 'ruff': '0.11.2', 's3fs':
'2025.3.0', 's3transfer': '0.11.3', 'safetensors': '0.5.2', 'sagemaker': '2.239.3', 'sagemaker-core': '1.0.23', 'schema': '0.7.7',
'scikit-image': '0.25.2', 'scikit-learn': '1.6.1', 'scipy': '1.15.2', 'secure': '0.3.0', 'sentencepiece': '0.2.0', 'setuptools':
'78.0.1', 'shellingham': '1.5.4', 'six': '1.17.0', 'slack-sdk': '3.30.0', 'smdebug-rulesconfig': '1.0.1', 'smmap': '5.0.2',
'sniffio': '1.3.1', 'soupsieve': '2.6', 'sqlalchemy': '2.0.39', 'sqlmodel': '0.0.18', 'sqlparse': '0.5.3', 'stack-data': '0.6.3',
'starlette': '0.36.3', 'statsmodels': '0.14.4', 'sympy': '1.13.1', 'tabulate': '0.9.0', 'tblib': '3.0.0', 'tenacity': '9.0.0',
'terminado': '0.18.1', 'threadpoolctl': '3.6.0', 'tifffile': '2025.3.13', 'tinycss2': '1.4.0', 'tldextract': '5.1.3', 'tokenizers':
'0.21.0', 'torch': '2.6.0', 'tornado': '6.4.2', 'tqdm': '4.67.1', 'traitlets': '5.14.3', 'transformers': '4.49.0', 'typer':
'0.15.2', 'types-python-dateutil': '2.9.0.20241206', 'typing-extensions': '4.12.2', 'typing-inspect': '0.9.0', 'tzdata': '2025.1',
'ujson': '5.10.0', 'uri-template': '1.3.0', 'urllib3': '1.26.20', 'uvicorn': '0.34.0', 'uvloop': '0.21.0', 'watchdog': '6.0.0',
'watchfiles': '1.0.4', 'wcwidth': '0.2.13', 'webcolors': '24.11.1', 'webencodings': '0.5.1', 'websocket-client': '1.8.0',
'websockets': '15.0', 'werkzeug': '3.1.3', 'widgetsnbextension': '4.0.13', 'wrapt': '1.17.2', 'xxhash': '3.5.0', 'yarl': '1.18.3',
'zenml': '0.74.0', 'zipp': '3.21.0', 'autocommand': '2.2.2', 'backports.tarfile': '1.2.0', 'inflect': '7.3.1',
'jaraco.collections': '5.1.0', 'jaraco.context': '5.3.0', 'jaraco.functools': '4.0.1', 'jaraco.text': '3.12.1', 'more-itertools':
'10.3.0', 'tomli': '2.0.1', 'typeguard': '4.3.0', 'wheel': '0.45.1'}

CURRENT STACK

Name: e2e_template_stack
ID: 9114bf8e-b6cc-4fcb-8184-74b39c12c3ba
User: default / fa65f2bb-b279-4e1d-b0e9-f66f0f68bd4c
Workspace: default / 0c68f923-9eb6-4453-87f8-4a712fc88cdf

ORCHESTRATOR: default

Name: default
ID: 01d16d47-7656-4184-977d-d9972abafb1e
Type: orchestrator
Flavor: local
Configuration: {}
Workspace: default / 0c68f923-9eb6-4453-87f8-4a712fc88cdf

MODEL_DEPLOYER: mlflow_local_e2e_template_stack

Name: mlflow_local_e2e_template_stack
ID: 244a1d97-d61b-45f5-bab4-5225f95cd57b
Type: model_deployer
Flavor: mlflow
Configuration: {}
User: default / fa65f2bb-b279-4e1d-b0e9-f66f0f68bd4c
Workspace: default / 0c68f923-9eb6-4453-87f8-4a712fc88cdf

ARTIFACT_STORE: default

Name: default
ID: 6d22c082-b7ce-47d3-bf1b-6d7bb9f37611
Type: artifact_store
Flavor: local
Configuration: {}
Workspace: default / 0c68f923-9eb6-4453-87f8-4a712fc88cdf

EXPERIMENT_TRACKER: mlflow_local_e2e_template_stack

Name: mlflow_local_e2e_template_stack
ID: 79daf541-5af1-4efc-90fc-efbae0a61982
Type: experiment_tracker
Flavor: mlflow
Configuration: {}
User: default / fa65f2bb-b279-4e1d-b0e9-f66f0f68bd4c
Workspace: default / 0c68f923-9eb6-4453-87f8-4a712fc88cdf

DATA_VALIDATOR: evidently_e2e_template_stack

Name: evidently_e2e_template_stack
ID: a41bd0fc-652b-4956-b8c4-72b7c36fb078
Type: data_validator
Flavor: evidently
Configuration: {}
User: default / fa65f2bb-b279-4e1d-b0e9-f66f0f68bd4c
Workspace: default / 0c68f923-9eb6-4453-87f8-4a712fc88cdf

MODEL_REGISTRY: mlflow_local_e2e_template_stack

Name: mlflow_local_e2e_template_stack
ID: f2b7231e-b34f-465e-8b6a-b79159376962
Type: model_registry
Flavor: mlflow
Configuration: {}
User: default / fa65f2bb-b279-4e1d-b0e9-f66f0f68bd4c
Workspace: default / 0c68f923-9eb6-4453-87f8-4a712fc88cdf

What happened?

When running the README code snippet here, specifically python run.py, the model_trainer step throws

MlflowException: Could not find an "MLmodel" configuration file at 
"file:///Users/username/Library/Application Support/zenml/local_stores/6d22c082-b7ce-47d3-bf1b-6d7bb9f37611/mlruns/622276372486226458/0cb24284d6534847a2675900c0600562/artifacts/model"

Upon manual inspection, the configuration file is there and is a valid yaml:

% ls -l "/Users/username/Library/Application Support/zenml/local_stores/6d22c082-b7ce-47d3-bf1b-6d7bb9f37611/mlruns/622276372486226458/0cb24284d6534847a2675900c0600562/artifacts/model" 
total 288
-rw-r--r--  1 username staff    2779 Mar 24 09:26 MLmodel
-rw-r--r--  1 username  staff     230 Mar 24 09:26 conda.yaml
-rw-r--r--  1 username  staff  127372 Mar 24 09:26 model.pkl
-rw-r--r--  1 username  staff     122 Mar 24 09:26 python_env.yaml
-rw-r--r--  1 username  staff     109 Mar 24 09:26 requirements.txt

% yq "/Users/username/Library/Application Support/zenml/local_stores/6d22c082-b7ce-47d3-bf1b-6d7bb9f37611/mlruns/622276372486226458/0cb24284d6534847a2675900c0600562/artifacts/model/MLmodel" 
artifact_path: model
flavors:
  python_function:
    env:
      conda: conda.yaml
      virtualenv: python_env.yaml
    loader_module: mlflow.sklearn
    model_path: model.pkl
    predict_fn: predict
    python_version: 3.12.9
  sklearn:
    code: null
    pickled_model: model.pkl
    serialization_format: cloudpickle
    sklearn_version: 1.6.1
is_signature_from_type_hint: false
mlflow_version: 2.21.0
...

HOWEVER, using the same file path with the file:// scheme doesn't appear to be working on Mac and may be the source of the error:

% yq "file:///Users/username/Library/Application Support/zenml/local_stores/6d22c082-b7ce-47d3-bf1b-6d7bb9f37611/mlruns/622276372486226458/0cb24284d6534847a2675900c0600562/artifacts/model/MLmodel"
... not found
# monkey patched into the throwing .py file inside mlflow (mlflow/models/model.py:736)
print(path)
print(os.path.exists(path))
print(urlparse(path).path)  # spaces added below for readability
print(os.path.exists(urlparse(path).path))
# outputs:
1. file:///Users/username/Library/Application Support/zenml/local_stores/6d22c082-b7ce-47d3-bf1b-6d7bb9f37611/mlruns/622276372486226458/982fa6dbe78045d2afc664d5b67fdc36/artifacts/model
2. False
3.        /Users/username/Library/Application Support/zenml/local_stores/6d22c082-b7ce-47d3-bf1b-6d7bb9f37611/mlruns/622276372486226458/982fa6dbe78045d2afc664d5b67fdc36/artifacts/model
4. True

Reproduction steps

A. Follow the commands from the introduction code snippet https://github.com/zenml-io/zenml/tree/main/examples/e2e#-introduction
B. Same as A, but preceded by template instantiation as described in https://github.com/zenml-io/template-e2e-batch?tab=readme-ov-file#-generate-a-zenml-project

No modification to the code was done otherwise.

Relevant log output

% python run.py || date
ArtifactConfig(..., is_model_artifact=True) is deprecated and will be removed soon. Use ArtifactConfig(..., artifact_type=ArtifactType.MODEL) instead. For more info: https://docs.zenml.io/user-guide/starter-guide/manage-artifacts
ArtifactConfig(..., is_deployment_artifact=True) is deprecated and will be removed soon. Use ArtifactConfig(..., artifact_type=ArtifactType.SERVICE) instead. For more info: https://docs.zenml.io/user-guide/starter-guide/manage-artifacts
Initiating a new run for the pipeline: e2e_use_case_training.
Using user: default
Using stack: e2e_template_stack
  orchestrator: default
  model_deployer: mlflow_local_e2e_template_stack
  artifact_store: default
  experiment_tracker: mlflow_local_e2e_template_stack
  data_validator: evidently_e2e_template_stack
  model_registry: mlflow_local_e2e_template_stack
Dashboard URL for Pipeline Run: http://127.0.0.1:8237/runs/86a1edeb-7fb4-4bc1-8e52-0444fb5aa732
Models can be viewed in the dashboard using ZenML Pro. Sign up for a free trial at https://www.zenml.io/pro/
Step data_loader has started.
Dataset with 541 records loaded!
Step data_loader has finished in 0.433s.
Step train_data_splitter has started.
Step train_data_splitter has finished in 0.735s.
Step train_data_preprocessor has started.
/Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/sklearn/pipeline.py:62: FutureWarning:

This Pipeline instance is not fitted yet. Call 'fit' with appropriate arguments before using other methods such as transform, predict, etc. This will raise an error in 1.8 instead of the current warning.

Step train_data_preprocessor has finished in 0.996s.
Step hp_tuning_search_decision_tree has started.
Running Hyperparameter tuning...
Step hp_tuning_search_decision_tree has finished in 8.717s.
Step hp_tuning_search_random_forest has started.
Running Hyperparameter tuning...
Step hp_tuning_search_random_forest has finished in 1.348s.
Using cached version of step hp_tuning_select_best_model.
Step model_trainer has started.
Training model RandomForestClassifier(criterion='entropy', max_depth=10, min_samples_leaf=2,
                       n_estimators=50)...
MLflow model registry does not take a version as an argument. Registering a new version for the model 'e2e_use_case' a version will be assigned automatically.
HELLO WORLD
file:///Users/username/Library/Application Support/zenml/local_stores/6d22c082-b7ce-47d3-bf1b-6d7bb9f37611/mlruns/622276372486226458/0cb24284d6534847a2675900c0600562/artifacts/model
/Users/username/Library/Application Support/zenml/local_stores/6d22c082-b7ce-47d3-bf1b-6d7bb9f37611/mlruns/622276372486226458/0cb24284d6534847a2675900c0600562/artifacts/model
True
Detected failure hook. Running...
/Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/mlflow/fastai/__init__.py:68: FutureWarning:

fastai flavor is deprecated and will be removed in MLflow 3.0.

Failed to run step model_trainer after 1 retries. Exiting.
Could not find an "MLmodel" configuration file at "file:///Users/username/Library/Application Support/zenml/local_stores/6d22c082-b7ce-47d3-bf1b-6d7bb9f37611/mlruns/622276372486226458/0cb24284d6534847a2675900c0600562/artifacts/model"
Traceback (most recent call last):
  File "/Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/orchestrators/step_launcher.py", line 255, in launch
    self._run_step(
  File "/Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/orchestrators/step_launcher.py", line 377, in _run_step
    self._run_step_without_step_operator(
  File "/Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/orchestrators/step_launcher.py", line 462, in _run_step_without_step_operator
    runner.run(
  File "/Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/orchestrators/step_runner.py", line 187, in run
    return_values = step_instance.call_entrypoint(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/steps/base_step.py", line 554, in call_entrypoint
    return self.entrypoint(**validated_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/username/ws/zenml/examples/zenml_batch_e2e/steps/training/model_trainer.py", line 95, in model_trainer
    mlflow_register_model_step.entrypoint(
  File "/Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/integrations/mlflow/steps/mlflow_registry.py", line 153, in mlflow_register_model_step
    model_version = model_registry.register_model_version(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/integrations/mlflow/model_registries/mlflow_model_registry.py", line 414, in register_model_version
    return self._cast_mlflow_version_to_model_version(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/integrations/mlflow/model_registries/mlflow_model_registry.py", line 740, in _cast_mlflow_version_to_model_version
    get_model_info(model_uri=mlflow_model_version.source)
  File "/Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/mlflow/models/model.py", line 1079, in get_model_info
    return Model.load(model_uri).get_model_info()
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/mlflow/models/model.py", line 742, in load
    raise MlflowException(
mlflow.exceptions.MlflowException: Could not find an "MLmodel" configuration file at "file:///Users/username/Library/Application Support/zenml/local_stores/6d22c082-b7ce-47d3-bf1b-6d7bb9f37611/mlruns/622276372486226458/0cb24284d6534847a2675900c0600562/artifacts/model"
Pipeline run e2e_use_case_training_run_2025_03_24_09_26_22 failed.
╭─────────────────────────── Traceback (most recent call last) ────────────────────────────╮
│ /Users/username/ws/zenml/examples/zenml_batch_e2e/run.py:216 in <module>           │
│                                                                                          │
│   213                                                                                    │
│   214                                                                                    │
│   215 if __name__ == "__main__":                                                         │
│ ❱ 216 │   main()                                                                         │
│   217                                                                                    │
│                                                                                          │
│ /Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/click/core.py │
│ :1130 in __call__                                                                        │
│                                                                                          │
│   1127 │                                                                                 │
│   1128 │   def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any:                   │
│   1129 │   │   """Alias for :meth:`main`."""                                             │
│ ❱ 1130 │   │   return self.main(*args, **kwargs)                                         │
│   1131                                                                                   │
│   1132                                                                                   │
│   1133 class Command(BaseCommand):                                                       │
│                                                                                          │
│ /Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/click/core.py │
│ :1055 in main                                                                            │
│                                                                                          │
│   1052 │   │   try:                                                                      │
│   1053 │   │   │   try:                                                                  │
│   1054 │   │   │   │   with self.make_context(prog_name, args, **extra) as ctx:          │
│ ❱ 1055 │   │   │   │   │   rv = self.invoke(ctx)                                         │
│   1056 │   │   │   │   │   if not standalone_mode:                                       │
│   1057 │   │   │   │   │   │   return rv                                                 │
│   1058 │   │   │   │   │   # it's not safe to `ctx.exit(rv)` here!                       │
│                                                                                          │
│ /Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/click/core.py │
│ :1404 in invoke                                                                          │
│                                                                                          │
│   1401 │   │   │   echo(style(message, fg="red"), err=True)                              │
│   1402 │   │                                                                             │
│   1403 │   │   if self.callback is not None:                                             │
│ ❱ 1404 │   │   │   return ctx.invoke(self.callback, **ctx.params)                        │
│   1405 │                                                                                 │
│   1406 │   def shell_complete(self, ctx: Context, incomplete: str) -> t.List["Completion │
│   1407 │   │   """Return a list of completions for the incomplete value. Looks           │
│                                                                                          │
│ /Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/click/core.py │
│ :760 in invoke                                                                           │
│                                                                                          │
│    757 │   │                                                                             │
│    758 │   │   with augment_usage_errors(__self):                                        │
│    759 │   │   │   with ctx:                                                             │
│ ❱  760 │   │   │   │   return __callback(*args, **kwargs)                                │
│    761 │                                                                                 │
│    762 │   def forward(                                                                  │
│    763 │   │   __self, __cmd: "Command", *args: t.Any, **kwargs: t.Any  # noqa: B902     │
│                                                                                          │
│ /Users/username/ws/zenml/examples/zenml_batch_e2e/run.py:186 in main               │
│                                                                                          │
│   183 │   │   pipeline_args[                                                             │
│   184 │   │   │   "run_name"                                                             │
│   185 │   │   ] = f"e2e_use_case_training_run_{dt.now().strftime('%Y_%m_%d_%H_%M_%S')}"
│ ❱ 186 │   │   e2e_use_case_training.with_options(**pipeline_args)(**run_args_train)      │
│   187 │   │   logger.info("Training pipeline finished successfully!")                    │
│   188 │                                                                                  │
│   189 │   # Execute Deployment Pipeline                                                  │
│                                                                                          │
│ /Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/pipelin │
│ es/pipeline_definition.py:1406 in __call__                                               │
│                                                                                          │
│   1403 │   │   │   return self.entrypoint(*args, **kwargs)                               │
│   1404 │   │                                                                             │
│   1405 │   │   self.prepare(*args, **kwargs)                                             │
│ ❱ 1406 │   │   return self._run()                                                        │
│   1407 │                                                                                 │
│   1408 │   def _call_entrypoint(self, *args: Any, **kwargs: Any) -> None:                │
│   1409 │   │   """Calls the pipeline entrypoint function with the given arguments.       │
│                                                                                          │
│ /Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/pipelin │
│ es/pipeline_definition.py:826 in _run                                                    │
│                                                                                          │
│    823 │   │   │   │   │   │   "`zenml login --local`."                                  │
│    824 │   │   │   │   │   )                                                             │
│    825 │   │   │                                                                         │
│ ❱  826 │   │   │   deploy_pipeline(                                                      │
│    827 │   │   │   │   deployment=deployment, stack=stack, placeholder_run=run           │
│    828 │   │   │   )                                                                     │
│    829 │   │   │   if run:                                                               │
│                                                                                          │
│ /Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/pipelin │
│ es/run_utils.py:161 in deploy_pipeline                                                   │
│                                                                                          │
│   158 │   │   │   # status to `Failed`                                                   │
│   159 │   │   │   publish_failed_pipeline_run(placeholder_run.id)                        │
│   160 │   │                                                                              │
│ ❱ 161 │   │   raise e                                                                    │
│   162 │   finally:                                                                       │
│   163 │   │   constants.SHOULD_PREVENT_PIPELINE_EXECUTION = previous_value               │
│   164                                                                                    │
│                                                                                          │
│ /Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/pipelin │
│ es/run_utils.py:145 in deploy_pipeline                                                   │
│                                                                                          │
│   142 │   constants.SHOULD_PREVENT_PIPELINE_EXECUTION = True                             │
│   143 │   try:                                                                           │
│   144 │   │   stack.prepare_pipeline_deployment(deployment=deployment)                   │
│ ❱ 145 │   │   stack.deploy_pipeline(                                                     │
│   146 │   │   │   deployment=deployment,                                                 │
│   147 │   │   │   placeholder_run=placeholder_run,                                       │
│   148 │   │   )                                                                          │
│                                                                                          │
│ /Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/stack/s │
│ tack.py:826 in deploy_pipeline                                                           │
│                                                                                          │
│   823 │   │   │   deployment: The pipeline deployment.                                   │
│   824 │   │   │   placeholder_run: An optional placeholder run for the deployment.       │
│   825 │   │   """
│ ❱ 826 │   │   self.orchestrator.run(                                                     │
│   827 │   │   │   deployment=deployment, stack=self, placeholder_run=placeholder_run     │
│   828 │   │   )                                                                          │
│   829                                                                                    │
│                                                                                          │
│ /Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/orchest │
│ rators/base_orchestrator.py:249 in run                                                   │
│                                                                                          │
│   246 │   │   │   logger.debug("Skipping client-side caching.")                          │
│   247 │   │                                                                              │
│   248 │   │   try:                                                                       │
│ ❱ 249 │   │   │   if metadata_iterator := self.prepare_or_run_pipeline(                  │
│   250 │   │   │   │   deployment=deployment,                                             │
│   251 │   │   │   │   stack=stack,                                                       │
│   252 │   │   │   │   environment=environment,                                           │
│                                                                                          │
│ /Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/orchest │
│ rators/local/local_orchestrator.py:78 in prepare_or_run_pipeline                         │
│                                                                                          │
│    75 │   │   │   │   │   step_name,                                                     │
│    76 │   │   │   │   )                                                                  │
│    77 │   │   │                                                                          │
│ ❱  78 │   │   │   self.run_step(                                                         │
│    79 │   │   │   │   step=step,                                                         │
│    80 │   │   │   )                                                                      │
│    81                                                                                    │
│                                                                                          │
│ /Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/orchest │
│ rators/base_orchestrator.py:281 in run_step                                              │
│                                                                                          │
│   278 │   │   │   step=step,                                                             │
│   279 │   │   │   orchestrator_run_id=self.get_orchestrator_run_id(),                    │
│   280 │   │   )                                                                          │
│ ❱ 281 │   │   launcher.launch()                                                          │
│   282 │                                                                                  │
│   283 │   @staticmethod                                                                  │
│   284 │   def requires_resources_in_orchestration_environment(                           │
│                                                                                          │
│ /Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/orchest │
│ rators/step_launcher.py:255 in launch                                                    │
│                                                                                          │
│   252 │   │   │   │   │   │   │   │   │   return None                                    │
│   253 │   │   │   │   │   │   │   │                                                      │
│   254 │   │   │   │   │   │   │   │   force_write_logs = _bypass                         │
│ ❱ 255 │   │   │   │   │   │   │   self._run_step(                                        │
│   256 │   │   │   │   │   │   │   │   pipeline_run=pipeline_run,                         │
│   257 │   │   │   │   │   │   │   │   step_run=step_run,                                 │
│   258 │   │   │   │   │   │   │   │   last_retry=last_retry,                             │
│                                                                                          │
│ /Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/orchest │
│ rators/step_launcher.py:377 in _run_step                                                 │
│                                                                                          │
│   374 │   │   │   │   │   last_retry=last_retry,                                         │
│   375 │   │   │   │   )                                                                  │
│   376 │   │   │   else:                                                                  │
│ ❱ 377 │   │   │   │   self._run_step_without_step_operator(                              │
│   378 │   │   │   │   │   pipeline_run=pipeline_run,                                     │
│   379 │   │   │   │   │   step_run=step_run,                                             │
│   380 │   │   │   │   │   step_run_info=step_run_info,                                   │
│                                                                                          │
│ /Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/orchest │
│ rators/step_launcher.py:462 in _run_step_without_step_operator                           │
│                                                                                          │
│   459 │   │   if last_retry:                                                             │
│   460 │   │   │   os.environ[ENV_ZENML_IGNORE_FAILURE_HOOK] = "false"
│   461 │   │   runner = StepRunner(step=self._step, stack=self._stack)                    │
│ ❱ 462 │   │   runner.run(                                                                │
│   463 │   │   │   pipeline_run=pipeline_run,                                             │
│   464 │   │   │   step_run=step_run,                                                     │
│   465 │   │   │   input_artifacts=input_artifacts,                                       │
│                                                                                          │
│ /Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/orchest │
│ rators/step_runner.py:187 in run                                                         │
│                                                                                          │
│   184 │   │   │                                                                          │
│   185 │   │   │   step_failed = False                                                    │
│   186 │   │   │   try:                                                                   │
│ ❱ 187 │   │   │   │   return_values = step_instance.call_entrypoint(                     │
│   188 │   │   │   │   │   **function_params                                              │
│   189 │   │   │   │   )                                                                  │
│   190 │   │   │   except BaseException as step_exception:  # noqa: E722                  │
│                                                                                          │
│ /Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/steps/b │
│ ase_step.py:554 in call_entrypoint                                                       │
│                                                                                          │
│    551 │   │   │   │   "pydantic error above for more details."
│    552 │   │   │   ) from e                                                              │
│    553 │   │                                                                             │
│ ❱  554 │   │   return self.entrypoint(**validated_args)                                  │
│    555 │                                                                                 │
│    556 │   @property                                                                     │
│    557 │   def name(self) -> str:                                                        │
│                                                                                          │
│ /Users/username/ws/zenml/examples/zenml_batch_e2e/steps/training/model_trainer.py: │
│ 95 in model_trainer                                                                      │
│                                                                                          │
│    92 │   )                                                                              │
│    93 │                                                                                  │
│    94 │   # register mlflow model                                                        │
│ ❱  95 │   mlflow_register_model_step.entrypoint(                                         │
│    96 │   │   model,                                                                     │
│    97 │   │   name=name,                                                                 │
│    98 │   )                                                                              │
│                                                                                          │
│ /Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/integra │
│ tions/mlflow/steps/mlflow_registry.py:153 in mlflow_register_model_step                  │
│                                                                                          │
│   150 │   │   setattr(metadata, "mlflow_run_id", mlflow_run_id)                          │
│   151 │                                                                                  │
│   152 │   # Register model version                                                       │
│ ❱ 153 │   model_version = model_registry.register_model_version(                         │
│   154 │   │   name=name,                                                                 │
│   155 │   │   version=version or "1",                                                    │
│   156 │   │   model_source_uri=model_source_uri,                                         │
│                                                                                          │
│ /Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/integra │
│ tions/mlflow/model_registries/mlflow_model_registry.py:414 in register_model_version     │
│                                                                                          │
│   411 │   │   │   │   f"Error: {e}"
│   412 │   │   │   )                                                                      │
│   413 │   │   # Return the registered model version.                                     │
│ ❱ 414 │   │   return self._cast_mlflow_version_to_model_version(                         │
│   415 │   │   │   registered_model_version                                               │
│   416 │   │   )                                                                          │
│   417                                                                                    │
│                                                                                          │
│ /Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/zenml/integra │
│ tions/mlflow/model_registries/mlflow_model_registry.py:740 in                            │
│ _cast_mlflow_version_to_model_version                                                    │
│                                                                                          │
│   737 │   │   │   from mlflow.models import get_model_info                               │
│   738 │   │   │                                                                          │
│   739 │   │   │   model_library = (                                                      │
│ ❱ 740 │   │   │   │   get_model_info(model_uri=mlflow_model_version.source)              │
│   741 │   │   │   │   .flavors.get("python_function", {})                                │
│   742 │   │   │   │   .get("loader_module")                                              │
│   743 │   │   │   )                                                                      │
│                                                                                          │
│ /Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/mlflow/models │
│ /model.py:1079 in get_model_info                                                         │
│                                                                                          │
│   1076 │   │   model_signature = model_info.signature                                    │
│   1077 │   │   assert model_signature == signature                                       │
│   1078 │   """                                                                           │
│ ❱ 1079 │   return Model.load(model_uri).get_model_info()                                 │
│   1080                                                                                   │
│   1081                                                                                   │
│   1082 class Files(NamedTuple):                                                          │
│                                                                                          │
│ /Users/username/ws/zenml/examples/.venv/lib/python3.12/site-packages/mlflow/models │
│ /model.py:742 in load                                                                    │
│                                                                                          │
│    739 │   │   │   print(urlparse(path).path)                                            │
│    740 │   │   │   print(os.path.exists(urlparse(path).path))                            │
│    741 │   │   │   ### MONKEY PATCHED DEBUG OUTPUT ABOVE, ORIGINALLY THIS WAS LINE 736                                                                  │
│ ❱  742 │   │   │   raise MlflowException(                                                │
│    743 │   │   │   │   f'Could not find an "{MLMODEL_FILE_NAME}" configuration file at " │
│    744 │   │   │   │   RESOURCE_DOES_NOT_EXIST,                                          │
│    745 │   │   │   )                                                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────╯
MlflowException: Could not find an "MLmodel" configuration file at 
"file:///Users/username/Library/Application 
Support/zenml/local_stores/6d22c082-b7ce-47d3-bf1b-6d7bb9f37611/mlruns/622276372486226458/0c
b24284d6534847a2675900c0600562/artifacts/model"
Mon Mar 24 09:26:42 PDT 2025

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions