License Error Installing on Databricks

[carl24k]

I'm trying to install econml on a databricks cluster with type r8g.large and I'm getting this error. The weird thing is i installed it on a different cluster with a different instance type, m5d.xlarge, and it worked. Has anyone seen this or know what to do about it?

Library installation attempted on the driver node of cluster 1009-205745-udzc9pvg and failed. Pip could not successfully build the wheel due to missing build dependencies or errors. Please check the wheel package contents and dependencies. Error code: ERROR_WHEEL_BUILD, error message: org.apache.spark.SparkException: Process List(/bin/su, libraries, -c, bash /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip install 'econml==0.16.0' --disable-pip-version-check) exited with code 1. error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [95 lines of output]
configuration error: project.license must be valid exactly by one definition (2 matches found):

      - keys:
          'file': {type: string}
        required: ['file']
      - keys:
          'text': {type: string}
        required: ['text']
  
  DESCRIPTION:
      `Project license <https://peps.python.org/pep-0621/#license>`_.
  
  GIVEN VALUE:
      "MIT"
  
  OFFENDING RULE: 'oneOf'
  
  DEFINITION:
      {
          "oneOf": [
              {
                  "properties": {
                      "file": {
                          "type": "string",
                          "$$description": [
                              "Relative path to the file (UTF-8) which contains the license for the",
                              "project."
                          ]
                      }
                  },
                  "required": [
                      "file"
                  ]
              },
              {
                  "properties": {
                      "text": {
                          "type": "string",
                          "$$description": [
                              "The license of the project whose meaning is that of the",
                              "`License field from the core metadata",
                              "<https://packaging.python.org/specifications/core-metadata/#license>`_."
                          ]
                      }
                  },
                  "required": [
                      "text"
                  ]
              }
          ]
      }
  Traceback (most recent call last):
    File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
             ^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.12/dist-packages/setuptools/build_meta.py", line 332, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=[])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.12/dist-packages/setuptools/build_meta.py", line 302, in _get_build_requires
      self.run_setup()
    File "/usr/local/lib/python3.12/dist-packages/setuptools/build_meta.py", line 318, in run_setup
      exec(code, locals())
    File "<string>", line 37, in <module>
    File "/usr/local/lib/python3.12/dist-packages/setuptools/__init__.py", line 117, in setup
      return distutils.core.setup(**attrs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.12/dist-packages/setuptools/_distutils/core.py", line 158, in setup
      dist.parse_config_files()
    File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.12/site-packages/_virtualenv.py", line 22, in parse_config_files
      result = old_parse_config_files(self, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.12/dist-packages/setuptools/dist.py", line 608, in parse_config_files
      pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
    File "/usr/local/lib/python3.12/dist-packages/setuptools/config/pyprojecttoml.py", line 71, in apply_configuration
      config = read_configuration(filepath, True, ignore_option_errors, dist)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.12/dist-packages/setuptools/config/pyprojecttoml.py", line 136, in read_configuration
      validate(subset, filepath)
    File "/usr/local/lib/python3.12/dist-packages/setuptools/config/pyprojecttoml.py", line 60, in validate
      raise ValueError(f"{error}\n{summary}") from None
  ValueError: invalid pyproject.toml config: `project.license`.
  configuration error: `project.license` must be valid exactly by one definition (2 matches found):
  
      - keys:
          'file': {type: string}
        required: ['file']
      - keys:
          'text': {type: string}
        required: ['text']
  
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.

[kbattocchi]

Hi @carl24k,
Some more information about your environment would be helpful (e.g. what OS and Python versions are you using). Also, is there more information in the traceback that indicates whether it's the building of EconML that's hitting that error or if it's actually when building some dependency?

We're not seeing that error in any of our nightly builds that basically just do a pip install in different environments without tightly constraining dependencies; one possibility is that there's no prebuilt wheel for a dependency in your environment but there is in our environments, so we never hit that because we can just install the wheel for the dependency without building it from source.

[carl24k]

This is in databricks using databricks runtime 16.4 LTS (includes Apache Spark 3.5.2, Scala 2.12) I'm not exactly sure the OS but its Python 3.12.13. The error came up when I was trying to launch a cluster using a specific recent AWS instance type r8g.large. Does that help?