[FEATURE] - Ensure dependency compatibility with installed Spark version

[FlavRei]

When specifying SCALA_VERSION=2.12, the setup downloads dependencies according to the previously installed Spark version. However, since Spark is at the core of Starlake, it would be better to ensure that the downloaded dependencies are always compatible with the installed Spark version, rather than requiring manual configuration.

Would it be possible to improve this dependency resolution to avoid version mismatches?

[hayssams]

Hi @FlavRei
From what I understand, you have a version with Scala 2.13 previously installed and when trying to reinstall with Scala 2.12, you get a mixed 2.12 and 2.13 environment.

I suggest that we request from the user to install Starlake in a different directory if there is a mismatch in the scala versions.
would such a solution make it for you ?

[FlavRei]

Actually I initially had a Starlake version running with Scala 2.12. When I updated Starlake, the new version was supposed to work with Scala 2.13, but I still had Scala 2.12 installed.

I understand that installing Starlake in a different directory could be a workaround, but wouldn’t it be better to ensure that the correct Scala version is used automatically when upgrading? Maybe by enforcing a check during installation or handling dependency resolution more dynamically?