support for project/package specific modulepaths

[sitepark-schaeper]

Currently the lsp cannot resolve imports from the modulepath (e.g. import "modulepath:/mymodule/somefile.pkl").
It would be nice if a user could somehow provide a local modulepath for the lsp to use for the project (or even package).

Here is our usecase:
In essence we have a system where our pkl files are deployed to locations we do not know beforehand and which are different from the systems they're beeing developed on.
We pass those locations via --modulepath and import the files from there. This works for evaluating, but since the pkl-lsp does not support modulepaths, development is really unpleasant.

We've tried alternatively to instead use packages and import the dependencies there via modulepath:

amends "pkl:Project"

package {
  name = "mypackage"
  // ...
}

dependencies {
  ["myotherpackage"] = import("modulepath:/myotherpackage/PklProject")
}

The idea here was that if pkl package resolve would write the resulting local path into the PklProject.deps.json, then the lsp could resolve package imports (like import "@mymodule/somefile.pkl") and only the PklProject file would report "Cannot resolve import.".

But it fails because of two reasons:

• the dependencies field has a constraint for these locally imported Projects projectFileUris to effectively start with "file://" (here)
• the cli tool evaluates Projects without ModulePathResolver (here)

The first one can be "worked around" by spoofing the projectFileUri field in the imported Projects:

local spoofedUri: String? = read?("env:MODULE_PATH").ifNonNull((path) -> "file://" + (path as String)
    .split(":")
    .map((e) -> if (!e.endsWith("/")) e + "/" else e)
    .map((e) -> e + module.package.name + "/" + "PklProject")
    .find((e) -> read?("file:" + e) != null))
projectFileUri = if (spoofedUri != null) spoofedUri else super.projectFileUri

For the second issue we had to patch the cli tool ourselfs, but after that we got the lsp to resolve these package dependencies by exporting a MODULE_PATH environment variable.
But obviously this isn't ideal.

If you can point me in the right direction I'd be happy to work on a PR. Or if there is another solution we haven't considered yet, please let me know.
We would be fine with writing an external reader if the lsp would support it (which might be easier to implement).
We would also be okay with publishing and fetching packages for development, but when they are deployed we have to use the local files.

[bioball]

We definitely should support modulepath URIs. It should probably be a setting that users can configure, and be a set of directories or zip files that get set as modulepath.

Your use-case is a little strange to me. It sounds like "I know the contents of my dependencies, but I don't know where they are placed in the file system when deployed". Is that right?

Can you publish your dependencies as packages, and simply depend on their remote coordinates? You can then use --cache-dir to control where Pkl looks for them at runtime.

[sitepark-schaeper]

Your use-case is a little strange to me. It sounds like "I know the contents of my dependencies, but I don't know where they are placed in the file system when deployed". Is that right?

Sorry for my rambling, let me try again:
Our deployment process downloads and extracts war files (we call them "modules") onto a server. These vary from server to server and maybe even from deployment to deployment. They each also originate from different repositories. One of them contains the "main" pkl file, and all others may contain pkl files it depends on. A relative path to such a dependency would contain a version number, that we do not know outside of the underlying java process.
Additionally imports also have to succeed in development environments where the corresponding repositories are located wherever. We could theoretically remove the version number but the disparity between production and development remains.

We ended up solving this by doing our imports via modulepath, which the java process can construct and the developer can pass manually to the cli.

Can you publish your dependencies as packages, and simply depend on their remote coordinates?

Technically we could. We do use maven for the majority of our dependency management, such that our modules may have SNAPSHOT versions, which are somewhat incompatible with semver. We could use different versions for the pkl packages and refer to them to get around that. For development across multiple repositories one would then have to change the dependency to be local.
And we would need an appropriate server.
All of this is doable, but would mean a lot more effort. The modulepath solution seems to fit very well for us, it's just the lsp that lacks support for it.