Better handling of user-defined modules

[Frnd-me]

I'm using the Pkl Java bindings in a runtime system, where Pkl is the basis for a DSL.
For my DSL, I have a standard library, for which it is convenient to add a module like dsl:Something.

The Java bindings support this neatly: internally, the dsl: scheme gets translated to a classpath resource shipped with the runtime build. However, current support within IDEs and the CLI does not enable ergonomic handling of user-defined modules.

It would be very convenient if there were a way to define (globally, for instance, in Pkl settings) the handler for such schemes (for example, pointing to a URL where the same module can be fetched).

As far as I can tell from the documentation, this is currently possible at a project level, but that may not always be desirable; sometimes you want a global setting without needing to create a project.

Currently, the handling of schemes in the IDE plugins and CLI appears to be fairly static.

Full support throughout the toolchain would make user-defined module support more useful.

Proposed solution

• Allow defining global scheme handlers in Pkl settings (CLI, IDE, or system-level config).
• Ensure that the same scheme resolution works consistently across:
  • CLI tooling
  • IDE integrations (VSCode, IntelliJ, etc.)
  • Runtime bindings (Java, etc.)

This would make user-defined modules a first-class feature in the Pkl ecosystem, improving ergonomics for DSLs and other custom uses of Pkl.

[HT154]

Hey there! This is something I've given some thought to.

It's worth noting that the pkl CLI does not run on the JVM and therefore cannot use resource or module URI schemes implemented using the Java API. The method for extending the CLI is External Readers implemented via a language binding library because they are runtime-agnostic. External readers are also usable when running on the JVM in lieu of in-JVM implementations, so you can avoid maintaining separate implementations.

You can read more about how I approached implementing external readers in SPICE-0009. One of the future directions mentioned was support for specifying external readers via the user's ~/.pkl/settings.pkl file. We didn't have a strong motivating use case for this today and there are some unanswered security questions around it, but it's certainly technically feasible.

I do think it would be good for pkl-lsp and pkl-intellij to provide better support for this kind of usage! It may be reasonable for these tools to support invoking external readers for module resolution and import autocompletion to make working with custom URI schemes a better experience. Even simpler, I think it would be good to support configuring these tools with a way to re-map URIs with custom schemes to a different (natively supported) URIs like https: or file:. One or both of these approaches would hopefully provide the capability you're looking for on the tooling side.

[Frnd-me]

Thanks!

External readers indeed make sense, and I was thinking of either support for specifying external readers in the settings.pkl file. However, I understand the security implications that this may have.

IDE support would be great, and a simple mapping would suffice.

I thought of something else that may be useful regarding this; since my current rewriting is quite simple, it could also be implemented as a Pkl function. It may be helpful to have Pkl functions defined to provide the mapping functionality, which can also help avoid any security issues.

[bioball]

Rather than using a custom scheme, can you publish your DSL as a package? This is something natively understood by pkl-intellij and pkl-lsp.

For package imports, take a look at https://pkl-lang.org/main/current/language-reference/index.html#package-asset-uri