Can PklProjects be packaged with their dependencies?

[avarei]

I want to use Pkl Packages in air-gapped environments.

Currently I have a Project that depends on another Project, which in turn depends on a third one.

graph LR
  a[My Package]
  b[Package B]
  c[Package C]
  a --depends on--> b --depends on--> c

Loading

This works fine... with internet access.
But I noticed that the Packages only contain references to their dependencies. not the pacakges themselves.

Can I package my Project, or run Pkl in a way that allows me to define a mirror, or package the Project with its dependencies?

[StefMa]

AFAIK it is not possible to package dependencies.

Running a mirror is also not supported (yet). However there is at SPICE for this at least:
apple/pkl-evolution#4

I am not sure if this is an option, but pkl caches it's dependencies. So if you can push that cache (~.pkl/ IIRC) to your environment, then you don't need internet connection.

[avarei]

Thank you for bringing this spice to my attention.
in that case I will open another discussion for adding the dependency to the package, since I think it would have some nice benefits. especially when considering OCI packages for future distribution.

For my case using the cache is not really a good option, since pkl is running in a container as a backend of a gRPC service.
I would like to avoid bakeing in any specific pkl packages into the container image since that should not be the concern of this container.

I think for the moment I will use a monorepo with all dependencies in one place.

[bioball]

Do you have a proxy available? We just released Pkl 0.26, with support for HTTP proxies (see https://pkl-lang.org/main/current/release-notes/0.26.html#http-proxying).

[avarei]

sadly in my case a proxy is only in dev. everything else is completely isolated...

[bioball]

See answer here for another option: #534 (comment)

[bioball]

One option right now is to use the cache dir as a poor man's vendoring mechanism.

You can download all packages imported by your application, and then populate your cache directory before deploying this to your end environment.

Something like this (this script requires jq)

cat PklProject.deps.json \
  | jq -r '.resolvedDependencies[] | select(.type == "remote") | .uri[7:] + "::sha256:" + .checksums.sha256' \
  | xargs -I {} pkl download-package --no-transitive {} --cache-dir vendor

Once you have that, you can pkl eval --cache-dir vendor, and this avoid any internet calls to download packages.

If you're using the CLI for evaluation, you can also set the cache dir via the PklProject file. This means you don't need to specify --cache-dir as a flag.

amends "pkl:Project"

evaluatorSettings {
  moduleCacheDir = "vendor"
}

If your project has a direct package imports (you import them directly instead of using PklProject, you can use the analyze-imports API to find them.

Something like this should work (untested). Note, we must omit --no-transitive this time, because we need all of the package's transitive dependencies.

pkl analze-imports -f json myEntrypoint.pkl \
  | jq -r '[.resolvedImports[] | split("#/")[0] | select(startswith("package://"))] | unique | .[]'
  | xargs -I {} pkl download-package {} --cache-dir vendor

[avarei]

@StefMa proposed a similar approach.

But I noticed only now, that the PklProject.deps.json contains ALL dependencies. thats nice!

I've pondered it a bit more and maybe It's a workaround that I can use to start with after all...
Though it would likely be relatively maintanance intensive since other colleagues will use the app as well. with their own Pkl files. And I will have to rebuild and deploy the container image for each new dependency...

I also saw that with the new Release it would be theoretically possible to create a new package format with batteries included and support it in the app using the new Scheme-agnostic project dependencies.

in this regard I would be eager to know your vision with regard of packaging. are you still considering OCI support #354 and would included dependencies be something you would be open to, or maybe you have a reason not to only reference to dependencies? :)

Sidenote:
Since PklProject.deps.json contains all dependencies, I wonder if it would be possible to use this as a poor man's mirroring mechanism...

pkl project resolve
cat PklProject.deps.json  | jq '.resolvedDependencies |= with_entries(.value.uri |= sub("package://"; "package://mirror/"))' | jq '.resolvedDependencies |= with_entries(.key |= sub("package://"; "package://mirror/"))'
pkl project package