Issue: embed Fails for Python .whl Files in Workerd Workers

[SagarSpace]

I’m trying to embed Python .whl dependencies inside my Workerd worker but facing an issue where Workerd fails to read the files.

Setup:
• Workerd is running inside a Docker container.
• .whl files are placed inside /workerd/libs/.
• The workerd.capnp configuration looks like this:

const vVFXeSAWorker :Workerd.Worker = (
  modules = [
    (name = "entry.py", pythonModule = embed "src/entry.py"),
    (name = "fastapi", pythonRequirement = embed "/workerd/libs/fastapi-0.110.0-py3-none-any.whl"),
    (name = "anyio", pythonRequirement = embed "/workerd/libs/anyio-4.2.0-py3-none-any.whl"),
    (name = "langchain_core", pythonRequirement = embed "/workerd/libs/langchain_core-0.1.25-py3-none-any.whl"),
    (name = "langchain_openai", pythonRequirement = embed "/workerd/libs/langchain_openai-0.0.6-py3-none-any.whl"),
  ],
  compatibilityDate = "2024-03-29",
  compatibilityFlags = [
    "python_workers"
  ]
);

Issue:

Workerd fails with the following error:

workerd.capnp:21:50-106: Couldn't read file for embed: /workerd/libs/fastapi-0.110.0-py3-none-any.whl
workerd.capnp:22:48-100: Couldn't read file for embed: /workerd/libs/anyio-4.2.0-py3-none-any.whl
workerd.capnp:23:57-119: Couldn't read file for embed: /workerd/libs/langchain_core-0.1.25-py3-none-any.whl
workerd.capnp:24:59-122: Couldn't read file for embed: /workerd/libs/langchain_openai-0.0.6-py3-none-any.whl

However, entry.py is embedded successfully from src/entry.py, so the issue seems specific to .whl files.

Troubleshooting Done:
1. Verified file existence and permissions:
• Ran ls -lah /workerd/libs/ inside the container and confirmed that the files exist.
• Set correct permissions using chmod -R 755 /workerd/libs/.
2. Tried using relative paths in workerd.capnp instead of absolute paths, but the issue persists.
3. Checked Workerd’s execution context inside Docker and confirmed that /workerd/libs/ is accessible.
4. Tried pythonPackage instead of embed, which downloads dependencies dynamically, but I want to avoid runtime package fetching for performance reasons.

Questions:
• Does embed support .whl files, or is it only meant for Python source files?
• Is there a recommended way to preinstall Python dependencies in Workerd without downloading them on each startup?
Also I am trying to pre-install pydiode and use it to avoid downloading again and again, how and where to use the pre-deployed pydiode??

@kentonv

Would appreciate any guidance on this. Thanks in advance! 🚀