feat: declarative `local_packages:` for adjacent Python packages, with cache-friendly late install

[andrinr]

Summary

Add a build_config.local_packages: block that pip-installs a Python package from an adjacent path. It should install in the last Dockerfile layer and read the source via RUN --mount=type=bind so a change to that package doesn't invalidate the rest of the image cache.

Why is this needed?

The current pattern is:

package_data:
  - ["../../../my_shared_pkg", "/tmp/my_shared_pkg"]
custom_build_steps:
  - "RUN pip install /tmp/my_shared_pkg"

package_data becomes a COPY near the top of the generated Dockerfile. Editing one line in my_shared_pkg changes the COPY's input, which invalidates that layer and every layer after it. For a project that shares a small schema package across many Tesseracts, a one-line schema change forces a full rebuild of every image.

A local_packages: block can fix this two ways:

• Install in the last Dockerfile layer, so a change to the package only reruns that one layer.
• Use RUN --mount=type=bind,source=... instead of COPY, so there is no COPY layer to invalidate at all.

Usage example

build_config:
  local_packages:
    - path: ../../../my_shared_pkg
    # optional: editable: bool, extras: [...], position: late|early

Rendered (last layer):

RUN --mount=type=bind,source=../../../my_shared_pkg,target=/tmp/pkg \
    --mount=type=cache,target=/root/.cache/pip \
    pip install /tmp/pkg

Notes

• position: early is for the rare case where an earlier custom_build_step needs to import the local package at build time.
• Multiple entries each become their own late layer.
• Works alongside requirements.provider: conda.

[dionhaefner]

This seems to be based on faulty assumptions.

package_data becomes a COPY near the top of the generated Dockerfile.

It's actually one of the last things to run, definitely after installing all dependencies (which is the expensive step). It's true that is runs before custom_build_steps (which may be expensive) but that should be well mitigated by the following pattern?

custom_build_steps:
    - RUN my_expensive_build
    - COPY ../my_package_data /target
    - RUN pip install /target

[dionhaefner]

Ah actually my last example does not work because the thing to COPY won't be in the build context if not declared as package data.

[andrinr]

Then I would still leave it as an open issue, as the custom build steps can take a very long time for certain tesseracts.

[dionhaefner]

Perhaps, although I'd like to drill down a bit more regarding what problem exactly we're trying to solve here. Case in point:

package_data:
  - ["../../../my_shared_pkg", "/tmp/my_shared_pkg"]
custom_build_steps:
  - "RUN pip install /tmp/my_shared_pkg"

What is it that prevents you from adding my_shared_pkg to tesseract_requirements.txt?