Allow installing individual backend.ai-* packages from source (PEP 517)

[rapsealk]

Problem

Downstream projects that depend on published backend.ai-* distributions (e.g. backend.ai-client, backend.ai-logging) can currently only install them from PyPI. There is no supported way to install an individual package from a git checkout or source tree, so a consumer cannot build against an unreleased change — it has to wait for the next PyPI release.

Root cause

The repository builds with Pants. Each distribution directory (src/ai/backend/client, src/ai/backend/logging, …) contains only a BUILD file with python_sources + python_distribution targets — there is no pyproject.toml or setup.py. Standard PEP 517 tooling (pip, uv) cannot build these directories.

A uv git source pointing at a package subdirectory fails:

× Failed to download and build `backend-ai-logging @ git+https://github.com/lablup/backend.ai@main#subdirectory=src/ai/backend/logging`
╰─▶ .../src/ai/backend/logging does not appear to be a Python project,
    as neither `pyproject.toml` nor `setup.py` are present in the directory

The root pyproject.toml declares a single backend.ai project (and has no [build-system] table), so it does not produce the individually-named distributions either.

Request

Make individual backend.ai-* distributions installable from source via standard PEP 517 tooling — for example, add a per-package pyproject.toml alongside each python_distribution BUILD target so that pip install ./src/ai/backend/client and uv git+subdirectory sources work.

Note that transitive dependencies (backend.ai-common, backend.ai-cli, backend.ai-plugin) would need the same treatment, so a source install of backend.ai-client / backend.ai-logging resolves consistently rather than mixing a source build with PyPI-pinned internals.

Why

This lets downstream consumers develop and test against unreleased upstream changes without waiting for a PyPI release, removing a significant lag from their development cycle.

[rapsealk]

Some investigation notes before anyone implements this — the "add a per-package pyproject.toml" suggestion above runs into the repo's build architecture.

Why per-package pyproject.toml is the wrong tool here

Each distribution's python_distribution target uses generate_setup=True (e.g. src/ai/backend/logging/BUILD). Pants infers the dependency list from its dependency graph and generates setup.py at build time — there is no static dependency list to lift into a hand-written pyproject.toml. Hand-authored metadata would immediately drift from the Pants graph, and a pyproject.toml sitting next to a generate_setup=True target is liable to conflict with Pants' own generated build metadata.

The repo also has effectively no per-package build files under src/ by design (only a few special-case setup.py files). So adding ~5 pyproject.toml files (client, logging, plus transitive common, cli, plugin) would be a structural change to how the monorepo is built, not a mechanical fix.

Alternatives that fit the existing build system

1. On-demand wheels via pants package (no upstream change at all). Because generate_setup=True already produces correct wheels, a downstream consumer can build them from any branch/commit:

pants package src/ai/backend/client:dist src/ai/backend/logging:dist
# -> dist/*.whl, then install via uv [tool.uv.sources] path entries or --find-links

This already unblocks "consume an unreleased change" today. It could simply be documented.

2. CI-published dev/nightly wheels. A workflow that runs pants package on main (or per-tag/per-commit) and publishes the resulting backend.ai-* wheels to a dev index. Downstream consumers then add that index as an extra-index-url. This is the idiomatic Pants-monorepo answer and keeps all build metadata Pants-generated.

Suggested next step

Option 1 needs only a docs note; option 2 is a CI/infra decision. Could a maintainer confirm which direction is preferred before any code is written? If per-package pyproject.toml is genuinely wanted despite the generate_setup overlap, that probably warrants a BEP given it changes the build model.