Skip to content

feat(marketplace): add sourceBase for nested package sources (#1519)#1731

Draft
leocamello wants to merge 2 commits into
microsoft:mainfrom
leocamello:feature/marketplace-sourcebase
Draft

feat(marketplace): add sourceBase for nested package sources (#1519)#1731
leocamello wants to merge 2 commits into
microsoft:mainfrom
leocamello:feature/marketplace-sourcebase

Conversation

@leocamello

Copy link
Copy Markdown

Description

Adds an optional marketplace.sourceBase field so marketplace authors on
self-managed hosts (canonical case: GitLab nested subgroups) can declare a git
base once and write each package source relative to it. The per-entry source
shorthand tops out at host.tld/owner/repo (3 segments); enterprise GitLab
nesting is routinely deeper (.../group/sub-group/team/project/<pkg>), which the
shorthand cannot express.

marketplace:
  sourceBase: https://gitlab.example.com/group/sub-group/team/projects
  packages:
    - name: my-package
      source: my-package                  # -> .../projects/my-package
      version: "^1.0.0"
    - name: from-github
      source: github.com/acme/helper      # host-prefixed: overrides the base
      ref: v1.0.0

Semantics ("relative to base, always"): a host-less source composes onto the
base; a source carrying its own host (host.tld/owner/repo or full URL) or a
local ./ source overrides the base; with no sourceBase declared, behaviour is
byte-for-byte unchanged and a host-less single-segment source stays rejected
(fail-closed — no silent routing to the default host).

Implementation: composition happens at parse time (yml_schema.py) — a
relative entry is rewritten to the equivalent host-prefixed form, so the
resolver, builder, and output mappers treat it identically to a #1288
host-prefixed entry with no changes to those modules. sourceBase is validated
with #1288's posture (https-only, FQDN host, no userinfo/port/query/fragment/
.git) plus validate_path_segments. apm marketplace plugin add accepts
relative sources when a base is declared.

Also fixes apm marketplace check: it was building a single default-host
resolver with no token and ignoring entry.host, so it failed (git ls-remote
exit 128) for any non-default host — every sourceBase entry, and the
pre-existing #1288 host-prefixed form. It now uses the same per-host resolver +
per-host token pattern as apm pack (extracted into a shared
marketplace/auth_helpers.resolve_token_for_host) and short-circuits local
packages, so check and pack agree.

Validated end-to-end against a real self-managed GitLab (4-segment nested group):
apm marketplace check exits 0, and apm pack emits a source: url entry with
the composed URL and a real resolved sha.

The consumer-side install gap for non-default-host marketplaces
(apm install <pkg>@<marketplace> strips the host) is independent, reproduces on
stock 0.18.0, and is tracked in #1010 — out of scope here.

Fixes #1519

Type of change

  • Bug fix
  • New feature
  • Documentation
  • Maintenance / refactor

Testing

  • Tested locally
  • All existing tests pass
  • Added tests for new functionality (if applicable)

tests/unit/marketplace/test_source_base.py (validation, composition for 1 and N
segments, override precedence, base-less rejection, composed marketplace.json
URL) and tests/unit/commands/test_marketplace_check.py (per-host resolution:
composed→GitLab+token, host-prefixed override→github, local→zero ls-remote).
uv run pytest tests/unit tests/test_console.py is green except one pre-existing,
environment-dependent test_auth_scoping failure unrelated to this change.
ruff check / ruff format --check clean.

Spec conformance (OpenAPM v0.1)

  • N/A -- this PR does not change OpenAPM-observable behaviour.

(src/apm_cli/marketplace/ is not a Mode-B critical path and
CONFORMANCE.{json,md} regenerate with no diff.)

…ft#1519)

Marketplace blocks in apm.yml can now declare an optional `sourceBase` --
a git base (https://host/path, arbitrary depth) that host-less relative
package `source` values compose onto. This enables deeply nested
enterprise GitLab group paths that the `host.tld/owner/repo` shorthand
cannot express.

Semantics ("relative to base, always"):
- A host-less `source` (`my-package`, `a/b/c`) composes onto the base.
- Host-prefixed (`host.tld/owner/repo`) and full-URL sources override the
  base (absolute, base ignored); local `./` sources are untouched.
- Without `sourceBase`, behavior is byte-for-byte unchanged, and a
  host-less relative source stays rejected (fail-closed -- no silent
  routing to the default host).

Composition happens at parse time: a relative entry is rewritten to the
equivalent host-prefixed form, so the resolver, builder, and output
mappers treat it identically to a microsoft#1288 host-prefixed entry -- both the
build-time `git ls-remote` and the emitted marketplace.json URL are
correct with no changes to those modules.

`sourceBase` is validated with microsoft#1288's security posture (https-only,
FQDN host, no userinfo/port/query/fragment/.git) plus
validate_path_segments on the path. `apm marketplace plugin add` accepts
relative sources when a base is declared.

Includes schema/authoring docs, a CHANGELOG entry, and unit tests
covering validation, composition (1 and N segments), override
precedence, base-less rejection, and the composed marketplace.json URL.
`apm marketplace check` built a single default-host RefResolver with no
token and called list_remote_refs(entry.source) ignoring entry.host, so
any package on a non-default host (self-managed GitLab, GHES, ADO,
Bitbucket DC) -- including every relative source composed onto
marketplace.sourceBase -- failed with `git ls-remote` exit 128 even when
`apm pack` against the same apm.yml succeeded.

Mirror the per-host resolver pattern that builder.py already uses: build
one RefResolver per effective host, with the host's token resolved via
AuthResolver. Local `./` packages now short-circuit before any network
call. Default-host entries keep today's behaviour exactly.

The per-host token logic is extracted into a shared
`marketplace/auth_helpers.resolve_token_for_host` that both the builder
and the check command use, so the two paths cannot drift again.

Surfaced while testing microsoft#1519 (sourceBase always yields a non-default
host); also fixes the pre-existing gap for microsoft#1288 host-prefixed sources.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

marketplace.packages[].source: generalize beyond host/owner/repo (deeply-nested GitLab subgroups, arbitrary git hosts)

1 participant