Skip to content

Commit 264e3f9

Browse files
feat(marketplace): add source parity for marketplace add
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 23b0518 commit 264e3f9

18 files changed

Lines changed: 984 additions & 65 deletions

File tree

CHANGELOG.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1717
of `~/.hermes/config.yaml` (written atomically with `0o600` perms, preserving
1818
unrelated config keys and refusing to overwrite a malformed file). `HERMES_HOME`
1919
overrides the Hermes home directory. See the [Hermes integration guide](https://microsoft.github.io/apm/integrations/hermes/).
20+
- `apm marketplace add` now accepts Anthropic-compatible git URL `#ref`, local
21+
file, and hosted `marketplace.json` sources, recording hosted JSON provenance
22+
in the lockfile. (closes #676)
2023

2124
## [0.19.0] - 2026-06-09
2225

docs/src/content/docs/consumer/installing-from-marketplaces.md

Lines changed: 29 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -65,36 +65,50 @@ reproducible across machines and CI.
6565

6666
## Local and self-hosted marketplaces
6767

68-
`apm marketplace add` accepts more than GitHub-hosted repos. The same
69-
register / browse / install / update workflow works against:
68+
`apm marketplace add` accepts the same source shapes as Anthropic's
69+
marketplace command. The same register / browse / install / update
70+
workflow works against:
7071

7172
- **Local filesystem paths** -- `apm marketplace add /srv/marketplaces/agent-forge`
72-
(or a relative path, or `~/code/marketplace`). Useful for
73-
privacy-sensitive packages, offline workflows, and air-gapped
74-
environments. Local marketplaces with relative plugin sources
75-
install by copying from disk via `LocalDependencySource`.
73+
(or a relative path, `~/code/marketplace`, or a direct
74+
`marketplace.json` file). Useful for privacy-sensitive packages,
75+
offline workflows, and air-gapped environments. Local marketplaces
76+
with relative plugin sources install by copying from disk via
77+
`LocalDependencySource`.
7678
- **`file://` URIs** -- `apm marketplace add file:///srv/marketplaces/agent-forge.git`.
7779
Behaves the same as a local path.
80+
- **Git HTTPS URLs with `#ref`** --
81+
`apm marketplace add https://gitlab.com/acme/agent-forge.git#v1.0.0`.
82+
APM strips the fragment into the stored ref. If you omit `#ref`
83+
(or `--ref`), APM warns because the default branch can move.
84+
- **Hosted `marketplace.json` URLs** --
85+
`apm marketplace add https://catalog.example.com/marketplace.json --name catalog`.
86+
APM fetches the JSON directly over HTTPS, caches ETag/Last-Modified
87+
metadata, and records the source URL plus content digest in the
88+
lockfile when packages are installed from it.
7889
- **Generic git URLs** -- any host APM does not classify as
7990
GitHub or GitLab family flows through subprocess `git` and
80-
`GitCache`. Includes Azure DevOps (auth via `ADO_APM_PAT`),
81-
Gitea, Bitbucket Server, and self-hosted git servers.
91+
`GitCache`. Includes Azure DevOps, Gitea, Bitbucket Server, and
92+
self-hosted git servers.
8293
- **SSH URLs** -- `git@gitea.example.com:org/repo.git`. The host
83-
is extracted, classified, and routed through the matching
84-
fetcher.
94+
is extracted, classified, and routed through the matching fetcher.
95+
96+
Phase 1 of source parity is public plus local only: direct hosted
97+
`marketplace.json` URLs use HTTPS without custom auth headers. Private
98+
URL auth is tracked separately.
8599

86100
For generic-git marketplaces, `marketplace.json` is fetched via a
87101
sparse-cone clone (only the manifest path is downloaded); APM does
88102
not forward `GITHUB_APM_PAT` or `GITLAB_APM_TOKEN` to non-GitHub /
89-
non-GitLab hosts. Authentication falls through to the host's
90-
`*_APM_PAT` (e.g. `ADO_APM_PAT`) or your local
91-
`git credential-manager`. See
103+
non-GitLab hosts. Authentication falls through to the host's local git
104+
credential helper when one is configured. See
92105
[Authentication](../authentication/).
93106

94107
**Lockfile note.** Installs from a local marketplace record a
95108
local-path source in `apm.lock.yaml`. Lockfiles produced this way
96-
are machine-specific -- do not commit them into a shared repo. See
97-
[lockfile reference](../../reference/lockfile-spec/).
109+
are machine-specific -- do not commit them into a shared repo. Remote
110+
`marketplace.json` installs record `source_url` and `source_digest`
111+
provenance. See [lockfile reference](../../reference/lockfile-spec/).
98112

99113
## Where next
100114

docs/src/content/docs/reference/cli/marketplace.md

Lines changed: 23 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -61,11 +61,16 @@ Register a marketplace from a source reference. Accepted forms:
6161
- `OWNER/REPO` -- GitHub shorthand (`acme/marketplace`).
6262
- `HOST/OWNER/.../REPO` -- non-GitHub host shorthand
6363
(`gitlab.com/team/marketplace`).
64-
- HTTPS URL -- any git host, including Azure DevOps, GitLab,
65-
Gitea, Bitbucket Server, or a self-hosted git server.
64+
- HTTPS git URL -- any git host, including Azure DevOps, GitLab,
65+
Gitea, Bitbucket Server, or a self-hosted git server. Add `#ref`
66+
to pin the marketplace, for example
67+
`https://gitlab.com/acme/marketplace.git#v1.0.0`.
68+
- Hosted `marketplace.json` URL --
69+
`https://catalog.example.com/marketplace.json`.
6670
- SSH URL -- `git@host:org/repo.git` style.
6771
- Local filesystem path -- absolute (`/srv/marketplaces/agent-forge`),
68-
relative (`./local-mkt`), or home-based (`~/code/marketplace`).
72+
relative (`./local-mkt`), home-based (`~/code/marketplace`), or a
73+
direct `marketplace.json` file.
6974
- `file://` URI -- `file:///srv/marketplaces/agent-forge.git`.
7075

7176
```bash
@@ -79,14 +84,18 @@ apm marketplace add gitlab.com/my-org/awesome-agents --host gitlab.com
7984
apm marketplace add https://dev.azure.com/contoso/eng/_git/agent-forge \
8085
--name agent-forge
8186

82-
# Gitea / Bitbucket Server / self-hosted git
83-
apm marketplace add https://gitea.example.com/org/repo.git --name custom
87+
# Gitea / Bitbucket Server / self-hosted git, pinned with #ref
88+
apm marketplace add https://gitea.example.com/org/repo.git#v1.0.0 --name custom
89+
90+
# Hosted marketplace.json URL
91+
apm marketplace add https://catalog.example.com/marketplace.json --name catalog
8492

8593
# SSH
8694
apm marketplace add git@gitea.example.com:org/repo.git --name custom
8795

88-
# Local filesystem (bare repo or working directory)
96+
# Local filesystem (bare repo, working directory, or marketplace.json file)
8997
apm marketplace add /srv/marketplaces/agent-forge.git --name agent-forge
98+
apm marketplace add ./vendor/marketplace.json --name vendor
9099

91100
# file:// URI
92101
apm marketplace add file:///srv/marketplaces/agent-forge.git --name agent-forge
@@ -95,19 +104,21 @@ apm marketplace add file:///srv/marketplaces/agent-forge.git --name agent-forge
95104
| Flag | Description |
96105
|---|---|
97106
| `--name`, `-n` | Display name. Defaults to the repo name. |
98-
| `--ref`, `-r` | Git ref (branch, tag, or SHA). Default: `main`. |
107+
| `--ref`, `-r` | Git ref (branch, tag, or SHA). Default: `main`. For HTTPS git URLs, a `#ref` fragment is equivalent and is stored as the ref. |
99108
| `--branch`, `-b` | Deprecated alias for `--ref`. |
100109
| `--host` | Git host FQDN. Default: `github.com`. Ignored when `SOURCE` is a URL or local path. |
101110
| `--verbose`, `-v` | Show detailed output. |
102111

103112
**Trust boundary.** APM forwards its authentication tokens
104113
(`GITHUB_APM_PAT`, `GITLAB_APM_PAT`) only when the marketplace
105-
host is classified as GitHub or GitLab family. For any other host
106-
-- generic HTTPS, SSH, Azure DevOps, self-hosted -- the
114+
host is classified as GitHub or GitLab family. For any other git
115+
host -- generic HTTPS, SSH, Azure DevOps, self-hosted -- the
107116
marketplace is fetched via subprocess `git` through `GitCache`,
108-
and authentication falls through to the host's APM PAT (e.g.
109-
`ADO_APM_PAT` for Azure DevOps) or your local
110-
`git credential-manager`. See
117+
and authentication falls through to the host's local git credential
118+
helper. Hosted `marketplace.json` URLs are public HTTPS only in this
119+
phase; private URL auth is tracked separately. When packages are
120+
installed from a hosted JSON URL, the lockfile records the source URL
121+
and fetched content digest. See
111122
[`getting-started/authentication`](../../../getting-started/authentication/).
112123

113124
**Azure DevOps.** ADO-hosted marketplaces fetch `marketplace.json`

packages/apm-guide/.apm/skills/apm-usage/commands.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -118,7 +118,7 @@ Credentials resolve via `APM_REGISTRY_TOKEN_{NAME}` env var (or `apm config set
118118

119119
| Command | Purpose | Key flags |
120120
|---------|---------|-----------|
121-
| `apm marketplace add SOURCE` | Register a marketplace. `SOURCE` accepts `OWNER/REPO`, `HOST/OWNER/REPO`, nested `HOST/group/sub/.../REPO`, HTTPS URL (any git host -- GitHub, GitLab, ADO, Gitea, self-hosted), SSH URL (`git@host:org/repo.git`), local filesystem path, or `file://` URI. | `-n NAME`, `-r REF`, `--host HOST` |
121+
| `apm marketplace add SOURCE` | Register a marketplace. `SOURCE` accepts `OWNER/REPO`, `HOST/OWNER/REPO`, nested `HOST/group/sub/.../REPO`, HTTPS git URL with optional `#ref`, hosted `marketplace.json` URL, SSH URL (`git@host:org/repo.git`), local directory or file path, or `file://` URI. | `-n NAME`, `-r REF`, `--host HOST` |
122122
| `apm marketplace list` | List registered marketplaces | -- |
123123
| `apm marketplace browse NAME` | Browse marketplace plugins | -- |
124124
| `apm marketplace update [NAME]` | Update marketplace index | -- |

src/apm_cli/commands/install.py

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -430,10 +430,7 @@ def warning_handler(msg):
430430
if logger:
431431
logger.validation_fail(package, reason)
432432
continue
433-
marketplace_provenance = {
434-
"discovered_via": marketplace_name,
435-
"marketplace_plugin_name": plugin_name,
436-
}
433+
marketplace_provenance = resolution.provenance(marketplace_name, plugin_name)
437434
package = canonical_str
438435
marketplace_dep_ref = getattr(resolution, "dependency_reference", None)
439436
except Exception as mkt_err:

src/apm_cli/commands/marketplace/__init__.py

Lines changed: 97 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
import sys
1414
import traceback
1515
from pathlib import Path
16+
from urllib.parse import urlsplit, urlunsplit
1617

1718
import click
1819
import yaml
@@ -524,19 +525,28 @@ def add(source, name, ref, branch, host, verbose):
524525
from ...marketplace.registry import add_marketplace
525526
from ...utils.github_host import is_valid_fqdn
526527

528+
source_arg, fragment_ref = _split_source_fragment_ref(source)
529+
527530
# --ref / --branch reconciliation. --branch stays as a hidden alias
528-
# for one release so legacy invocations keep working; passing both
529-
# is a hard error so we never silently pick one.
531+
# for one release so legacy invocations keep working; passing multiple
532+
# ref sources is a hard error so we never silently pick one.
533+
explicit_ref = ref is not None or branch is not None
530534
if ref is not None and branch is not None:
531535
logger.error(
532536
"--ref and --branch are mutually exclusive. Use --ref (--branch is a deprecated alias).",
533537
symbol="error",
534538
)
535539
sys.exit(1)
536-
effective_ref = ref if ref is not None else (branch if branch is not None else "main")
540+
if fragment_ref and explicit_ref:
541+
logger.error(
542+
"Do not combine a git URL #ref with --ref or --branch. Use one ref source.",
543+
symbol="error",
544+
)
545+
sys.exit(1)
546+
effective_ref = fragment_ref or ref or branch or "main"
537547

538548
try:
539-
url, kind, resolved_host = _parse_marketplace_source(source, host)
549+
url, kind, resolved_host = _parse_marketplace_source(source_arg, host)
540550
except PathTraversalError:
541551
logger.error(
542552
f"Invalid source '{source}': contains a path-traversal sequence. "
@@ -557,6 +567,8 @@ def add(source, name, ref, branch, host, verbose):
557567
# --host is meaningful only for shorthand OWNER/REPO inputs. For URL
558568
# / SSH / local-path inputs the host is already embedded; warn that
559569
# --host is being ignored rather than silently overriding.
570+
is_direct_url = _is_remote_marketplace_json_url(url)
571+
560572
if host is not None and kind == "local":
561573
logger.warning(
562574
"--host is ignored when SOURCE is a local filesystem path.",
@@ -566,7 +578,7 @@ def add(source, name, ref, branch, host, verbose):
566578
host is not None
567579
and host.strip().lower() != (resolved_host or "").lower()
568580
and kind in ("git", "github", "gitlab")
569-
and (source.startswith(("https://", "git@", "file://")))
581+
and (source_arg.startswith(("https://", "git@", "file://")))
570582
):
571583
logger.warning(
572584
"--host is ignored when SOURCE is a full URL.",
@@ -607,18 +619,32 @@ def add(source, name, ref, branch, host, verbose):
607619

608620
# Surface progress before the slow probe + fetch (5-30s for generic-git)
609621
# so the user sees activity instead of staring at a blank terminal.
610-
provisional_label = name or _default_alias_from_url(url)
622+
provisional_label = name or (
623+
_default_alias_from_remote_url(url) if is_direct_url else _default_alias_from_url(url)
624+
)
611625
logger.start(f"Registering marketplace '{provisional_label}'...", symbol="gear")
626+
if _should_warn_unpinned_git_url(
627+
source_arg, kind, is_direct_url, fragment_ref, explicit_ref
628+
):
629+
logger.warning(
630+
"Pin this git marketplace with a #ref (for example, "
631+
f"{source_arg}#v1.0.0) or --ref to avoid mutable branch updates.",
632+
symbol="warning",
633+
)
612634

613635
# Probe for marketplace.json location. The probe source's name is a
614636
# placeholder -- _auto_detect_path only consults url/ref/path/kind.
615637
probe_name = provisional_label
616638
probe_source = MarketplaceSource(
617639
name=probe_name,
618640
url=url,
619-
ref=effective_ref,
641+
ref="" if is_direct_url else effective_ref,
642+
path="" if is_direct_url else "marketplace.json",
620643
)
621-
detected_path = _auto_detect_path(probe_source)
644+
if is_direct_url or _local_source_points_to_file(probe_source):
645+
detected_path = ""
646+
else:
647+
detected_path = _auto_detect_path(probe_source)
622648

623649
if detected_path is None:
624650
logger.error(
@@ -632,7 +658,7 @@ def add(source, name, ref, branch, host, verbose):
632658
fetch_source = MarketplaceSource(
633659
name=probe_name,
634660
url=url,
635-
ref=effective_ref,
661+
ref="" if is_direct_url else effective_ref,
636662
path=detected_path,
637663
)
638664
manifest = fetch_marketplace(fetch_source, force_refresh=True)
@@ -663,15 +689,16 @@ def add(source, name, ref, branch, host, verbose):
663689
)
664690

665691
logger.verbose_detail(f" Source: {fetch_source.display_source}")
666-
logger.verbose_detail(f" Kind: {kind}")
667-
logger.verbose_detail(f" Ref: {effective_ref}")
692+
logger.verbose_detail(f" Kind: {fetch_source.kind}")
693+
if not is_direct_url:
694+
logger.verbose_detail(f" Ref: {effective_ref}")
668695
logger.verbose_detail(f" Detected path: {detected_path}")
669696
logger.verbose_detail(f" Alias source: {alias_source}")
670697

671698
final_source = MarketplaceSource(
672699
name=display_name,
673700
url=url,
674-
ref=effective_ref,
701+
ref="" if is_direct_url else effective_ref,
675702
path=detected_path,
676703
)
677704
add_marketplace(final_source)
@@ -696,6 +723,61 @@ def add(source, name, ref, branch, host, verbose):
696723
sys.exit(1)
697724

698725

726+
def _split_source_fragment_ref(source: str) -> tuple[str, str]:
727+
"""Split an HTTPS git URL #ref fragment from the URL stored in the registry."""
728+
raw = (source or "").strip()
729+
if not raw.lower().startswith("https://"):
730+
return raw, ""
731+
parsed = urlsplit(raw)
732+
if not parsed.fragment:
733+
return raw, ""
734+
clean_url = urlunsplit((parsed.scheme, parsed.netloc, parsed.path, parsed.query, ""))
735+
return clean_url, parsed.fragment
736+
737+
738+
def _is_remote_marketplace_json_url(url: str) -> bool:
739+
"""Return True when *url* names a hosted marketplace.json document."""
740+
try:
741+
parsed = urlsplit(url)
742+
except ValueError:
743+
return False
744+
path = (parsed.path or "").rstrip("/")
745+
return parsed.scheme.lower() == "https" and path.endswith("/marketplace.json")
746+
747+
748+
def _should_warn_unpinned_git_url(
749+
source: str,
750+
kind: str,
751+
is_direct_url: bool,
752+
fragment_ref: str,
753+
explicit_ref: bool,
754+
) -> bool:
755+
"""Return True when a git URL source uses the implicit mutable default ref."""
756+
if is_direct_url or fragment_ref or explicit_ref:
757+
return False
758+
return source.lower().startswith("https://") and kind in {"github", "gitlab", "git"}
759+
760+
761+
def _local_source_points_to_file(source) -> bool:
762+
"""Return True when a local marketplace source points directly to a file."""
763+
if source.kind != "local":
764+
return False
765+
try:
766+
return Path(source.local_path).expanduser().is_file()
767+
except OSError:
768+
return False
769+
770+
771+
def _default_alias_from_remote_url(url: str) -> str:
772+
"""Derive a stable default alias for a direct remote marketplace.json URL."""
773+
try:
774+
parsed = urlsplit(url)
775+
except ValueError:
776+
return "marketplace"
777+
host = (parsed.hostname or "marketplace").lower()
778+
return host.split(":", 1)[0]
779+
780+
699781
def _default_alias_from_url(url: str) -> str:
700782
"""Derive a default marketplace alias from a parsed URL.
701783
@@ -850,7 +932,7 @@ def update(name, verbose):
850932
if name:
851933
source = get_marketplace_by_name(name)
852934
logger.start(f"Refreshing marketplace '{name}'...", symbol="gear")
853-
clear_marketplace_cache(name, host=source.host)
935+
clear_marketplace_cache(source=source)
854936
manifest = fetch_marketplace(source, force_refresh=True)
855937
logger.success(
856938
f"Marketplace '{name}' updated ({len(manifest.plugins)} plugins)",
@@ -864,7 +946,7 @@ def update(name, verbose):
864946
logger.start(f"Refreshing {len(sources)} marketplace(s)...", symbol="gear")
865947
for s in sources:
866948
try:
867-
clear_marketplace_cache(s.name, host=s.host)
949+
clear_marketplace_cache(source=s)
868950
manifest = fetch_marketplace(s, force_refresh=True)
869951
logger.tree_item(f" {s.name} ({len(manifest.plugins)} plugins)")
870952
except Exception as exc:
@@ -910,7 +992,7 @@ def remove(name, yes, verbose):
910992
return
911993

912994
remove_marketplace(name)
913-
clear_marketplace_cache(name, host=source.host)
995+
clear_marketplace_cache(source=source)
914996
logger.success(f"Marketplace '{name}' removed", symbol="check")
915997

916998
except Exception as e:

0 commit comments

Comments
 (0)