Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 47 additions & 18 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,8 @@ plugin runtime. The filesystem (or S3) is the only source of truth.
┌───────────────────────────┼───────────────────────────┐
│ │ │
┌──────▼──────┐ ┌───────▼───────┐ ┌───────▼───────┐
│ Docker │ │ Maven │ ... │ Debian │
│ /v2/* │ │ /maven2/* │ (x15) │ /deb/* │
│ Docker │ │ Maven / npm │ ... │ Debian │
│ /v2/* │ │/repository/* │ (x15) │ /deb/* │
└──────┬──────┘ └───────┬───────┘ └───────┬───────┘
│ │ │
└───────────────────────────┼───────────────────────────┘
Expand All @@ -72,9 +72,26 @@ plugin runtime. The filesystem (or S3) is the only source of truth.
└─────────────────────┘
```

Every HTTP request follows this path top-to-bottom. The registry handler is
selected by URL prefix (`/v2/` = Docker, `/maven2/` = Maven, etc.). Curation
runs only on proxy downloads — hosted artifacts are trusted at publish time.
Every HTTP request follows this path top-to-bottom. Most registry handlers are
selected by URL prefix. Maven and npm share `/repository/{name}/...`; a named
dispatcher selects the protocol from validated configuration, not from a name
heuristic. Their repository names therefore occupy one global namespace.
`/maven2/` and `/npm/` remain compatibility aliases only when a default named
repository is configured. Curation runs only on proxy downloads — hosted
artifacts are trusted at publish time.

Named Maven and npm each support hosted, proxy, and ordered group repositories.
Groups own no storage. Maven hosted/proxy objects are isolated under
`maven/repositories/{name}/...`. npm keeps hosted version manifests,
content-addressed tarball blobs, dist-tags, and deprecation overlays under
`npm/repositories/{hosted}/...`, while proxy packuments and tarballs use
`npm/repositories/{proxy}/proxy/...`. An npm group synthesizes a response from
its members in configured order.

This topology has a single-writer contract on local and object storage.
Conditional create protects one immutable key, but mutable Maven/npm metadata
updates span multiple keys and use an in-process lock. S3 or Ceph exact-key CAS
therefore does not provide multi-replica write coordination or HA.

### Trust Boundaries

Expand Down Expand Up @@ -272,9 +289,17 @@ in front of other registries.
**Rationale:** Each storage backend is a maintenance surface. S3 covers
every cloud provider and on-prem S3-compatible stores. Local covers single-node and
development. A third backend (e.g., GCS-native, Azure Blob) adds testing
burden without meaningful capability gain — both are S3-compatible. For
migrating away from other registries, the `nora migrate` CLI copies
artifacts directly rather than proxying through the old system.
burden without meaningful capability gain — both are S3-compatible.
`nora migrate` copies an existing NORA storage layout between local and S3; it
is not a Nexus-to-named-repository migrator.

Named Maven/npm are a fresh-install contract. For Nexus replacement, keep the
source read-only and copy only hosted content through the protocol-aware named
hosted endpoints, then verify reads through the configured groups. Do not copy
Nexus group state or proxy caches. Direct-storage import remains a legacy/local
layout tool and is not supported for named npm or for Maven when named
repositories are configured. No in-place migration from older NORA Maven/npm
layouts is provided.

### ADR-4: Explicit Handlers over Plugin Traits

Expand Down Expand Up @@ -391,9 +416,10 @@ are implemented per-registry following each format's upstream specification.
There is no shared conditional-request middleware.

**Context:** RFC 9110 defines conditional requests for HTTP. Each registry
protocol has its own immutability model: Docker uses content-addressable
digests, Maven/npm/Cargo/PyPI enforce version immutability at publish time,
Raw has no upstream spec. Implementing a generic conditional-request layer
protocol has its own coordinate model: Docker uses content-addressable
digests, Maven/npm apply repository-specific write policy, Cargo/PyPI enforce
version immutability at publish time, and Raw has no upstream spec.
Implementing a generic conditional-request layer
would either be too narrow (not matching protocol-specific semantics) or too
broad (imposing HTTP semantics on protocols that don't need them).

Expand Down Expand Up @@ -445,12 +471,14 @@ added rarely — the explicit approach trades one-time boilerplate for
permanent simplicity, compile-time completeness checks, and full test
coverage of each format in isolation.

**No high availability.** NORA runs as a single instance with a single
RWO volume. This is a design decision, not a missing feature. Artifact
**No high availability.** NORA runs with exactly one writer, including when
the storage backend is S3 or Ceph. Exact-key conditional create does not
serialize the multi-key mutable metadata used by named Maven/npm. This is a
design decision, not a missing feature. Artifact
registries have a read-heavy, write-light workload — a single instance
with S3 storage handles thousands of pulls per minute. Kubernetes
`Recreate` strategy ensures zero-downtime upgrades for reads served from
client-side caches.
with S3 storage handles thousands of pulls per minute. Kubernetes `Recreate`
or an explicit scale-to-zero upgrade prevents overlapping writers, at the cost
of a short service interruption. High availability requires a separate design.

**DRY violations between handlers.** Registry handlers share structural
patterns (proxy logic, curation calls, config loading) but differ in
Expand All @@ -475,5 +503,6 @@ external tools (Grafana dashboards, git-based rule management).
- **Not a CDN.** For geo-distributed artifact delivery, put a CDN
(CloudFront, Cloudflare) in front of NORA.
- **Not a middleware.** NORA is a standalone registry, not a caching
layer in front of Nexus or Artifactory. For migration, use
`nora migrate`.
layer in front of Nexus or Artifactory. Nexus hosted Maven/npm content is
migrated through protocol-aware named endpoints; `nora migrate` only copies
existing NORA storage between local and S3.
10 changes: 8 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
# Changelog
## [Unreleased]

### Added
- **Nexus-style named Maven and npm repositories** — fresh installations can declare independent `hosted`, `proxy`, and `group` repositories under the shared `/repository/{repository}/...` namespace. Maven and npm names are validated globally, group members resolve in configured order, npm groups may nominate one hosted `writable_member`, and groups persist no authoritative objects. Hosted write policy is explicit: `allow` replaces a Maven/npm coordinate and reapplies mutable npm publish state, while `allow_once` (the default) keeps exact retries idempotent and rejects different bytes. An incomplete `allow_once` publish must be retried before later versions/dist-tag/deprecation mutations are accepted, preventing delayed repair from rewinding newer mutable state. npm hosted version manifests reference content-addressed SHA-512 blobs and remain physically separate from proxy packuments/tarballs/negative cache. Plain-text npm search merges hosted/proxy/group results with member-order precedence; every upstream response is capped at 8 MiB, direct proxies preserve exact pagination when namespace filtering is inactive, hosted scans cap packages/versions at 10,000, and filtered direct/group proxy backfill caps results at 10,000 and pages at 40 under a shared 30-second deadline. Group search is fail-soft and labels its successful-member total estimate as approximate, so a failed member can make it an undercount. Bulk, quick, and full audit requests are forwarded only after bounded internal-namespace filtering. Maven repositories are isolated below `maven/repositories/{name}/`. GC, retention, browse index, UI, metrics, auth classification and OpenAPI understand named repositories.

### Changed
- **Maven/npm migration is protocol-first and fresh-layout-only** — old NORA storage layouts are not migrated in place. Nexus hosted content must be copied into named hosted repositories through the Maven/npm protocol endpoints and verified through groups; proxy caches and group materializations are rebuilt, never imported. Direct-storage npm import is rejected, and direct Maven import is rejected when named Maven repositories are configured.

### Fixed
- **Maven keeps server-generated artifact metadata authoritative** — a Maven client that re-uploads a stale artifact-level `maven-metadata.xml` after a concurrent deploy no longer overwrites the version list NORA generates: an uploaded artifact-level metadata document (and its checksums) is recognized by its shape and dropped, while version-level (SNAPSHOT) and group-level (plugin) metadata are still stored verbatim. On a proxy refresh, locally hosted versions are merged into the refreshed upstream document instead of being replaced by it, and the `.md5`/`.sha1`/`.sha256`/`.sha512` sidecars are recomputed from the merged document. The proxy-side merge runs under the same `publish_lock` as the upload-side regeneration, so the document and its checksums are written as one critical section and stay mutually consistent under concurrent fetch and publish (#886).

Expand All @@ -21,7 +27,7 @@
- **Retention over rpm, deb, and raw** — the three formats retention previously skipped silently. rpm/deb versions are collected from the metadata sidecars (payloads never read) and grouped per `{repo}/{arch}/{package}` — structured-layout deb packages per `{repo}/{distribution}/{component}/{arch}/{package}`, so `keep_last` counts within each distribution × architecture's independent index (`all`/`noarch` packages form their own group); deleting a version removes the package and its sidecar, and every touched repo's indexes are rebuilt **and re-signed** under the publish lock afterwards — retention can no longer leave a signed index advertising deleted packages. Raw groups depth-2 path prefixes (`raw/{name}/{version}/…`) as the aging unit, so a directory of related files ages out together; root-level files are never collected. New optional `name_glob` on retention rules targets groups within a registry (e.g. `*-dev-*/*` for an age-only policy on dev repositories, `*-stream-*/*` for a keep-last window) — first matching rule wins, and no matching rule still means keep forever. `nora retention-apply --yes` signs regenerated indexes with the same key as the server.
- **Per-rule namespace scope for OIDC role rules** — an `[[auth.oidc.providers.role_rules]]` entry may set `namespace_scope = ["ci-transport/**"]` to narrow the provider's scope for identities matched by that rule. A write must satisfy **both** the provider scope and the rule scope — the provider scope stays a hard ceiling, and a rule cannot widen past it. Lets one issuer grant, e.g., pull-request CI builds write access confined to a transport prefix while main/tag builds keep the provider-wide scope. Absent = inherit the provider's `namespace_scope`; enforcement mode stays provider-level. Also corrects the config doc example for `role_rules`, which showed a map form that fails to parse (the real shape is an array of tables with `pattern`/`role`).
- **Intra-segment `*` wildcards in `namespace_scope`** — scope patterns now match a `*` within a single path segment (e.g. `team-*/ci`), not only the `**` cross-segment form, so a scope can target a naming convention inside one level without opening the whole subtree.
- **A geo/policy-blocked upstream is now observable** — when a proxied upstream answers a fetch with a 4xx carrying a policy/geo block signature (`x-amzn-waf-reason`, as an AWS CloudFront + WAF geo-block does), NORA emits a `warn` log naming the registry and reason and increments `nora_upstream_policy_blocked_total{registry,reason}`, instead of relaying an anonymous 404 indistinguishable from a genuinely absent artifact. The relayed 404 status and the circuit breaker are deliberately unchanged — a policy block is not an availability failure, so it must not trip the breaker — but it is no longer silent.
- **A geo/policy-blocked upstream is now observable** — when a proxied upstream answers a fetch with a 4xx carrying a policy/geo block signature (`x-amzn-waf-reason`, as an AWS CloudFront + WAF geo-block does), NORA emits a `warn` log naming the registry and reason and increments `nora_upstream_policy_blocked_total{registry,reason}`. Maven maps a policy-blocked 404 to 502 and never stores it in the negative cache, so a group cannot treat the block as an authoritative miss; the circuit breaker remains unchanged because a policy response is not an availability failure.

## [1.0.1] - 2026-07-13

Expand Down Expand Up @@ -54,7 +60,7 @@

### Added
- **Admin-gated admin-token minting (`POST /api/v1/admin/tokens`)** — a dedicated route, reachable only behind the `/api/v1/admin/` gate (`auth::is_admin_path`), that mints an API token of any role including `admin` for a given subject without the `auth.admin_users` self-service check. Anonymous, `anonymous_read`, Basic-auth (no role) and Read/Write callers are denied fail-closed before the handler; `ttl_days = 0` is rejected and every mint is audit-logged (actor, target, role, ttl — never the token). `auth.admin_users` (`NORA_AUTH_ADMIN_USERS`) thus becomes a bootstrap-only fallback on the unchanged public `POST /api/tokens` route, so GHSA-78cx-cfhm-rgmx stays closed; with auth disabled the route returns `503` (#746, #808).
- **`npm audit` proxied to upstream for remote repos** — `npm audit` POSTs to `/-/npm/v1/security/advisories/bulk` (npm7) or `/-/npm/v1/security/audits/quick` (npm6), which previously hit the `405` fallback and failed. NORA keeps no advisory database, so for a proxy repo it now forwards the request to the configured upstream and returns the response verbatim. Both audit POSTs are read-eligible under `auth.anonymous_read`, so anonymous `npm audit` works wherever anonymous install does; non-audit npm POSTs stay gated. Under an active `internal_namespaces` filter the `bulk` request strips internal-package keys before forwarding and fail-closes (`200 {}`) on any body it cannot verify, the gzipped `quick` lockfile is refused wholesale, the client `Authorization` is never forwarded, and the body is bounded at 8 MB; upstream 5xx/network → `502`, circuit-open → `503`, no proxy configured → `200 {}` (#597, #805).
- **`npm audit` proxied to upstream for remote repos** — `npm audit` POSTs to `/-/npm/v1/security/advisories/bulk` (npm7) or `/-/npm/v1/security/audits/quick` (npm6), which previously hit the `405` fallback and failed. NORA keeps no advisory database, so for a proxy repo it now forwards the request to the configured upstream and returns a fully read response verbatim. Both audit POSTs are read-eligible under `auth.anonymous_read`, so anonymous `npm audit` works wherever anonymous install does; non-audit npm POSTs stay gated. Under an active `internal_namespaces` filter the `bulk` request strips internal-package keys before forwarding and fail-closes with `400` on any body it cannot verify, the gzipped `quick` lockfile is refused wholesale, the client `Authorization` is never forwarded, and both request and upstream-response bodies are bounded at 8 MiB; truncated/oversized responses and upstream 5xx/network → `502`, circuit-open → `503`, no proxy configured → explicit `400` (#597, #805).
- **S3 virtual-hosted-style addressing (`storage.s3_virtual_hosted`, `NORA_STORAGE_S3_VIRTUAL_HOSTED`)** — some S3-compatible providers reject signed path-style requests (Alibaba Cloud OSS answers `403 SecondLevelDomainForbidden`), which made the S3 backend unusable there because the addressing style was hardcoded to path-style. A new default-off toggle threads through `Storage::new_s3` / `S3Storage::new` into `AmazonS3Builder::with_virtual_hosted_style_request`; when enabled, `object_store` uses the configured endpoint verbatim, so it must include the bucket host. Default (`false`) preserves current path-style behavior (#795, #798).
- **Chinese (Simplified) UI translation** — a `中文` entry in the language switcher backed by a full `zh` translation table; language detection now normalizes BCP-47 / POSIX tags to their primary subtag, so `zh-CN`, `zh-Hans` and `ru_RU.UTF-8` resolve correctly (#788).

Expand Down
Loading
Loading