upload: per-arch distributions in results JSON; optional logs/SBOM repos

Author: jajreidy

.cursor/rules/llm-development-guidelines.mdc

@@ -69,13 +69,13 @@ When you do generate it, use a clear heading such as `### PR description (GitHub
    make test
    ```
    **IMPORTANT**: Use the full test suite, not only a single test file, so regressions surface. If anything fails, fix it and **run `make test` again**. **Repeat** until all tests pass.
-7. **Verify 100% diff coverage for GitHub merge** - Every new or changed line in the PR must be executed by tests (CI blocks merge below 100% on the diff). Use coverage output from `make test` and/or:
+7. **Verify 100% diff coverage for GitHub merge** - CI uses `diff-cover` on the PR diff vs the base branch (same as `make test-diff-coverage`). After `git fetch origin`, run:
    ```bash
-   python3 -m pytest --cov=pulp_tool --cov-report=term-missing
+   make test-diff-coverage
    ```
-   If the diff is not fully covered, add or extend tests and **return to step 6** until tests pass and the PR diff would satisfy diff coverage.
+   For a PR targeting a branch other than `main`, use `make test-diff-coverage COMPARE_BRANCH=origin/<base>`. If this fails, add or extend tests and **return to step 6** until the diff is fully covered. Use `python3 -m pytest --cov=pulp_tool --cov-report=term-missing` only for debugging specific files; the **authoritative local gate** is `make test-diff-coverage`.
 
-**Never skip linting, pre-commit (must finish with a fully passing run), or tests (must finish with a fully passing `make test` and diff-ready coverage)** - They will fail in CI/CD and waste pipeline resources.
+**Never skip linting, pre-commit (must finish with a fully passing run), or tests (must finish with a fully passing `make test` and `make test-diff-coverage` before a PR)** - They will fail in CI/CD and waste pipeline resources.
 
 ## Linting Tools
 
@@ -189,6 +189,12 @@ pre-commit run flake8 --all-files
 # Run all tests with coverage (REQUIRED for new changes)
 make test
 
+# Same as PR CI: 100% of lines changed vs merge base (run after git fetch origin)
+make test-diff-coverage
+
+# Optional: non-main base branch
+make test-diff-coverage COMPARE_BRANCH=origin/my-release-branch
+
 # Run tests without coverage (faster, but won't verify coverage)
 make test-fast
 
@@ -199,15 +205,16 @@ python3 -m pytest tests/path/to/test_file.py -v
 python3 -m pytest --cov=pulp_tool/path/to/module --cov-report=term-missing tests/
 ```
 
-**IMPORTANT**: New changes require **100% coverage on the PR diff** for GitHub merge. Always run `make test` and confirm your branch would pass diff coverage before opening or updating a PR.
+**IMPORTANT**: New changes require **100% coverage on the PR diff** for GitHub merge. Always run `make test-diff-coverage` (not only `make test`) before opening or updating a PR, after ensuring the compare branch exists locally (`git fetch origin`).
 
 ### Comprehensive Checks
 
 ```bash
 # Run all checks (lint + test)
 make check
 
-# Run comprehensive check script (lint + test + coverage)
+# Lint + tests + optional 100% diff-cover (see scripts/check-all.sh header)
+git fetch origin
 ./scripts/check-all.sh
 ```
 
@@ -234,6 +241,7 @@ make format                  # Fix formatting if needed
 # Loop until pre-commit passes completely:
 pre-commit run --all-files   # fix failures → run again → repeat until green
 make test                    # fix failures / coverage → run again → repeat until green
+make test-diff-coverage      # before PR: 100% on git diff vs origin/main (git fetch origin first)
 ```
 
 ### 2. Fix Linting Errors Immediately
@@ -283,21 +291,23 @@ When in doubt, check the configuration files:
 Run tests with coverage until the branch is PR-ready:
 
 ```bash
-make test    # Full test suite with coverage (REQUIRED)
+make test                 # Full test suite with coverage (REQUIRED)
+make test-diff-coverage   # After git fetch origin: 100% on PR diff vs origin/main (REQUIRED before PR)
 ```
 
-Review the coverage report for **lines that belong to your PR diff**. If anything is uncovered or tests fail:
+`make test-diff-coverage` runs `diff-cover` on `coverage.xml` and lists **only uncovered lines that are part of the git diff**—the same signal as GitHub PR CI. If anything is uncovered or tests fail:
 
 1. Fix tests or implementation, add tests for uncovered paths (edge cases, branches, errors)
-2. Re-run `make test` until **all tests pass** and **100% of the diff** is covered (GitHub merge gate).
+2. Re-run `make test` then `make test-diff-coverage` until both pass.
 
 Treat “green once” as insufficient if a later fix breaks coverage—re-run the full suite after each substantive change.
 
 ### 7. Use Scripts for Comprehensive Checks
 
-The `scripts/check-all.sh` script runs all checks including tests:
+The `scripts/check-all.sh` script runs linters, tests (with `coverage.xml`), and—when `diff-cover` is installed and `origin/main` (or `DIFF_COVER_COMPARE_BRANCH`) exists—checks PR-style diff coverage at 100%:
 
 ```bash
+git fetch origin   # recommended so diff-cover can compare
 ./scripts/check-all.sh
 ```
 
@@ -360,13 +370,12 @@ Example addition under `[Unreleased]`:
 
 **Solution**: The **merge gate is 100% coverage on the PR diff**. Ensure every new or changed line in the branch is executed by tests:
 
-1. Run `make test` to see coverage report
-2. Identify uncovered lines in your changes (match them to the diff GitHub/CI reports)
-3. Write tests for all uncovered code paths
-4. Include edge cases, error conditions, and all branches
-5. Re-run `make test` until all tests pass **and** diff coverage is 100% on your changes
+1. Run `git fetch origin` (or your PR base remote ref) so the compare branch exists
+2. Run `make test-diff-coverage`; it prints only **diff** uncovered lines (same tool as CI)
+3. Write tests for all uncovered code paths (edge cases, errors, branches)
+4. Re-run `make test` then `make test-diff-coverage` until both pass
 
-The CI/CD pipeline uses `diff-cover` (or equivalent) to check only new/changed lines against the merge base; **anything short of 100% on that diff blocks merging the PR in GitHub.** Keep extending tests and re-running `make test` until that bar is met.
+The CI/CD pipeline uses `diff-cover` against the merge base; **anything short of 100% on that diff blocks merging the PR in GitHub.**
 
 ## References
 
@@ -385,14 +394,16 @@ The CI/CD pipeline uses `diff-cover` (or equivalent) to check only new/changed l
 1. ✅ Run `make lint` to check for linting errors
 2. ✅ Run `make format` to ensure proper formatting
 3. ✅ Run `pre-commit run --all-files` and **repeat** (fix → re-run) until **all hooks pass**
-4. ✅ Run `make test` (full suite) and **repeat** (fix → re-run) until **all tests pass** and **100% diff coverage** is satisfied for GitHub merge
-5. ✅ If there are substantive edits that could ship as a PR, **ask whether the user wants PR materials**; **only after they confirm**, draft the paste-ready GitHub PR description **and** update `CHANGELOG.md` when required (same turn)—see **GitHub PR description (ask first; generate on confirmation)** and **Update CHANGELOG.md when drafting the PR**
+4. ✅ Run `make test` (full suite) and **repeat** (fix → re-run) until **all tests pass**
+5. ✅ Run `make test-diff-coverage` after `git fetch origin` and **repeat** (fix → re-run) until **100% diff coverage** passes (same gate as PR CI)
+6. ✅ If there are substantive edits that could ship as a PR, **ask whether the user wants PR materials**; **only after they confirm**, draft the paste-ready GitHub PR description **and** update `CHANGELOG.md` when required (same turn)—see **GitHub PR description (ask first; generate on confirmation)** and **Update CHANGELOG.md when drafting the PR**
 
 **Critical Requirements**:
 - **Pre-commit must end clean** - Run `pre-commit run --all-files` in a loop until zero failures; do not stop after one failed run without a subsequent fully passing run
-- **Tests must end green with diff coverage** - Run `make test` in a loop until the suite passes and new/changed lines are covered for the PR diff
-- **100% diff coverage for GitHub merge** - CI enforces this; extend tests and re-run until the diff is fully covered
+- **Tests must end green** - Run `make test` in a loop until the suite passes
+- **100% diff coverage before a PR** - Run `make test-diff-coverage` (compare branch defaults to `origin/main`); CI enforces the same; extend tests and re-run until the diff is fully covered
 - **Ask before PR text** - End with a short question about drafting the GitHub PR; provide the full paste-ready body only after the user says they are ready
 - **`CHANGELOG.md` at PR draft time only** - Do not touch the changelog on every iteration; add `[Unreleased]` entries when the user confirms PR drafting (unless they explicitly ask earlier)
+- **`make test-diff-coverage`** - Treat as required before declaring a branch PR-ready alongside `make test`
 
 Following this workflow ensures your changes will pass CI/CD checks and avoids unnecessary pipeline runs.

CHANGELOG.md

@@ -8,6 +8,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Added
+- `make test-diff-coverage` runs `diff-cover` at 100% vs `COMPARE_BRANCH` (default `origin/main`) after `make test`, matching the PR merge gate; `scripts/check-all.sh` also generates `coverage.xml` and runs `diff-cover` when the tool and compare ref exist (`DIFF_COVER_COMPARE_BRANCH` optional)
+- `upload --target-arch-repo`: `pulp_results.json` includes per-architecture RPM distribution base URLs under `distributions` with keys `rpm_<arch>` (e.g. `rpm_x86_64`); serialized `distributions` uses sorted keys for stable `{name: url}` output, alongside build-scoped entries when those repos exist
+- Upload optionally skips creating logs and SBOM repositories when no log or SBOM uploads are expected; `skip_logs_repo` / `skip_sbom_repo` on `UploadContext` and `PulpHelper.setup_repositories` (defaults preserve creating all repos for programmatic callers who omit the flags)
 - `upload --target-arch-repo`: per-architecture RPM repos/distributions (``{namespace}/{arch}/Packages/...``); logs/SBOM/artifacts stay build-scoped; lazy repo creation at upload; works with `--results-json`, `--signed-by`, and `--overwrite`; with `--signed-by`, same arch repo and `signed_by` is label-only
 - `upload --overwrite`: RPM-only; remove existing RPM package units in the target repo that match local file SHA256 (and `signed_by` when set) via `remove_content_units` before upload
 - `upload --results-json`: Upload artifacts from pulp_results.json; files resolved from JSON directory or --files-base-path; --build-id and --namespace optional (extracted from artifact labels)
@@ -21,6 +24,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Username/password (Basic Auth) support for packages.redhat.com
 
 ### Changed
+- `upload --target-arch-repo`: `pulp_results.json` `distributions` keys for per-arch RPM bases are `rpm_<arch>` instead of bare architecture names (e.g. `rpm_x86_64` not `x86_64`)
+- `upload` / `upload-files`: infer whether log and SBOM repos are needed before repository setup (directory `*.log` scan or `--results-json` artifact keys; SBOM via `--sbom-path` or SBOM-classified keys); omitted types are excluded from results `distributions`; clear errors if uploads are attempted without the matching repository
 - Upload orchestration uses `RpmUploadResult` per architecture instead of ad-hoc dicts; gather/collect uses `PulpContentRow`, `ExtraArtifactRef`, and `FileInfoMap` for clearer typed data flow
 - Upload flow populates `pulp_results.json` artifact entries incrementally as RPMs, logs, SBOMs, and generic files finish; final gather still reconciles via merge (keeps incremental entries when keys already exist)
 - Repository setup logs use the concrete repo slug (e.g. ``rpms-signed``) instead of a generic ``Rpms`` label; distribution creation logs state that ``name`` and ``base_path`` match the repository name on one line

CONTRIBUTING.md

@@ -114,6 +114,11 @@ def my_function(client: "PulpClient", value: Optional[str] = None) -> bool:
 # Run all tests with coverage (preferred; enforces 85%+)
 make test
 
+# Before a PR: full suite + 100% coverage on git diff vs origin/main (matches CI)
+# git fetch origin   # ensure compare branch exists
+make test-diff-coverage
+# Override base branch: make test-diff-coverage COMPARE_BRANCH=origin/my-base
+
 # Or with pytest directly
 pytest -v --tb=short --cov=pulp_tool --cov-report=term-missing --cov-fail-under=85
 
@@ -130,6 +135,7 @@ pytest -m integration
 - All new code must include tests
 - Overall test coverage must remain at or above 85%
 - New and changed lines must have 100% coverage (enforced in CI via diff coverage)
+- **Before opening or updating a PR**, run `make test-diff-coverage` after `git fetch origin` so your local report matches the merge gate (`diff-cover` vs `origin/main` by default). Use `make test-diff-coverage COMPARE_BRANCH=origin/<base>` if the PR targets another branch.
 - Tests should be fast and isolated
 - Use fixtures from `tests/conftest.py` when possible
 - Mark slow tests with `@pytest.mark.slow`
@@ -166,13 +172,15 @@ def test_upload_content_success():
 1. **Update CHANGELOG.md**: Add an entry describing your changes
 2. **Update documentation**: If adding features, update README.md as needed
 3. **Run all checks**: Ensure `make check` passes (or run `make lint`, then `pre-commit run --all-files` twice, then `make test`)
-4. **Write tests**: Add tests for new functionality (new/changed code requires 100% diff coverage)
-5. **Update type hints**: Ensure all functions have proper type annotations
+4. **Verify PR diff coverage**: Run `make test-diff-coverage` (requires `git fetch origin` so `origin/main` or your `COMPARE_BRANCH` exists); CI fails below 100% on the diff
+5. **Write tests**: Add tests for new functionality (new/changed code requires 100% diff coverage)
+6. **Update type hints**: Ensure all functions have proper type annotations
 
 ### PR Checklist
 
 - [ ] Code follows the project's style guidelines
 - [ ] All tests pass (`make test`)
+- [ ] PR diff coverage is 100% (`make test-diff-coverage` after `git fetch origin`)
 - [ ] All linters pass (`make lint`)
 - [ ] Pre-commit hooks pass (`pre-commit run --all-files`, run twice after fixing issues)
 - [ ] Test coverage is maintained or improved (85%+ overall, 100% for new/changed lines)

Makefile

@@ -1,17 +1,22 @@
 # Makefile for pulp-tool development tasks
 
-.PHONY: help install install-dev test lint format check clean
+COMPARE_BRANCH ?= origin/main
+
+.PHONY: help install install-dev test test-diff-coverage lint format check clean
 
 # Default target
 help:
 	@echo "Available targets:"
 	@echo "  make install      - Install package"
 	@echo "  make install-dev  - Install package with dev dependencies"
 	@echo "  make test         - Run tests with coverage"
+	@echo "  make test-diff-coverage - make test + diff-cover 100% vs COMPARE_BRANCH (same gate as PR CI)"
 	@echo "  make lint         - Run all linters"
 	@echo "  make format       - Format code with Black"
 	@echo "  make check        - Run all checks (lint + test)"
 	@echo "  make clean        - Clean build artifacts"
+	@echo ""
+	@echo "  Diff coverage base: COMPARE_BRANCH=origin/main (override for e.g. origin/release-1.0)"
 
 # Installation
 install:
@@ -25,6 +30,13 @@ install-dev:
 test:
 	python3 -m pytest -v --tb=short --cov=pulp_tool --cov-report=term-missing --cov-report=html --cov-report=xml --cov-fail-under=85
 
+# Same as GitHub Actions: 100% coverage on lines changed vs merge base (requires coverage.xml from test).
+# Requires diff-cover (included in pip install -e ".[dev]" / make install-dev).
+test-diff-coverage: test
+	@command -v diff-cover >/dev/null 2>&1 || { echo "diff-cover not found. Run: make install-dev"; exit 1; }
+	@echo "Diff coverage vs $(COMPARE_BRANCH) (fail under 100%)..."
+	diff-cover coverage.xml --compare-branch=$(COMPARE_BRANCH) --fail-under=100
+
 test-fast:
 	python3 -m pytest -v --tb=short
 

README.md

@@ -139,7 +139,7 @@ Upload RPM packages, logs, and SBOM files.
 
 **Overwrite:** When `--overwrite` is set, for each RPM about to be uploaded the tool searches Pulp by the file’s SHA256 (same mechanism as `search-by` checksum mode), keeps only matches that exist in the target RPM repository’s latest version, then calls the repository modify API with `remove_content_units` before uploading and adding the new RPMs. Use with `--signed-by` to scope the search to signed packages.
 
-**Target-arch-repo:** When `--target-arch-repo` is set, RPM repositories and distributions are named by architecture only (`{arch}`), including when `--signed-by` is set (no separate `rpms-signed` path). Published paths look like `…/pulp-content/{namespace}/{arch}/`. The aggregate `{build}/rpms` repo is not created; RPM repos are created when each arch is uploaded. Results JSON uses each artifact’s `arch` label to build RPM URLs. Logs, SBOM, and generic artifacts still use `{build}/logs`, `{build}/sbom`, and `{build}/artifacts`.
+**Target-arch-repo:** When `--target-arch-repo` is set, RPM repositories and distributions are named by architecture only (`{arch}`), including when `--signed-by` is set (no separate `rpms-signed` path). Published paths look like `…/pulp-content/{namespace}/{arch}/`. The aggregate `{build}/rpms` repo is not created; RPM repos are created when each arch is uploaded. `pulp_results.json` `distributions` maps string names to base URLs (sorted keys when serialized); per-arch RPM bases use keys `rpm_<arch>` (e.g. `rpm_x86_64`). Logs, SBOM, and generic artifacts still use `{build}/logs`, `{build}/sbom`, and `{build}/artifacts`.
 
 ### upload-files
 
@@ -291,14 +291,15 @@ dist.pull_data(filename="pkg.rpm", file_url="...", arch="x86_64", artifact_type=
 ## Development
 
 ```bash
-make install-dev   # Install with dev deps + pre-commit
+make install-dev   # Install with dev deps + pre-commit (includes diff-cover)
 make test          # Run tests with coverage (85%+ required)
+make test-diff-coverage  # After git fetch origin: 100% coverage on git diff vs origin/main (PR CI gate)
 make lint          # Black, flake8, pylint, mypy
 make format        # Format with Black
 make check         # Lint + test
 ```
 
-Before committing: `pre-commit run --all-files` (run twice after fixes). See [CONTRIBUTING.md](CONTRIBUTING.md).
+Before committing: `pre-commit run --all-files` (run twice after fixes). Before a PR: `git fetch origin` and `make test-diff-coverage`. See [CONTRIBUTING.md](CONTRIBUTING.md).
 
 ## Troubleshooting
 
@@ -314,7 +315,7 @@ Before committing: `pre-commit run --all-files` (run twice after fixes). See [CO
 1. Fork the repository
 2. Create a feature branch
 3. Make changes and add tests
-4. Run `make test` and `pre-commit run --all-files`
+4. Run `make test`, `make test-diff-coverage` (after `git fetch origin`), and `pre-commit run --all-files`
 5. Submit a pull request
 
 ## License