Full rebuild mode empties non-stable distributions due to incomplete per-distribution pool migration

[mairas]

Summary

The scheduled daily rebuild (Nov 17, 06:00 UTC) emptied all non-stable distributions (trixie-unstable, bookworm-unstable, etc.) because full rebuild mode was not updated to handle the per-distribution pool structure introduced in PR #16.

Impact

• Severity: HIGH
• Affected: All unstable and distribution-specific repositories
• Lost packages: cockpit-apt_0.2.0-1_all.deb from trixie-unstable
• Current state: Only stable distribution receives packages during scheduled rebuilds

Root Cause

Commit dfb2b04 (Nov 15) changed from shared pool/main/ to distribution-specific pools like pool/trixie-unstable/main/. However, full rebuild mode (.github/workflows/update-repo.yml lines 217-237) was not updated:

Current buggy behavior:

# Full rebuild mode
echo "\${{ steps.discover.outputs.repos }}" | while IFS= read -r repo; do
    download_from_repo "$repo" "latest"  # ❌ Hardcoded to stable releases only
done

Problems:

1. Downloads only stable releases (not pre-releases for unstable channels)
2. Downloads go to pool/stable/main/ (default when TARGET_DIST not set)
3. Each distribution now scans only its own pool directory
4. Result: Non-stable distributions find nothing and become empty

Evidence

• Git commit 95ec949 (Nov 17, ~06:00 UTC): delete mode 100644 pool/trixie-unstable/main/cockpit-apt_0.2.0-1_all.deb
• Commit message from dfb2b04 warned: "This is a breaking change in pool structure. Migration strategy for existing packages needs to be decided before deployment."

Expected Behavior

Full rebuild mode should:

1. Iterate through each distribution separately
2. Determine channel from distribution name (e.g., "trixie-unstable" → "unstable")
3. Download appropriate release types (stable=latest, unstable=prerelease)
4. Set TARGET_DIST correctly so packages land in correct pool directories

Proposed Fix

Update .github/workflows/update-repo.yml full rebuild mode to iterate through distributions and set appropriate release tags and target distributions for each.

Immediate Actions Needed

1. Restore missing packages: Trigger targeted updates to repopulate unstable distributions
2. Disable scheduled rebuilds: Comment out schedule: '0 6 * * *' until fix is verified
3. Implement fix: Update full rebuild mode logic
4. Add validation: Check package counts before/after rebuild to detect this issue

Related

• PR feat: Implement per-distribution package filtering #16: feat: implement per-distribution package filtering
• Commit dfb2b04: Initial per-distribution pool implementation
• Commit 95ec949: Deploy that emptied trixie-unstable

[mairas]

Additional Critical Finding: Push Trigger Also Causes Full Rebuild

Actual Root Cause of Nov 16 Package Loss

The package was not removed by the scheduled rebuild (which runs at 06:00 UTC). Instead, it was removed at Nov 16, 21:48:27 UTC when PR #23 (adding LICENSE file) was merged to main.

Timeline:

• Nov 16, 23:47:58 +0200 (21:47:58 UTC): PR docs: add MIT LICENSE file #23 merged to main
• Nov 16, 21:48:27 UTC: Push to main triggered workflow → full rebuild mode → trixie-unstable emptied
• Nov 17, 06:01:01 UTC: Scheduled rebuild ran (also full rebuild, but damage already done)

Second Critical Bug: Push Trigger

The workflow has this trigger:

on:
  push:
    branches:
      - main  # ← Triggers full rebuild on EVERY main branch push

Since push events have no client_payload.repository, MODE is always set to "full rebuild", which runs the buggy logic.

Impact:

• Every documentation change triggers full rebuild and empties non-stable distributions
• Every workflow update triggers full rebuild
• Every PR merge to main triggers full rebuild
• This happens multiple times per day, not just once at 06:00 UTC

Recommended Fix

Either:

1. Remove the push trigger entirely - rely only on repository_dispatch for updates
2. Add path filtering - only trigger on changes to workflows or scripts:

   push:
     branches:
       - main
     paths:
       - '.github/workflows/**'
       - 'scripts/**'

3. Make push trigger use targeted mode - but this requires determining what changed

Option 1 (remove push trigger) is simplest and safest until the full rebuild mode is fixed.

[mairas]

Implementation Plan for Issue #24

Following DEVELOPMENT_WORKFLOW.md Explore-Plan-Code-Commit pattern.

Phase 1: Explore ✓

• Identified two critical bugs in update-repo.yml:
  1. Full rebuild mode incompatible with per-distribution pools (introduced in PR feat: Implement per-distribution package filtering #16)
  2. Push trigger causes unintended full rebuilds on every main branch change

Phase 2: Plan

Bug 1: Full Rebuild Mode Logic

Root Cause:

• Lines 230-237: Full rebuild mode hardcodes download_from_repo "$repo" "latest"
• Downloads only stable releases to pool/stable/main/
• Per-distribution pools (from PR feat: Implement per-distribution package filtering #16) mean trixie-unstable scans only pool/trixie-unstable/main/
• Result: Non-stable distributions find no packages and become empty

Fix Approach:

• Iterate through each distribution in full rebuild mode
• For each distribution, determine channel from name:
  • "*-unstable" or "unstable" → channel="unstable" → RELEASE_TAG="prerelease"
  • "*-stable" or "stable" → channel="stable" → RELEASE_TAG="latest"
• Set TARGET_DIST before downloading so packages go to correct pool
• Call download_from_repo separately for each distribution

Implementation Steps:

1. Extract distribution iteration logic (lines 230-237)
2. Add for loop: for dist in ${{ steps.payload.outputs.all_distributions }}
3. Determine channel from distribution name
4. Set TARGET_DIST=$dist before calling download_from_repo
5. Add logging to show what's being downloaded for each distribution

Bug 2: Push Trigger

Root Cause:

• Lines 9-11: Workflow triggers on push to main
• Push events have no client_payload.repository
• MODE is set to "full" instead of "targeted"
• Every documentation/workflow change triggers full rebuild

Fix Approach:

• Remove push trigger entirely (simplest and safest)
• Rely on repository_dispatch for package updates
• Rely on manual workflow_dispatch if needed
• No more unintended full rebuilds from CI activity

Implementation Steps:

1. Delete lines 9-11 (push trigger)
2. Keep repository_dispatch, workflow_dispatch, and schedule

Phase 3: Testing & Verification Strategy

Since GitHub Actions workflows can't be unit tested, verification will involve:

Manual Testing:

1. Trigger full rebuild manually using workflow_dispatch
2. Check gh-pages deployment logs for:
   • Each distribution being processed
   • Correct RELEASE_TAG being used for each distribution
   • Packages being downloaded to correct pool directories
3. Verify each distribution's Packages file:
   • stable: Should have packages
   • unstable: Should be empty (no pre-releases exist)
   • trixie-unstable: Should be populated after restoring cockpit-apt
   • bookworm-unstable: Should be empty initially

Logging Added:

• Show which distributions are processed in full rebuild mode
• Show which RELEASE_TAG is being used for each distribution
• Show package count for each distribution after rebuild

Restoration Step:

1. After fix is merged, trigger targeted update from cockpit-apt
2. Verify cockpit-apt_0.2.0-1_all.deb appears in trixie-unstable
3. Verify all other distributions remain intact

Phase 4: Verification Criteria

Fix is complete when:

• ✅ Push trigger removed (no more unintended full rebuilds)
• ✅ Full rebuild mode processes each distribution separately
• ✅ Each distribution gets correct release type (stable=latest, unstable=prerelease)
• ✅ Packages are downloaded to correct pool directories
• ✅ Logging shows the process clearly
• ✅ cockpit-apt restored to trixie-unstable
• ✅ All distributions have expected package counts
• ✅ PR documents the bug, fix, and testing steps

Files to Modify

1. .github/workflows/update-repo.yml
   • Remove push trigger (lines 9-11)
   • Fix full rebuild logic (lines 230-237)
   • Add logging for verification

Branch & PR Strategy

• Branch: fix/full-rebuild-per-distribution-pools
• Single comprehensive PR with both bug fixes
• PR body will document:
  • Both bugs and their root causes
  • Testing performed
  • Manual verification steps
  • Link to this issue

[mairas]

Implementation Complete

PR #25 created with full implementation of both bug fixes:

What was fixed:

1. Removed push trigger - No more unintended full rebuilds on every main branch merge
2. Rewrote full rebuild logic - Each distribution now gets correct release type:
   • Stable distributions → download from latest releases
   • Unstable distributions → download from pre-releases
   • All packages go to correct distribution-specific pools

Testing approach documented in PR #25:

• Manual verification via workflow_dispatch
• Log checks to confirm each distribution processed correctly
• Package count verification
• Restoration of missing cockpit-apt_0.2.0-1_all.deb via targeted update

Next steps (after PR merge):

1. Merge PR fix: resolve full rebuild mode bugs in APT repository workflow (issue #24) #25 to main
2. Trigger full rebuild manually to verify fix
3. Trigger targeted update from cockpit-apt to restore trixie-unstable
4. Verify all distributions have expected packages

See PR #25 for full testing strategy and implementation details.