Merge remote-tracking branch 'origin/main' into pmm

Author: yyyyyyyan

.github/instructions/migrations.instructions.md

@@ -17,7 +17,7 @@ SEP has three independent Alembic migration tracks, one per sub-application. Eac
 CI runs `make checkmigrations` on every PR — it catches missing migrations and out-of-date heads automatically. Don't manually flag "model changed but no migration": CI already does. Focus on the manual-judgment items below.
 
 - **Migration lives in the right track.** A model change under `app/sep/` MUST have its migration in `app/sep/migrations/versions/` (or under `app/sep/apps/<name>/migrations/versions/` for a sub-app that owns its own chain — those dirs are auto-listed in `[sep] version_locations` by `scripts/sync_alembic_version_locations.py`), NOT under `app/tasks/migrations/` or `app/inventory/migrations/`. Cross-track placement is a hard fail — the migration won't run on the right database.
-- **Deleting a sub-app's migration chain is an explicit step.** The sync script refuses to drop an entry already listed in `[sep] version_locations`, because that configured-but-absent path is how the orphan-head filter recognises a stripped app. A deliberate deletion needs `python scripts/sync_alembic_version_locations.py --allow-removals`; anything else (a locally stripped app, a half-deleted package) should restore the directory, or skip the script and run `alembic --name sep upgrade heads` directly — the refusal leaves the entry in place, which is exactly what arms the filter.
+- **Deleting a sub-app's migration chain is an explicit step.** The sync script refuses to drop an entry already listed in `[sep] version_locations`, because a configured location that is absent from disk or contributes no migration scripts is how the orphan-head filter recognises a stripped app. A deliberate deletion needs `python scripts/sync_alembic_version_locations.py --allow-removals`; anything else (a locally stripped app, a half-deleted package) should restore the directory, or skip the script and run `alembic --name sep upgrade heads` directly — the refusal leaves the entry in place, which is exactly what arms the filter.
 - **`upgrade()` and `downgrade()` are both implemented.** Empty `downgrade()` (just `pass`) is a red flag — migrations must be reversible.
 - **`downgrade()` reverses `upgrade()`.** New tables in `upgrade()` need `op.drop_table()` in `downgrade()`; new columns need `op.drop_column()`; new indexes need `op.drop_index()`. The pair must be symmetric. A `downgrade()` that re-narrows a column's allowed domain (removes enum members, re-adds `NOT NULL`, shrinks `VARCHAR` length) must first `DELETE`/`UPDATE` rows holding the now-invalid values — otherwise it aborts at runtime the moment such data exists.
 - **No data loss without a written strategy.** `op.drop_table()` or `op.drop_column()` on a column that has live data needs an explicit data-migration step (or a justification in the PR description). Same for column type changes that may truncate (`TEXT` → `INTEGER`, `VARCHAR(255)` → `VARCHAR(50)`). A parser branch or deserializer that silently remaps an old stored value at read time is NOT a substitute for a one-shot data migration (`UPDATE`/`DELETE`) — flag it.

.github/instructions/python-docstrings.instructions.md

@@ -83,7 +83,7 @@ The reverse of overstating: when a docstring uses applicability language (`Use w
 - **Describe intent, not a code-controlled enumeration** — a docstring listing the members of a set the code controls (a `parametrize` list, a module constant, registry entries) goes stale the moment the set changes. Name the intent and the controlling symbol ("every app in `BESPOKE_BASE_APP_PLUGINS`"), not the members. Applies in `tests/` too.
 - **Framework/core symbols stay app-agnostic** — a shared framework/core symbol's docstring describes its behaviour in generic terms; naming one downstream app's domain concept (`backup_type`, `snippet_filename`) couples the abstraction's contract to one consumer. Restate as the generic role.
 - **State the role; don't pin what the reader can grep.** Two shapes rot on the next change and are never edited at the change site: **a count** ("read by seven non-alerts apps", "the only two callers") and **an enumeration of callers or consumers**. Write the *property* instead — not "read by seven non-alerts apps" but "read by every app offering `alert_on_fail`"; not "used by `a.py`, `b.py`, `c.py`" but "shared across the subtree". When the enumeration is genuinely load-bearing, the enforcement belongs in code (a registry, an `__all__`, a guard), with the docstring pointing at it — prose is not a mechanism.
-- **A parity claim is scoped to the members it covers.** "The expected row orders are the same literals `TestListQueryPaginatedPostgres` asserts" reads as a guarantee over the whole class; when it holds for only some members it is false for the rest and nothing marks which. Scope the assertion and name the exceptions ("…the all-NULL and `select_related` cases are MySQL-only"). Cross-dialect test classes are the recurring shape.
+- **A parity claim is scoped to the members it covers.** "The expected row orders are the same literals `TestListQueryPaginatedPostgres` asserts" reads as a guarantee over the whole class; when it holds for only some members it is false for the rest and nothing marks which. Scope the assertion and name the exceptions ("…the all-NULL case is PostgreSQL-only"). Cross-dialect test classes are the recurring shape.
 
 ## Say it once, in the surface that owns it
 

.github/instructions/python-duplication.instructions.md

@@ -42,7 +42,7 @@ Concrete cases: `3306`/`5432` ports → `DEFAULT_MYSQL_PORT`/`DEFAULT_POSTGRESQL
 | `len(v) > 0` `field_validator` | `NonEmptyStr` |
 | `.strip().lower()` `field_validator` | `Annotated[str, StringConstraints(strip_whitespace=True, to_lower=True)]` or `LowercaseStr` |
 | `field_validator` doing a string-*shape* check (split on a separator, reject empty halves, reject stray whitespace) | `Annotated[str, StringConstraints(pattern=...)]` field type |
-| `.nulls_last()` on an `ORDER BY` term | `app/core/db/utils.py::NullsLastOrdering(column, *, descending=False)` — `.nulls_last()` emits SQL MySQL cannot parse. Pass the bare column plus `descending=`, never a pre-`desc()`-ed expression |
+| `.nulls_last()` on an `ORDER BY` term | `app/core/db/utils.py::NullsLastOrdering(column, *, descending=False)` — one shared cache-keyed construct; pass the bare column plus `descending=`, never a pre-`desc()`-ed expression (which would render `<expr> DESC ASC NULLS LAST`) |
 
 **Rule of thumb.** If a new decorator or class has 15+ lines of state management (timestamps, eviction, key hashing, TTL math), ask "why isn't this `@alru_cache` or `@ttl_cache`?" Flag as **Important**.
 

.github/labeler.yml

@@ -13,17 +13,6 @@ frontend:
       - 'frontend/**'
       - 'package.json'
 
-skip-test:
-- all:
-  - changed-files:
-    - any-glob-to-all-files:
-      - 'CODEOWNERS'
-      - 'README.md'
-      - '.gitignore'
-      - 'dist/**'
-- head-branch:
-  - '^dependabot/'
-
 # svc:* labels track the mounted services, which share a name with SEP apps but
 # are different code. The prefix keeps app/tasks/ (the service) distinct from
 # app/sep/apps/tasks/ (the SEP app); same for inventory.

.github/workflows/ci.yml

@@ -38,9 +38,9 @@ jobs:
       - name: Require QA approval label
         if: >-
           !contains(github.event.pull_request.labels.*.name, 'qa passed') &&
-          !contains(github.event.pull_request.labels.*.name, 'skip-test')
+          !contains(github.event.pull_request.labels.*.name, 'qa not required')
         run: |
-          echo "::error::PR requires 'qa passed' or 'skip-test' label to merge."
+          echo "::error::PR requires 'qa passed' or 'qa not required' label to merge."
           exit 1
 
   # Detect changed file types to gate downstream jobs.
@@ -79,8 +79,13 @@ jobs:
               - 'sidecar/settings-env.sh'
               - 'sidecar/supervisord.conf'
               - 'sidecar/healthcheck.sh'
+              - 'sidecar/entrypoint.sh'
+              - 'sidecar/wait_for_schema.sh'
+              - 'sidecar/Containerfile.sidecar'
               - 'Makefile'
               - 'frontend/packages/api/specs/**'
+              - '.github/labeler.yml'
+              - '.github/workflows/ci.yml'
             precommit:
               - '.pre-commit-config.yaml'
               - '.github/workflows/ci.yml'
@@ -301,6 +306,7 @@ jobs:
             test -f /home/sep/app/supervisord.conf
             test -f /home/sep/app/healthcheck.sh
             test -x /home/sep/app/entrypoint.sh
+            test -x /home/sep/app/wait_for_schema.sh
             test -f /home/sep/app/settings-env.sh
             test -f /home/sep/app/settings.yaml
           '
@@ -330,6 +336,35 @@ jobs:
       - name: Smoke test - restricted image ships exactly the activated apps
         run: sidecar/verify_image_apps.sh "sep:HEAD" restricted
 
+      - name: Verify the purge layer is the last package-manager operation
+        run: python3 scripts/check_sidecar_purge.py --check-ordering
+
+      - name: Smoke test - purged packages are absent from the built image
+        run: |
+          pkgs="$(python3 scripts/check_sidecar_purge.py --print-packages | tr '\n' ' ')"
+          if [ -z "$pkgs" ]; then
+            echo "::error::the purge checker named no packages, so this check cannot run"
+            exit 1
+          fi
+          docker run --rm --entrypoint /bin/sh -e PURGED="$pkgs" "sep:HEAD" -c '
+            command -v dpkg-query > /dev/null 2>&1 || {
+              echo "::error::dpkg-query is absent, so package presence cannot be established"
+              exit 1
+            }
+            status=0
+            for p in $PURGED; do
+              if dpkg-query -s "$p" 2>/dev/null | grep -q "^Status: install ok installed"; then
+                echo "::error::$p is still installed in the built image"
+                status=1
+              fi
+            done
+            if command -v perl > /dev/null 2>&1; then
+              echo "::error::a perl binary resolves on PATH"
+              status=1
+            fi
+            exit $status
+          '
+
   pipeline-syntax:
     if: >-
       ${{

.github/workflows/labels.yaml

@@ -30,12 +30,13 @@ jobs:
           sparse-checkout: |
             .github
             scripts
-      - name: Compute blast-radius labels
+      - name: Sync computed PR labels
         if: github.event_name == 'pull_request_target'
         env:
           GITHUB_TOKEN: ${{ github.token }}
         run: |
-          python3 scripts/compute_blast_radius_labels.py \
+          python3 scripts/sync_pr_labels.py \
             --owner "${{ github.repository_owner }}" \
             --repo "${{ github.event.repository.name }}" \
-            --pr-number "${{ github.event.pull_request.number }}"
+            --pr-number "${{ github.event.pull_request.number }}" \
+            --head-ref "$GITHUB_HEAD_REF"

.github/workflows/python.yaml

@@ -80,7 +80,7 @@ jobs:
       - name: Install test dependencies
         run: poetry sync --no-root
       - name: Run tests
-        run: make test PYTEST_MARKERS='not postgres and not mysql' COV=${{ matrix.python == '3.11.9' && '1' || '0' }}
+        run: make test PYTEST_MARKERS='not postgres' COV=${{ matrix.python == '3.11.9' && '1' || '0' }}
       - name: Generate coverage comment data
         id: coverage_comment
         if: matrix.python == '3.11.9' && github.event_name == 'pull_request'
@@ -125,34 +125,6 @@ jobs:
       - name: Run PostgreSQL tests
         run: make test PYTEST_MARKERS='postgres' PYTEST_WORKERS=0 COV=0
 
-  test_mysql:
-    name: test (mysql)
-    runs-on: ubuntu-latest
-    services:
-      mysql:
-        image: mysql:8.0.46
-        env:
-          MYSQL_ROOT_PASSWORD: sep
-          MYSQL_DATABASE: sep_test
-        ports: ["3306:3306"]
-        options: >-
-          --health-cmd "mysqladmin ping -h 127.0.0.1 -psep --silent"
-          --health-interval 10s --health-timeout 5s --health-retries 10
-    env:
-      SEP_TEST_MYSQL_DSN: mysql+aiomysql://root:sep@127.0.0.1:3306/sep_test
-    steps:
-      - name: Checkout codebase
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
-      - name: Setup Python
-        uses: ./.github/actions/setup-python-job
-        with:
-          python-version: "3.11.9"
-          cache: poetry
-      - name: Install test dependencies
-        run: poetry sync --no-root --with mysql
-      - name: Run MySQL tests
-        run: make test PYTEST_MARKERS='mysql' PYTEST_WORKERS=0 COV=0
-
   build:
     strategy:
       matrix:

.pre-commit-config.yaml

@@ -53,6 +53,13 @@ repos:
         additional_dependencies: ["python-minifier==2.11.3"]
         files: ^app/sep/apps/(?!dipper/)(?:.*/)?(?:payload|[^/]*_payload)$
         pass_filenames: true
+      - id: check-sidecar-purge
+        name: Check side-car purge layer ordering
+        description: "Ensure no apt/dpkg instruction follows the purge layer"
+        entry: python3 scripts/check_sidecar_purge.py --check-ordering
+        language: system
+        files: ^(sidecar/Containerfile\.sidecar|scripts/check_sidecar_purge\.py)$
+        pass_filenames: false
       - id: gen-xtrabackup-payload-variants
         name: Check xtrabackup payload variants
         description: "Ensure the generated xtrabackup payload variants match the canonical payload"
@@ -95,6 +102,7 @@ repos:
     hooks:
       - id: check-ast
       - id: check-case-conflict
+      - id: check-added-large-files
       - id: check-json
       - id: check-merge-conflict
       - id: check-toml