percona-obs is the management tool for syncing the local root/ packaging tree to an OBS instance.
The following OBS service binaries must be installed on the machine running percona-obs.
They are invoked locally for packages that declare mode="manual" services (e.g. Go
dependency vendoring):
| Binary | Package (Debian/Ubuntu) | Package (RPM) |
|---|---|---|
obs_scm |
obs-service-obs-scm |
obs-service-obs_scm |
go_modules |
obs-service-go_modules |
obs-service-go_modules |
download_url |
obs-service-download_url |
obs-service-download_url |
Binaries are expected at /usr/lib/obs/service/<name>.
Services that are not installed are skipped with a warning. Only
mode="manual"service outputs (e.g.vendor.tar.gz) need to be produced locally — all other services run server-side on OBS.
Python 3.8+ is required. Create a virtualenv and install dependencies:
python3 -m venv venv
venv/bin/pip install -r requirements.txtCredentials are read from ~/.config/osc/oscrc. Run the osc first-run wizard to
create the file:
osc -A http://<your-obs-host>:8000 listFollow the prompts to enter your username and password. The file is created once and
reused by all subsequent osc and percona-obs invocations.
Every percona-obs command needs to know the OBS API URL (-A) and the root
project (-R). The recommended way to avoid typing these on every invocation is
to create a connection profile once and then use -P <name> to activate it.
A profile stores apiurl and rootprj in .profile/<name>.yaml (git-ignored).
Create one with the profile create command, passing -A and -R explicitly:
./percona-obs -A http://my-obs.local:8000 -R home:Admin profile create local
# + local (.profile/local.yaml)
# ✔ profile create: localRunning the same command again with different values overwrites the profile
(shown with ~ instead of +).
List all available profiles and their settings:
./percona-obs profile list
# local
# apiurl: http://my-obs.local:8000
# rootprj: home:AdminOnce a profile exists, activate it with -P:
./percona-obs -P local sync ppg:staging:17 etcd --dry-runExplicit -A/-R flags always override the profile values when both are given.
./percona-obs -P local sync push --dry-runRuns all services locally and shows what would be uploaded to OBS. Nothing is written.
./percona-obs -P local sync pushWalks the entire root/ tree, creates or updates all OBS projects and packages, and
uploads any changed obs/ files as a single revision per package.
./percona-obs -P local sync push common:deps:runtime percona-telemetry-agentPG extensions live under a subproject (ppg/staging/17/). Pass the subproject and
package name separately. (There is also a ppg/devel/17/ tier for dev-branch builds —
see root/README.md — not covered further here.)
./percona-obs -P local sync push ppg:staging:17 percona-pg-telemetry./percona-obs -P local sync push ppg:staging:17--branch-from <profile> speeds up syncing a new environment by reusing already-built
binaries from an existing OBS project instead of re-uploading sources and waiting for
every package to build again from scratch.
For each package that is unchanged since the branch profile's last sync, percona-obs
uploads only a small _aggregate file. OBS then pulls the pre-built binaries directly
from the branch project's repository — no source fetch, no compilation, no wait.
Only packages that have actually changed are uploaded with their full source files
and built fresh.
Suppose you maintain a stable production profile prod (home:Admin:percona) and want
to spin up a test environment (home:Admin:percona-test) that tracks a feature branch.
Most packages are identical; only one or two have been modified.
Step 1 — Create a profile for the new environment:
./percona-obs -A http://my-obs.local:8000 -R home:Admin:percona-test profile create testStep 2 — Sync the test environment, branching from prod:
./percona-obs -P test sync push --branch-from prodFor every unchanged package, percona-obs uploads an _aggregate pointing at
home:Admin:percona — the prod project — and OBS serves the binaries from there
instantly. Modified packages get their sources uploaded and build normally.
Output example:
+ project meta home:Admin:percona-test
+ project meta home:Admin:percona-test:ppg
+ project meta home:Admin:percona-test:ppg:staging:17
= files home:Admin:percona-test:ppg:staging:17/percona-postgresql17
@ home:Admin:percona-test:ppg:staging:17/percona-postgresql17 → home:Admin:percona:ppg:staging:17/percona-postgresql17
~ 4 files home:Admin:percona-test:ppg:staging:17/percona-pg-telemetry ← changed, uploaded
✔ sync successful
After branching, when you want a package (or all packages) to build from local sources instead
of pulling pre-built binaries from the branch project, run sync promote:
./percona-obs -P test sync promote # promote all branch packages
./percona-obs -P test sync promote ppg:staging:17 # promote all packages under a subproject
./percona-obs -P test sync promote ppg:staging:17 etcd # promote a single packageFor each package whose latest OBS revision was created by a --branch-from sync,
percona-obs replaces the _aggregate with the full local obs/ source files
(running any mode="manual" services as needed). Packages that already hold
real sources are skipped with =.
Preview what would be promoted without writing to OBS:
./percona-obs -P test sync promote --dry-runWhen branching, packages that depend on a changed package must also be rebuilt from
source — otherwise they might link against stale binaries from the branch project.
percona-obs handles this automatically.
After the initial changed/unchanged classification, percona-obs queries OBS
_builddepinfo for the branch project to determine which packages build-depend on
which others. It then applies bidirectional dep propagation:
- If package A is promoted (uploaded with sources), every package that depends on A (directly or transitively) is also promoted.
- Conversely, every package that A depends on is also promoted, so A builds against fresh locally-controlled binaries rather than the branch copy.
This fixed-point iteration continues until no more promotions are triggered. The result is a minimal set of packages that must be built from source, with all others remaining as lightweight aggregates.
Example: if golang-1.25 has changed locally, percona-telemetry-agent and
etcd (which both build-depend on it) are automatically promoted even if their own
source files are identical to what was last synced to the branch project.
Use build dependency to inspect these relationships before syncing:
./percona-obs -P local build dependencyIn addition to per-package file comparisons, percona-obs detects when a project's
configuration has changed and promotes all packages in that project for rebuild —
even if their source files are identical to the branch project. This handles scenarios
like adding a new architecture to root/project.yaml, which is inherited by every
subproject that does not define its own repositories: list.
Two cases are handled:
- Re-sync of an existing PR project: the local desired config is compared against the PR project's current meta on OBS. If they differ (e.g. a new arch was added), all packages in that project are promoted.
- New PR project (first sync of a new PR): since the PR project does not exist on OBS yet, the local desired config is compared against the corresponding production project instead. If the production project exists and its config differs from local, the affected PR projects are created and all their packages promoted.
This ensures that an architecture or repository change introduced in a PR is always built and tested for the packages that belong to affected projects, even when those packages have no direct source file changes.
percona-obs uses a two-level decision for each package:
-
Fast path — reads the last OBS revision comment on the branch project. If it contains a clean
sync: <branch>@<sha> (...)message,git logchecks whether any local commits touch that package since that SHA. No commits → aggregate. Commits → upload. -
Content check fallback — used when the revision message is absent, in a different format, or was written from an unpushed branch. Compares MD5s of every local
obs/file against what OBS holds, and also verifies that the upstream source commit hash in the.obsinfofile matches the current remote HEAD viagit ls-remote. Both must match → aggregate.
A full-tree sync push normally runs services and compares files for every package,
even when almost nothing changed. The features below cut that down to a handful of
API calls on a quiet tree, and keep CI polite towards the OBS traffic limiter.
venv/bin/python -m percona_obs -P main sync push --skip-unchanged --report-json /tmp/sync-report.jsonSkips packages whose latest OBS revision comment records a clean
sync: <branch>@<sha> (...) from a git SHA with no changes since. "No changes since"
is checked locally against three inputs: commits touching the package directory,
uncommitted edits in it, and the rendered values of the %!{NAME} macros the package
references (compared between the SHA and the working tree — moving a macro between
ancestor macros.yaml files or bumping a macro the package never uses is not a change). A skipped
package runs no services and uploads nothing — it costs one API call (the
revision-history fetch), or zero when the sync-state manifest is warm (see below).
Skipped packages are printed as = skip <project>/<package> (unchanged).
Plain pushes only: combining --skip-unchanged with --branch-from is rejected
(the branch decision already has its own unchanged detection).
Caveats:
- Packages tracking a moving upstream ref are never skipped. If the package's
_servicehas an upstreamobs_scmwhoserevisionis missing,HEAD, or names a remote branch (the devel-tier pattern of trackingmain), the tarball must be re-resolved from upstream on every sync — upstream-only commits would otherwise never reach OBS. Branch-vs-tag classification runsgit ls-remoteonce per(url, revision)pair and is cached persistently in.cache/sync_state/scm_ref_types.json; full 40-hex SHA revisions are always immutable and need no lookup. Classification errors conservatively count as "branch" (no skip) and are not cached. - Any doubt falls back to the normal promote path. No revision comment, a
non-
sync:comment, a dirty (local changes on ...) sync, or changes since the recorded SHA all route the package through the regular upload, whose per-file MD5 comparison is authoritative — an unchanged upload is a no-op anyway.--forcedisables skipping entirely. - Out-of-band OBS edits are not detected. Changes made directly on OBS that
create no source revision (e.g. a manual
_metaedit) are invisible to the skip decision — the same trust model as--branch-fromaggregates.
--skip-unchanged also maintains a local manifest .cache/sync_state/<hash>.json
mapping project/package → last-synced git SHA, keyed by (apiurl, rootprj) so
different profiles and instances never share entries. When the manifest entry for a
package is present and the same local checks pass — package-directory tree diff
against the recorded SHA, uncommitted edits, referenced macro values — the package is
skipped with zero API calls, without even fetching the OBS revision history.
Entries are recorded only for uploads made from a pushed, clean HEAD: an unpushed
SHA can be reset away, and dirty inputs would bake in content no SHA represents. In
CI the manifest is persisted between runs by actions/cache (it lives in the same
cache entry as the service outputs). A missing, corrupt, or evicted manifest simply
falls back to the OBS revision-comment check above.
Writes a JSON sync report:
{
"rebuild_projects": ["home:Admin:percona:ppg:staging:17", "..."],
"promoted": ["home:Admin:percona:ppg:staging:17/etcd", "..."],
"skipped": 42,
"head_sha": "<full git HEAD sha>"
}rebuild_projects lists every project whose builds this run may have triggered —
committed file uploads plus meta/prjconf creates or updates. The CI poll script
(.github/scripts/poll_obs_builds.py) consumes the report via the OBS_SYNC_REPORT
environment variable and monitors only those projects instead of the full tree,
finishing with one full-tree sweep to adopt cross-project rebuild cascades.
head_sha anchors the report to the checkout that produced it; the poll fails
open — full-tree polling as before — on a missing, corrupt, or stale (head_sha
mismatch) report. The report is also written in --dry-run (it reflects the
computed would-be changes).
All osc HTTP requests made by percona-obs are paced client-side: a shared
minimum interval between request starts reshapes thread-pool bursts into a steady
stream (default 8 requests/s; tune with PERCONA_OBS_MAX_RPS, 0 disables pacing).
Requests failing with HTTP 429 or 503 are retried for every method, and 502/504
additionally for GETs, honoring Retry-After when present (otherwise exponential
backoff), up to 5 attempts. On any throttling response the shared pacing slot is
pushed forward so all threads back off together, not just the throttled one.
Note that osc's internal urllib3 layer performs its own connection-level retries
(500/502/503) beneath this wrapper.
The CI build poll (poll_obs_builds.py) starts at OBS_POLL_INTERVAL seconds
between polls (default 30) and ramps the interval ×1.5 per cycle while build states
are unchanged, up to OBS_POLL_MAX_INTERVAL (default 300); any state change resets
it to the base.
The sync-main.yml workflow is split into two jobs: a sync job serialized on its
own concurrency group (never cancelled mid-upload, so back-to-back pushes queue up
short sync runs) and a poll job in a cancel-superseding group (newest poll wins —
a newer push's poll replaces an older run's). The version lists and release tags are
diffed from the head SHA of the last successful run, so the surviving poll covers
any superseded runs' ranges too.
./percona-obs -P local build trigger # all packages
./percona-obs -P local build trigger ppg:staging:17 # all packages under a subproject
./percona-obs -P local build trigger ppg:staging:17 etcd # single packageSends an OBS service run request (runservice) for each targeted package, causing
OBS to re-fetch sources and queue a new build.
./percona-obs -P local build statusPrints a color-coded tree of live build statuses fetched from OBS. Succeeded packages display the built version next to the status:
home:Admin:percona
├── common
│ ├── deps
│ │ ├── build
│ │ │ ├── golang-1.25
│ │ │ │ ├── RockyLinux_9 ✔ succeeded
│ │ │ │ ├── Debian_13 ✔ succeeded
│ │ │ │ └── xUbuntu_24.04 ✔ succeeded
│ │ │ └── obs-service-tar_scm
│ │ │ ├── RockyLinux_9 ✔ succeeded
│ │ │ ├── Debian_13 ✗ failed
│ │ │ └── xUbuntu_24.04 ✔ succeeded
│ │ └── runtime
│ │ └── percona-telemetry-agent
│ │ ├── RockyLinux_9 ✔ succeeded 3.5.26-6.1
│ │ ├── Debian_13 ✔ succeeded 3.5.26-6.1
│ │ └── xUbuntu_24.04 ✔ succeeded 3.5.26-6.1
└── ppg
└── staging
└── 17
├── etcd
│ ├── RockyLinux_9 ✔ succeeded 3.5.26-6.1
│ ├── Debian_13 ✔ succeeded 3.5.26-6.1
│ └── xUbuntu_24.04 ✔ succeeded 3.5.26-6.1
└── percona-pg-telemetry:17
├── RockyLinux_9 ✔ succeeded 1.0.0-1.1
├── Debian_13 ✔ succeeded 1.0.0-1.1
└── xUbuntu_24.04 ◌ scheduled
| Symbol | Color | Meaning |
|---|---|---|
✔ |
green | succeeded |
✗ |
red | failed / unresolvable / broken |
● |
cyan | building / dispatching |
◌ |
yellow | scheduled / blocked |
– |
dim | excluded / disabled |
Scope can be narrowed the same way as other commands:
./percona-obs -P local build status ppg:staging:17 # subproject only (tree rooted there)
./percona-obs -P local build status ppg:staging:17 etcd # single package
./percona-obs -P local build status --repo RockyLinux_9 # all packages, one distro onlySet NO_COLOR=1 to disable color output.
./percona-obs -P local build dependencyQueries OBS _builddepinfo for all packages and prints a dependency tree grouped by
root packages — packages that no other local package depends on. Each root package
is shown with its direct and transitive build dependencies indented beneath it.
Packages in the tree are annotated with the OBS project they belong to.
etcd (home:Admin:percona:ppg:staging:17)
└── golang-1.25 (home:Admin:percona:common:deps:build)
percona-pg-telemetry (home:Admin:percona:ppg:staging:17)
├── percona-postgresql-common (home:Admin:percona:ppg:staging:17)
└── percona-postgresql17 (home:Admin:percona:ppg:staging:17)
percona-telemetry-agent (home:Admin:percona:common:deps:runtime)
└── golang-1.25 (home:Admin:percona:common:deps:build)
obs-service-recompress (home:Admin:percona:common:deps:build)
obs-service-set_version (home:Admin:percona:common:deps:build)
obs-service-tar_scm (home:Admin:percona:common:deps:build)
Packages with no local build dependencies and that nothing else depends on are listed at the bottom as isolated packages. Scope can be narrowed to a subproject:
./percona-obs -P local build dependency ppg:staging:17qa run reads a project's qa: block from project.yaml, expands its matrix
into one Jenkins job per combination, triggers each via Jenkins'
buildWithParameters REST endpoint, and (with --wait) polls every triggered
build until it reaches a terminal state.
The same machinery powers the QA jobs in .github/workflows/obs-pr-check.yml:
the detect-qa-matrix job discovers the QA-enabled subprojects of the PR's
OBS root project, calls qa show <project> --json for each, concatenates the
matrices into the GitHub Actions matrix, and the qa job runs
qa run --wait --report-json ... --filter ... once per combo with one commit
status posted per combo on the PR head.
# project.yaml
qa:
pipeline: <jenkins-job-name> # required
parameters: # required
SCALAR_PARAM: value # passed as-is
LIST_PARAM: # if NOT in `matrix:` → joined with \n
- a # (Jenkins multi-line text param)
- b # if in `matrix:` → expanded combinatorially
matrix: # optional, list[str]
- LIST_PARAM${VAR} tokens in any value are substituted from the active profile's env:
section, plus auto-injected OBS_ROOTPRJ and OBS_CONTAINER_REGISTRY_ROOTPRJ (the root
project name with : replaced by /, useful for registry URLs).
| Command | Purpose |
|---|---|
qa show <project> |
Print the resolved matrix (humans). --json emits one entry per combo with project, pipeline, label, axis_filters, status_context, params — used by CI to drive a GitHub matrix. Empty [] when the project has no qa: block. |
qa run <project> |
Trigger Jenkins for every matrix combo. Fire-and-forget by default; pass --wait to block until terminal results arrive. --filter AXIS=val[,val…] narrows the matrix; --param NAME=VAL overrides a parameter at runtime; --dry-run prints the POST bodies without calling Jenkins; --report-json PATH writes the per-combo result table for CI. |
qa status --run-id <id> |
Re-poll non-terminal combos of a previous run and print the current state. |
qa retry --run-id <id> |
Re-trigger only the combos whose latest attempt is non-SUCCESS. Re-uses the recorded params so retries are reproducible. By default skips ABORTED combos; pass --include-aborted to retry them too. |
qa list |
Tabulate recent runs (run-id, project, pipeline, summary). |
State files for qa run / qa retry live at .percona-obs/qa/<run-id>.json
(gitignored). Each combo records every trigger attempt, so the file accumulates
history across qa retry invocations.
qa run resolves Jenkins URL + user from (in order) the env vars
JENKINS_URL, JENKINS_USER, then the optional jenkins: section of
.profile/<name>.yaml:
# .profile/dev.yaml
jenkins:
url: https://jenkins.example.com
user: ricardo
# token never stored in the profileJENKINS_API_TOKEN is read only from the environment, never the profile.
Each matrix combo runs as its own GitHub Actions job, so the PR's "Checks" tab "Re-run failed jobs" button re-runs only the combos that failed. The detect job's matrix output is reused across attempts, so the set of combos stays identical between the original run and the re-run.
For local invocations, qa retry --run-id <id> does the equivalent — re-triggers
only the failed combos of the named run, appending a new attempt to each combo's
history in .percona-obs/qa/<run-id>.json.
project install prints the shell commands needed to configure the OBS-hosted package
repositories on a target machine, grouped by distribution.
This command contacts the OBS instance to resolve the download URL, so it requires a profile (or explicit
-A/-R).
./percona-obs -P local project install./percona-obs -P local project install ppg:staging:17./percona-obs -P local project install --repo RockyLinux_9Example output for a Rocky Linux 9 repository:
────────────────────────────────────────────────────────────────────────
RockyLinux_9
# home:Admin:percona:ppg:staging:17
rpm --import http://my-obs.local/home:/Admin:/percona:/ppg:/staging:/17/RockyLinux_9/repodata/repomd.xml.key
tee /etc/yum.repos.d/home_Admin_percona_ppg_staging_17.repo << 'EOF'
[home:Admin:percona:ppg:staging:17]
name=home:Admin:percona:ppg:staging:17 - RockyLinux_9
baseurl=http://my-obs.local/home:/Admin:/percona:/ppg:/staging:/17/RockyLinux_9/
enabled=1
gpgcheck=0
EOF
For Debian-based distributions, instructions use echo … | tee + curl … | gpg --dearmor | tee
followed by apt update. For openSUSE/SLE repositories, zypper addrepo +
zypper --gpg-auto-import-keys refresh is emitted instead.
Projects that set install: false in their project.yaml, or that contain no packages,
are silently excluded from the output.
A release captures a point-in-time snapshot of a source OBS project by copying
its built binaries into a dedicated release project. The source project is ppg:staging:<V>
— the full, tag-built package set (see root/README.md for the
devel/staging/releases tier layout). Release IDs include the PG minor version and a
counter (e.g. 17.9-1, 17.9-2, 17.10-1). A single OBS release project
(ppg:releases:17) covers all minor versions for a PG major.
The process is PR-based:
project release— auto-derives the release ID from OBS, regenerates the fullreleases/<name>/mirror tree from staging's current config, updatesrelease.yamlandCHANGELOG.md, and commits locally. Pushing the branch and opening the review PR remain manual steps.- PR review —
obs-pr-check.ymldetects the release-only PR and runssync release <releases-project> --dry-run, posting the genuine outcome (mirrors complete, changelog section present, tag not already used, source reachable, staging green) as a PR comment. Check it andCHANGELOG.md, then merge. obs-pr-cleanup.yml— on close of a merged release PR, tags the merge commit (e.g.ppg/17.9-1) withGITHUB_TOKENand dispatchesobs-release.ymlviagh workflow runwith that tag as input.obs-release.yml— its sole trigger isworkflow_dispatch; it runssync releasein CI, then creates a GitHub release with the changelog contents.
Constraint: the tag+dispatch automation in step 3 only fires for a PR whose
root/changes are entirely underroot/*/releases/(obs-pr-cleanup.yml's release-only detection). Do not mix in unrelatedroot/edits — including the legacyPPG_RELEASEcounter bump in a per-major<staging>/<V>/macros.yaml, which would break release-only detection if it were ever reintroduced there. This is currently dormant in practice becausePPG_RELEASElives in the sharedroot/ppg/staging/macros.yaml, whichproject releasedoes not touch or bump.
# Fully automatic — both release-name and release-id derived from OBS
./percona-obs -P local project release ppg:staging:17
# Override release-name only
./percona-obs -P local project release ppg:staging:17 --release-name 17
# Override both
./percona-obs -P local project release ppg:staging:17 --release-name 17 --release-id 17.9-1source-project must be a staging-tier project — any other tier is rejected with an
error. project release <source-project> does the following:
- Auto-derives
release-id(if not given with--release-id):- Queries OBS for the built version of
percona-postgresql<major>in the source project. - Extracts
MAJOR.MINOR(e.g.17.9from17.9.0-1.1). - Counts existing entries in
release.yamlwhose tag matches/<MAJOR.MINOR>-*and appends the next counter (e.g.17.9-1, or17.9-2if17.9-1already exists).
- Queries OBS for the built version of
- Fetches the source project's repository topology from OBS.
- Regenerates the full mirror tree, on every release, not just the first: the
top-level
project.yamlplus one nested mirror directory per staging subproject (containers/,extras/,extras/containers/,tarballs/). All mirrors getbuild: false,publish:flags are carried over, and subproject paths — including intra-project sibling paths — are rewritten to the release namespace. A mirror directory whose staging source no longer exists is deleted. - Builds
CHANGELOG.md— compares built package versions from the top-level staging project,extras, andtarballsagainst the release OBS project (for updates) and generates entries with upstream URLs. Container images are listed per flavor (ubi8andubi9as distinct entries); the previous-release image diff is queried from whichever project actually holds the previous release's images, producing realprev → newdiffs instead of full package dumps (a full dump remains only for a true first release). - Appends the new tag to
release.yaml'sreleases:list and prepends the new section toCHANGELOG.md. - Commits the generated files locally with
git commit -s. Pushing the branch and opening the PR remain manual steps —project releasedoes not do either.
The git tag (e.g. ppg/17.9-1) is created automatically by obs-pr-cleanup.yml, on the
merge commit, once the release PR is merged. That tag is then passed as input when
obs-pr-cleanup.yml dispatches obs-release.yml — pushing a tag does not trigger
anything by itself.
repository: ${PERCONA_OBS_PACKAGING_REPO}
project: ppg:staging:17
releases:
- ppg/17.9-1 # first release
- ppg/17.9-2 # update release
- ppg/17.10-1 # minor version bumpsync release always uses the last entry as the current tag. Old entries are
kept for history. The backward-compat revision: field (plain string) is still
accepted and treated as a single-element list.
Each release creates a section following Keep a Changelog. New sections are prepended above previous ones. Package entries are auto-populated:
## [17.9-1] - 2025-03-01
### Added
- percona-pg_tde: updated to upstream version 2.0.0 (https://github.com/percona/pg_tde/releases/tag/2.0.0)
- percona-postgresql17: updated to upstream version 17.9.0 (https://www.postgresql.org/docs/release/17.9/)
### Changed
### FixedFor update releases, only packages whose version changed appear (under ### Changed):
## [17.9-2] - 2025-04-10
### Added
### Changed
- percona-postgresql17: updated to upstream version 17.9.1 (https://www.postgresql.org/docs/release/17.9/)
### FixedUpstream URLs are derived from the package's obs/_service upstream obs_scm entry:
- GitHub (
https://github.com/…): appends/releases/tag/<revision>. - PostgreSQL (
git.postgresql.org): maps tohttps://www.postgresql.org/docs/release/<MAJOR.MINOR>/. - Other: uses the bare
urlvalue.
Release PRs are review-only — obs-pr-check.yml detects them and, instead of running an
OBS build, runs a real sync release <releases-project> --dry-run and posts the genuine
outcome as a PR comment (no label is required for this check). On merge:
sync-main.ymlrunssync pushnormally (releases/ directories are excluded from traversal). It no longer creates any tag.obs-pr-cleanup.yml, on close of the merged release PR, tagsmerge_commit_shawithGITHUB_TOKEN(e.g.ppg/17.9-1) and dispatchesobs-release.ymlviagh workflow run obs-release.yml --ref main -f tag=<tag>. A tag-creation or dispatch API error fails the job; only a genuine "tag already exists" response is tolerated.obs-release.yml's sole trigger isworkflow_dispatch(thepush: tagstrigger has been removed, so pushing a tag by hand is inert). Given the dispatchedtaginput, it derives the OBS release project name (e.g.ppg/17.9-1→ppg:releases:17) and runssync release.- After
sync releasefinishes,obs-release.ymlupdates version list docs and creates a GitHub release with theCHANGELOG.mdsection for that release ID.
Manual recovery (e.g. obs-pr-cleanup.yml tagged but the dispatch failed):
gh workflow run obs-release.yml -f tag=ppg/17.9-1Pending-cancellation race: obs-release.yml shares the sync-main-sync concurrency
group (cancel-in-progress: false) with sync-main.yml's sync job, and GitHub keeps at
most one pending run per group. If a root/** push queues another sync-main sync job
while the dispatched obs-release.yml run is still pending behind an in-progress one,
the pending obs-release.yml run is cancelled rather than requeued — the symptom is
a cancelled "OBS Release" run with no release having happened. Recovery is the same
manual command above, re-run once the concurrency group is free.
For local testing or recovery from a failed CI run:
./percona-obs -P local sync release ppg:releases:17 --skip-tag-checkFlags:
| Flag | Default | Effect |
|---|---|---|
--dry-run |
off | Read-only: checks that every staging subproject has a mirror, the CHANGELOG.md section for the release ID exists, the tag does not already exist, the source OBS project is reachable, and every staging package currently reports succeeded. Does not freeze, release, or write anything. |
--force |
off | Skips the divergence (tag) check entirely. |
--skip-tag-check |
off | Skips only the git diff <tag>..HEAD check — use when the tag doesn't exist locally yet. |
--no-freeze |
off | Skips the drain/freeze/restore sequence (§ below) — for recovery scenarios where staging is already known idle. |
--freeze-timeout |
3600 (seconds) |
Bound on the drain step; the release fails rather than waiting forever for a stuck package. |
--verify-timeout |
600 (seconds, 0 disables) |
Bound on the post-release poll that confirms each released repo actually holds binaries. |
sync release <release-project> reads release.yaml and performs:
- Divergence check (skipped with
--force):- Checks that no files under the source project have changed since the release tag
(
git diff <tag>..HEAD). Pass--skip-tag-checkwhen the tag doesn't exist locally. - Runs
sync push --dry-runto confirm OBS is up-to-date (first release only).
- Checks that no files under the source project have changed since the release tag
(
- Mirror check — a staging subproject with no corresponding local mirror directory is a hard error (never silently skipped).
- Applies the release project config — the top-level release project meta is re-applied on the update path too, not just on first creation, and this happens before the freeze sequence below. Release targets are added only for repos present on both sides.
- Freeze sequence (skipped with
--no-freeze), applying to every release, first or update:- Drain — poll
_resultfor the staging project and all subprojects until no package isbuilding/scheduled/dispatching/blocked/finished/signing, bounded by--freeze-timeout. - Assert green — every package is
succeeded(or legitimatelyexcluded/disabled); anyfailed/unresolvable/brokenaborts the release. - Freeze — snapshot each project's meta, then disable builds on staging and every subproject.
- Release —
osc releasefor the top-level project and every subproject. - Verify — still inside the freeze window, polls (bounded by
--verify-timeout) until each released repo holds binaries. - Restore — in a
finally(so it still runs even if verify times out or raises), re-apply each project's exact snapshotted meta last (never a blanket enable, since subprojects carry per-repo flags — e.g. tarballs'publish:flags — that must survive the round trip).
- Drain — poll
- Release-tier OBS projects with no local mirror are reported loudly as orphans;
deletion is never automatic — run
sync deletemanually.
Skip all divergence checks:
./percona-obs -P local sync release ppg:releases:17 --forceroot/
└── ppg/
└── releases/
└── 17/ # one directory per PG major version
├── release.yaml # releases: list of tags
├── project.yaml # ppg:releases:17 (builds disabled)
├── CHANGELOG.md # keep-a-changelog, all releases
├── containers/ # nested, mirrors staging's containers/ subproject
│ └── project.yaml # builds disabled, paths → release project
├── extras/
│ ├── project.yaml
│ └── containers/
│ └── project.yaml
└── tarballs/
└── project.yaml
Subproject mirrors are nested directories, never colon-named — containers:ubi9/
describes only the pre-restructure layout and no longer reflects reality.
project release regenerates the entire mirror tree from staging's current
config on every release, not just the first: mirrors for subprojects that still
exist in staging are refreshed in place, and mirror directories whose staging
source no longer exists are deleted.
The releases/ directory is excluded from sync push traversal — release
directories (identified by release.yaml) are never synced to OBS as source
projects.
sync release-pr still exists, but no workflow invokes it anymore — it is a
manual/recovery-only command. Previously obs-pr-cleanup.yml ran it before deleting
the PR project; that step has been removed from the workflow.
sync release-pr copies the PR's built binaries to the corresponding production OBS
projects via the <releasetarget> entries that sync push --branch-from added to
every active PR project's repository configuration.
Before copying binaries, sync release-pr automatically:
- Reads the
<releasetarget>entries from each PR project's live OBS meta to discover the production project counterparts — no naming convention is assumed. - Applies any pending project configuration changes (new repositories, added architectures) to those production projects. This ensures that a config change introduced by the PR takes effect in production at the same time as the new binaries.
- Runs
osc release <pr-project> --no-delayfor each PR project that exists on OBS.
To run manually for recovery (replace <pr-rootprj> with e.g. home:Admin:percona:PR:pr-42):
./percona-obs -A <apiurl> -R <pr-rootprj> sync release-pr./percona-obs -P local sync delete --dry-run
./percona-obs -P local sync delete ppg:staging:17 --dry-run./percona-obs -P local sync delete --yes --recursiveDeletes the root project and all sub-projects (deepest first). Prompts for confirmation
unless --yes is given. Use --recursive to delete projects that still contain packages.
./percona-obs -P local sync delete ppg:staging:17 --yes --recursive./percona-obs -P local sync delete ppg:staging:17 etcd --yes- Copy an existing standalone package as a template:
cp -r root/common/deps/runtime/percona-telemetry-agent root/common/deps/runtime/my-new-service
- Edit
obs/_service— update the upstream source URL and any service parameters. - Edit
rpm/*.specanddebian/control,debian/changelogwith the new package name and version. - Optionally create
package.yamlwith a title and description. - Sync to OBS:
./percona-obs -P local sync push common:deps:runtime my-new-service
- Copy an existing PG extension as a template:
cp -r root/ppg/staging/17/percona-pg-telemetry root/ppg/staging/17/my-pg-extension
- Replace all
percona-pg-telemetryreferences withmy-pg-extensionthroughout the copied files. - Update
obs/_serviceto point to the new package's upstream repo. - Update
obs/_multibuildwith the PG major versions to build for. - Update
rpm/*.specanddebian/control— keep@BUILD_FLAVOR@placeholders. - Sync to OBS:
./percona-obs -P local sync push ppg:staging:17 my-pg-extension