Improving PR Deploy Strategy

[mekarpeles]

Feature Request

Better strategy for implementing #11259

Problem / Opportunity

Right now, PR Deploys must be done within the VPN which means Leads cannot deploy PRs to testing.

Furthermore, the current model accepts risk in that, redeploying PRs en masse could add unreviewed commits to the testing server.

This issue advocates for two things:

1. A {instance}.openlibrary.org endpoint (for ol-dev1 testing instances only) that allows those with permission (e.g. usergroup) to PR deploy to that instance
2. A change in model such that, by default PRs persist (are additive) across PR deploys, but are pinned to a specific commit hash

In an ideal world, https://testing.openlibrary.org/status would have a management dashboard that lets you bump the commit for a specific PR, add a PR, or remove a PR and then apply the modified table via jenkins.

Proposal

Breakdown

No response

[mekarpeles]

Proposal: Replace /status with a Testing Environment Management Dashboard

Problem Summary

The current testing deploy workflow has three issues:

1. High friction — deploying PRs requires a Chrome bookmarklet with a Jenkins token, triggered from GitHub. There's no self-service UI.
2. No commit pinning — the unit of deploy is a PR (branch ref), not a specific commit. If someone pushes to a branch after it's deployed to testing, the testing environment silently picks up unreviewed changes on next rebuild.
3. Poor visibility — the only way to see what's on testing is to visit /status, which shows a static text dump with no management capability.

Proposed Solution

Turn https://testing.openlibrary.org/status (when running in dev/testing mode) into an interactive management dashboard. This page should only be enabled when a config flag like testing_mode = true is set — it must not be accessible in production.

See:

• https://github.com/internetarchive/openlibrary/blob/master/openlibrary/plugins/openlibrary/status.py

• openlibrary/scripts/make-integration-branch.sh

  Line 3 in 0b1e4f8

  # Util script to merge multiple branches as listed in a file into a new branch

---

Data Model

The dashboard needs a persistent state file (e.g. JSON on disk or in a lightweight DB) tracking the current PR set:

testing_prs:
  - pr_number: 12235
    pinned_commit: abc1234      # the exact SHA deployed
    active: true                # whether it's included in the current merge
    added_at: 2025-03-28T...
    added_by: "mekarpeles"

Key invariant: the deploy always builds from pinned commits, never from branch HEADs, unless the user explicitly clicks "pull to latest."

---

Dashboard UI: Table View

The main view is a table of all PRs currently in the testing set:

☐	PR	Title	Pinned Commit	Branch HEAD	Drift	Active	Added
☐	#12235	chore(deps): update vite to v8	abc1234	def5678	3 commits behind	✅	Mar 28
☐	#12188	Add missing translation	111aaaa	111aaaa	up to date	✅	Mar 27
☐	#11895	remove web.py partials code	222bbbb	333cccc	1 commit behind	⏸️	Mar 25

Column definitions:

• PR — link to the GitHub PR
• Title — PR title, fetched from GitHub API
• Pinned Commit — the exact commit SHA currently deployed (short hash, linking to the commit on GitHub)
• Branch HEAD — the current HEAD of the PR's branch, fetched live from GitHub API
• Drift — number of commits between pinned commit and branch HEAD. Visually flag when drift > 0 (yellow) or drift > 5 (red). If the PR branch has been force-pushed such that the pinned commit is no longer in the branch history, show a ⚠️ warning.
• Active — whether this PR is included in the merged testing build. Inactive PRs stay in the list but are excluded from the deploy merge.
• Added — when the PR was added to the testing set

---

Actions

Row-level (via checkboxes + action bar, or per-row buttons):

1. Pull to latest — update the pinned commit to the current branch HEAD for selected PRs. This is the only way a PR's deployed commit changes. Triggers a rebuild.
2. Deactivate / Reactivate — toggle the active flag. Deactivated PRs are excluded from the merge but remain in the list. Useful for isolating conflicts without losing track of what you were testing. Triggers a rebuild.
3. Remove — remove selected PRs from the testing set entirely. Triggers a rebuild.

Global:

4. Add PR — input field that accepts a PR number (or GitHub PR URL). Resolves the PR via GitHub API, pins it at the branch's current HEAD commit, adds it to the list as active, and triggers a rebuild.
5. Rebuild — force a rebuild from the current state (all active PRs at their pinned commits). Useful if a rebuild failed or was interrupted.

---

Auto-cleanup: Merged PRs

On each page load (or via a periodic background check), query the GitHub API for the merge status of each PR in the list. If a PR has been merged to master:

• Visually flag it (e.g. strikethrough or a "merged" badge)
• Auto-remove it from the active set on next rebuild, or prompt the user

This prevents stale merged PRs from accumulating.

---

Build Mechanics

When a rebuild is triggered, the backend should:

1. Start from the base branch (master or whatever the target is)
2. For each active PR in the set, cherry-pick or merge the pinned commit (not the branch HEAD)
3. Record the resulting merge SHA (like the current dev-merged created (SHA: dbeccdcc2))
4. Deploy that SHA to testing
5. Update the dashboard status to show build result (success/failure, timestamp, merge SHA)

If a merge conflict occurs during the build, the dashboard should report which PR(s) conflicted and suggest deactivating one of them.

---

Implementation Notes

• GitHub API access: the backend needs a GitHub token with read access to the internetarchive/openlibrary repo (PR metadata, branch HEADs, commit history, merge status). This likely already exists for the Jenkins integration.
• Auth: the dashboard should require some form of authentication — even basic shared-secret or IP allowlisting — since it triggers deploys. If Open Library's testing env already has an admin auth mechanism, reuse that.
• State persistence: a JSON file on disk is fine for an MVP. The state is small (a list of PRs with pinned commits) and doesn't need a database.
• Rebuild trigger: this replaces the Jenkins bookmarklet. The dashboard's backend should either call the same Jenkins job or run the merge/deploy script directly, depending on the current infrastructure.
• No production exposure: gate the entire dashboard behind a config check. If testing_mode is not enabled, /status should return a 404 or a plain text "not available."

---

Migration Path

1. Build the dashboard UI and state management on /status
2. Wire the "Add PR" and "Rebuild" actions to the existing Jenkins-based build pipeline
3. Once stable, deprecate the bookmarklet
4. Optionally, add a GitHub webhook to auto-flag merged PRs in real time instead of polling

---

Out of Scope (for now)

• Branch protection / approval workflows for testing deploys
• Automatic conflict resolution
• Multiple testing environments
• Notifications (Slack/email) on build status — though this would be a nice addition later

---

Features

• Commit pinning is the key architectural change. The current system treats PRs as mutable references — every rebuild could silently pick up new, unreviewed pushes. The proposed model makes the pinned commit the source of truth, and "pull to latest" is an explicit, deliberate action. This is the single most important thing to get right.
• The "deactivate" feature is critical for conflict management. Right now if two PRs conflict, you'd have to fully remove one and re-add it later. Deactivation lets you keep your working set intact while temporarily excluding a PR from the merge.
• Auto-cleanup of merged PRs prevents the list from becoming stale — which is what currently happens when people forget what's deployed.

---

Bookmarklet

javascript:(async function() {     function usage() {         alert(` Unable to find a PR to deploy USAGE: Either (1) Click this bookmarklet from a PR page, or (2) Go to https://github.com/internetarchive/openlibrary/pulls, check the PRs you want to deploy, and then click this bookmarklet `);     }      let prs = [];     try {         prs = await Promise.all(             (location.toString().includes('/pull/')                 ? [ location.toString() ]                 : Array.from(document.querySelectorAll(%27.js-issues-list-check:checked%27))                     .map(checkbox => checkbox.closest(%27.js-issue-row%27).querySelector(%27a%27).href)             )             .map(link => link                 .replace(%27github.com/%27, %27api.github.com/repos/%27)                 .replace(%27/pull/%27, %27/pulls/%27)                 .replace(/\/(files|checkes|commits)(#.*)?$/, '')             )             .map(link => fetch(link).then(r => r.json()))         );     } catch (err) {         usage();         return;     }         if (!prs.length) {         usage();         return;     }      const lines = prs.map(pr => %60origin pull/${pr.number}/head  # ${pr.title}%60).join('\n');     const do_deploy = confirm(%60Deploy the following PRs?\n${lines}%60);      if (!do_deploy) {         alert("Deploy Cancelled");         return;     }      const url = %60https://jenkins.openlibrary.org/job/testing-deploy/buildWithParameters?${new URLSearchParams({         token: 'XXX',         GH_REPO_AND_BRANCH: lines     })}%60;      const w = window.open(url);     await new Promise(res => setTimeout(res, 1000));     w.location = 'https://jenkins.openlibrary.org/blue/organizations/jenkins/testing-deploy/'; })();

---

Out of Scope (for now)

• Branch protection / approval workflows for testing deploys
• Automatic conflict resolution
• Multiple testing environments
• Notifications (Slack/email) on build status — though this would be a nice addition later

[mekarpeles]

Implementation Approach: Evolving the Existing Status Page

The good news is that most of the infrastructure already exists. make-integration-branch.sh already does the merge work, Jenkins already handles triggering, and status.py already renders the deployed state. The main change is adding a persistent state layer between "user requests deploy" and "script runs," and making that state manageable from /status.

---

1. New: Persistent State File

A JSON file (e.g. /opt/openlibrary/testing-prs.json) becomes the source of truth for what's deployed:

[
  {"pr": 12235, "ref": "origin pull/12235/head", "commit": "abc1234", "active": true, "title": "chore(deps): update vite to v8"},
  {"pr": 12188, "ref": "origin pull/12188/head", "commit": "111aaaa", "active": true, "title": "Add missing translation"}
]

Both the bookmarklet path (via Jenkins) and the /status dashboard read and write this same file.

---

2. Changes to make-integration-branch.sh

Currently it takes GH_REPO_AND_BRANCH as a newline-delimited list of refs and merges them onto master. Required changes:

• Read from the state file as its source of truth (in addition to or instead of the parameter)
• Merge pinned commits, not branch HEADs. Instead of git fetch origin pull/12235/head && git merge FETCH_HEAD, do git fetch origin pull/12235/head && git merge <pinned_sha>
• Skip inactive PRs — only merge entries where active: true
• Write build results back to the state file (merge SHA, timestamp, and which PR caused a conflict if one occurs)

The actual diff to the script is modest — mostly replacing the parameter-parsing loop with a JSON-reading step and adding the SHA pin logic. Estimated ~30-40 lines changed.

---

3. Changes to status.py

The current read-only status page becomes an interactive dashboard. Changes:

Read path (everyone): Read the state file and render the PR table. For each PR, call the GitHub API to fetch current branch HEAD and compute drift (commits between pinned SHA and HEAD). Also check merge status — if a PR has been merged to master, flag it.

Write path (maintainers only): POST endpoints for management actions. These action buttons and POST endpoints should only be available to logged-in users with "maintainer" usergroup permissions. Non-maintainers (and logged-out visitors) see the table as read-only.

Endpoints needed:

• POST /status/add — accepts a PR number (or GitHub PR URL), resolves it via GitHub API, pins at current branch HEAD, adds to state file, triggers rebuild
• POST /status/remove — removes selected PR(s) from the state file, triggers rebuild
• POST /status/toggle — toggles active/inactive for selected PR(s), triggers rebuild
• POST /status/pull-latest — updates pinned commit to current branch HEAD for selected PR(s), triggers rebuild
• POST /status/rebuild — triggers a rebuild from current state (all active PRs at their pinned commits)

"Trigger rebuild" means calling the same Jenkins buildWithParameters endpoint the bookmarklet currently uses, or invoking make-integration-branch.sh directly if co-hosted.

Since Open Library uses web.py, these are straightforward additions to the existing plugin structure. Estimated ~200-300 lines (endpoints + GitHub API calls + HTML template with table and action forms).

---

4. Bookmarklet Evolution (Additive Behavior)

Currently the bookmarklet sends the full list of refs to Jenkins. The key behavioral change: the bookmarklet should add PR(s) to the existing batch, not replace it. Removal happens only from /status.

Two options for the transition:

Option A — Update the bookmarklet to call POST /status/add with the PR number(s), then trigger a rebuild. Existing PRs in the state file are preserved. This is the cleaner path.

Option B — Keep the bookmarklet unchanged during transition. Have Jenkins itself merge new PRs into the existing state file as a pre-build step: parse GH_REPO_AND_BRANCH, add any new PRs to the state file (pinned at current HEAD), then build from the combined state. The bookmarklet keeps working identically and /status picks up the changes. This avoids a flag day.

Either way, the result is the same: bookmarklet adds, /status manages, make-integration-branch.sh builds from the state file.

---

5. Permissions Model

Visitor type	Can view table	Can see action buttons	Can POST actions
Logged out	✅	❌	❌
Logged in, non-maintainer	✅	❌	❌
Logged in, maintainer	✅	✅	✅

No new dependencies required — GitHub API calls use requests (already in the project), state file is plain JSON, and the template is standard web.py.

---

Migration Path

1. Add the state file and utility module
2. Update make-integration-branch.sh to read from the state file and pin commits
3. Build the dashboard UI on /status with maintainer-gated actions
4. Bridge the bookmarklet (Option A or B above)
5. Once stable, deprecate the bookmarklet entirely — /status becomes the single interface