Enhance deployment workflow: free code PVC before syncing to prevent … #314
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Deploy jaseci-blogs with jac-scale | |
| on: | |
| push: | |
| branches: | |
| - main | |
| workflow_dispatch: | |
| permissions: | |
| id-token: write | |
| contents: read | |
| jobs: | |
| deploy-jaseci-blogs: | |
| runs-on: ubuntu-latest | |
| permissions: | |
| id-token: write | |
| contents: read | |
| steps: | |
| - name: Checkout code | |
| uses: actions/checkout@v4 | |
| - name: Configure AWS Credentials | |
| uses: aws-actions/configure-aws-credentials@v4.2.0 | |
| with: | |
| aws-region: us-east-2 | |
| role-to-assume: arn:aws:iam::776241927220:role/GitHubActionsJaseciDeployRole | |
| role-session-name: GitHubActions-JaseciBlogs | |
| audience: sts.amazonaws.com | |
| - name: Update kubeconfig for EKS | |
| run: | | |
| aws eks update-kubeconfig --region us-east-2 --name jaseci-cluster | |
| kubectl config current-context | |
| - name: Set up Python 3.12 | |
| uses: actions/setup-python@v5 | |
| with: | |
| python-version: '3.12' | |
| - name: Install Jac toolchain and project dependencies | |
| run: | | |
| pip install --upgrade pip | |
| # jaclang core ships the `jac` CLI and is pip-installable. The Jac | |
| # plugins (jac-scale, jac-client) no longer carry a pyproject.toml | |
| # (jaseci-labs/jaseci#6322), so they install via `jac install`, which | |
| # reads [dependencies] in jac.toml into the project's .jac/venv. | |
| # `jac start` adds that venv to sys.path automatically. | |
| # Pin jaclang: the unpinned `pip install jaclang` pulls the latest, | |
| # whose newer in-pod binary bootstrap ("[jac] Fetching latest version | |
| # … Could not determine latest version from GitHub Releases") crash- | |
| # loops the pod. Pinned to the last pre-breakage release. Bump in | |
| # lockstep with the jac-scale/jac-client pins in jac.toml. | |
| pip install jaclang==0.16.0 | |
| jac install | |
| - name: Free the code PVC before syncing (scale down code-server) | |
| # ROOT CAUSE (jac-scale 0.2.23): the shared code PVC jaseci-blogs-code-pvc | |
| # is ReadWriteOnce on EBS (attaches to one node at a time). The long-lived | |
| # `code-server` pod holds it, and jac-scale runs the `code-sync` pod — which | |
| # needs the SAME RWO volume — BEFORE it recreates code-server. So during | |
| # sync, code-sync competes with the still-running code-server pod; if it | |
| # lands on a different node it wedges in Multi-Attach and the deploy times | |
| # out at "Syncing application code to PVC". Plain retry does NOT converge — | |
| # the scheduler keeps re-placing code-sync on the same off-node (observed | |
| # 5/5 failures in run 29217559535). | |
| # | |
| # DETERMINISTIC FIX: scale code-server to 0 first, detaching the volume, so | |
| # code-sync can attach on ANY node. jac-scale delete+recreates code-server | |
| # at the end of the deploy (kubernetes_target.jac:547-556), so this only | |
| # moves that teardown earlier. The APP pod serves from an emptyDir (it wgets | |
| # code from code-server over HTTP; it does NOT mount this PVC), so the live | |
| # site stays UP throughout. Permanent fix (RWX/EFS): .github/CODE_SYNC_RWO_FIX.md. | |
| run: | | |
| NS=jaseci-blogs | |
| if kubectl get deploy/jaseci-blogs-code-server -n "$NS" >/dev/null 2>&1; then | |
| SEL=$(kubectl get deploy/jaseci-blogs-code-server -n "$NS" \ | |
| -o go-template='{{range $k,$v := .spec.selector.matchLabels}}{{$k}}={{$v}},{{end}}' | sed 's/,$//') | |
| kubectl scale deploy/jaseci-blogs-code-server -n "$NS" --replicas=0 | |
| echo "Waiting for code-server pod to terminate (EBS volume detach)…" | |
| kubectl wait --for=delete pod -l "$SEL" -n "$NS" --timeout=120s || true | |
| fi | |
| # Clear any code-sync pod wedged by a prior run so jac-scale makes a fresh one. | |
| kubectl delete pod jaseci-blogs-code-sync -n "$NS" \ | |
| --ignore-not-found --force --grace-period=0 2>/dev/null || true | |
| - name: Deploy jaseci-blogs with jac-scale | |
| working-directory: . | |
| env: | |
| # Submission-portal secrets. jac.toml's [plugins.scale.secrets] | |
| # interpolates these ${...} env vars at deploy time into the | |
| # jaseci-blogs-secrets k8s Secret. Unset values deploy as empty, | |
| # which gracefully disables /submit + /reviewer. Only an OAuth App is | |
| # needed — every GitHub action runs as the signed-in user (fork + PR). | |
| # See SUBMIT_FLOW.md. | |
| GH_OAUTH_CLIENT_ID: ${{ vars.BLOG_OAUTH_CLIENT_ID }} | |
| GH_OAUTH_CLIENT_SECRET: ${{ secrets.BLOG_OAUTH_CLIENT_SECRET }} | |
| # One random value backs both: jac-scale signs the session JWT with | |
| # JWT_SECRET, and SUBMIT_SESSION_SECRET is the Fernet key for the GitHub | |
| # token at rest. Reused so there's no extra secret to manage (and so an | |
| # unset JWT_SECRET can't silently fall back to jac-scale's default). | |
| JWT_SECRET: ${{ secrets.BLOG_SESSION_SECRET }} | |
| SUBMIT_SESSION_SECRET: ${{ secrets.BLOG_SESSION_SECRET }} | |
| run: | | |
| # `jac install` put jaclang + jac-scale + jac-client into the project | |
| # venv (.jac/venv). Run THAT venv's jac so the jac-scale plugin loads | |
| # at startup and registers the `--scale` flag — the global `jac` (from | |
| # `pip install jaclang`) builds its arg parser before the venv is on | |
| # the path, so it does not recognize `--scale`. | |
| # | |
| # The "Free the code PVC" step above detached the RWO code volume, so the | |
| # code-sync pod can now attach on any node and this should succeed on the | |
| # first try. The short retry is belt-and-suspenders for transient EBS | |
| # detach latency; it force-deletes any wedged code-sync pod between tries. | |
| # Permanent fix (RWX/EFS): .github/CODE_SYNC_RWO_FIX.md. | |
| NS=jaseci-blogs | |
| for attempt in 1 2 3; do | |
| echo "::group::Deploy attempt ${attempt}/3" | |
| kubectl delete pod jaseci-blogs-code-sync -n "$NS" \ | |
| --ignore-not-found --force --grace-period=0 2>/dev/null || true | |
| kubectl wait --for=delete pod/jaseci-blogs-code-sync -n "$NS" \ | |
| --timeout=60s 2>/dev/null || true | |
| if .jac/venv/bin/jac start main.jac --scale; then | |
| echo "::endgroup::" | |
| echo "Deployed successfully on attempt ${attempt}." | |
| exit 0 | |
| fi | |
| echo "::endgroup::" | |
| echo "Attempt ${attempt} failed; retrying…" | |
| done | |
| echo "All attempts failed. The live app pod was never touched, so the site is" | |
| echo "still serving the last-synced code. See .github/CODE_SYNC_RWO_FIX.md." | |
| exit 1 | |
| - name: Ensure code-server is running | |
| if: always() | |
| # Safety net: jac-scale recreates code-server (at replicas=1) only after a | |
| # successful sync. If the deploy failed before that, code-server may be left | |
| # at 0 replicas from the "Free the code PVC" step — the site stays up (app | |
| # pod serves from emptyDir), but restore code-server so new app-pod boots and | |
| # the next sync aren't blocked. No-op once jac-scale has recreated it at 1. | |
| run: | | |
| NS=jaseci-blogs | |
| kubectl scale deploy/jaseci-blogs-code-server -n "$NS" --replicas=1 2>/dev/null || true | |
| - name: Verify deployment | |
| run: | | |
| echo "Checking deployment status..." | |
| kubectl get deployments -n jaseci-blogs | |
| kubectl get services -n jaseci-blogs | |
| kubectl get pods -n jaseci-blogs | |
| - name: Deployment Summary | |
| if: always() | |
| run: | | |
| echo "## jaseci-blogs Deployment Summary" >> $GITHUB_STEP_SUMMARY | |
| echo "### Configuration" >> $GITHUB_STEP_SUMMARY | |
| echo "- **App Name**: jaseci-blogs" >> $GITHUB_STEP_SUMMARY | |
| echo "- **Namespace**: jaseci-blogs" >> $GITHUB_STEP_SUMMARY | |
| echo "- **Cluster**: jaseci-cluster" >> $GITHUB_STEP_SUMMARY | |
| echo "- **Region**: us-east-2" >> $GITHUB_STEP_SUMMARY | |
| echo "- **Triggered By**: ${{ github.event_name }}" >> $GITHUB_STEP_SUMMARY | |
| echo "" >> $GITHUB_STEP_SUMMARY | |
| echo "### Deployment Status" >> $GITHUB_STEP_SUMMARY | |
| echo "\`\`\`" >> $GITHUB_STEP_SUMMARY | |
| kubectl get all -n jaseci-blogs 2>&1 >> $GITHUB_STEP_SUMMARY || echo "Unable to fetch deployment status" >> $GITHUB_STEP_SUMMARY | |
| echo "\`\`\`" >> $GITHUB_STEP_SUMMARY |