Skip to content

Conversation

@jcgruenhage
Copy link
Member

@jcgruenhage jcgruenhage commented Nov 14, 2025

GHA job URL as fallback

We have a job like this everywhere where we use the deploy queue:

- name: Get URL for current job
  id: get-current-job-url
  env:
    GH_TOKEN: ${{ github.token }}
  run: |
    html_url=$(gh api --paginate repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/jobs \
      --jq '.jobs[] | select(.runner_name =="${{ runner.name }}") | .html_url')
    echo "html-url=$html_url" | tee -a ${GITHUB_OUTPUT}
  if: ${{ vars.DEPLOY_QUEUE_ENABLED == 'true' }}

For job steps, is definitely not necessary, we get the same URL like this:

- run:
    echo "${SERVER_URL}/${REPOSITORY}/actions/runs/${RUN_ID}/job/${CHECK_RUN_ID}"
  env:
    SERVER_URL: ${{ github.server_url }}
    REPOSITORY: ${{ github.repository }}
    RUN_ID: ${{ github.run_id }}
    CHECK_RUN_ID: ${{ job.check_run_id }}

Hoping that composite actions also get access to job.check_run_id, I'm trying to fully inline this.

Small follow-up fixes for #67

  • align output of list cells with slack message
  • expose list-cells through gh action
  • name list cells consistently

Adjustments to github output handling

  • support multi-line outputs
  • use advisory locks to prevent accidental concurrent writes

Added after flakyness like seen in https://github.com/neondatabase/dev-actions/actions/runs/19367137928/job/55414533340?pr=70 printing things like this:

All tests completed successfully
Error: Unable to process file command 'output' successfully.
Error: Invalid format '6'

Not knowing that this is a concurrency problem and just trying a random thing that came to mind I implemented multi-line output support. This turned it from flaky to always failing, but gave very clear output showing that this is a concurrency issue:

All tests completed successfully
Error: Unable to process file command 'output' successfully.
Error: Invalid value. Matching delimiter not found '<<<<EOFEOFEOF'

Now with the concurrency problem fixed, it's stable again.

Test updates:

  • Add compatibility tests that use the action itself for the happy path to ensure basic e2e backwards compatibility
  • Run integration tests of older versions against the new schema to ensure query compatibility

I'm adding both because the compatibility problem between 0.5.1 and 0.6.0 with missing migrations being treated as a critical problem would not have been caught by the latter, while most more nuanced query-level problems would not be caught by the former.

@jcgruenhage jcgruenhage requested a review from a team as a code owner November 14, 2025 13:48
@jcgruenhage jcgruenhage requested review from bayandin and removed request for a team November 14, 2025 13:48
@jcgruenhage jcgruenhage force-pushed the jc.gruenhage/push-yxwprytzrkor branch from d055fc3 to 4495659 Compare November 14, 2025 13:58
@jcgruenhage jcgruenhage force-pushed the jc.gruenhage/push-yxwprytzrkor branch 3 times, most recently from 7c356e1 to 392fe81 Compare November 14, 2025 16:48
@jcgruenhage jcgruenhage force-pushed the jc.gruenhage/push-yxwprytzrkor branch 3 times, most recently from e23be37 to 56057da Compare November 18, 2025 10:57
An issue that only surfaced in tests is that concurrent writes to
`GITHUB_OUTPUT` sometimes mess up outputs. Using advisory locks prevents
this effectively.
As the deploy-queue runs in a bunch of different places and in some
cases with git repos that are not up-to-date, we'll regularly have older
versions of deploy-queue still in service. Requiring all migrations to
be present in the source code means that as soon as a new migration is
deployed, all older versions will break. That is unacceptable, so we'll
have to allow running with missing migrations.
@jcgruenhage jcgruenhage force-pushed the jc.gruenhage/push-yxwprytzrkor branch from 56057da to 44b0c6d Compare November 18, 2025 11:36
Copy link
Contributor

@bayandin bayandin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the workflow part — nice!
And added a question, if everything is fine — LGTM

@jcgruenhage jcgruenhage force-pushed the jc.gruenhage/push-yxwprytzrkor branch from 44b0c6d to 4e51c78 Compare November 18, 2025 12:48
@jcgruenhage
Copy link
Member Author

Only new changes are the newly introduced compatibility tests and the matrix strategy for the integration tests, the rest remained unchanged:

❯ git range-diff 781f934..44b0c6d 781f934..4e51c782
 1:  4734560 =  1:  4734560 impr(deploy-queue): use GHA job URL as fallback
 2:  6c99e77 =  2:  6c99e77 fix(deploy-queue): name `list cells` consistently
 3:  adadde0 =  3:  adadde0 fix(deploy-queue): expose `list-cells` through gh action
 4:  f460f8f =  4:  f460f8f fix(deploy-queue): align output of `list cells` with slack message
 5:  f8d2320 =  5:  f8d2320 chore(deploy-queue): don't set URL for e2e tests
 6:  f3aec19 =  6:  f3aec19 fix(deploy-queue): support multi-line github outputs
 7:  4afb423 =  7:  4afb423 fix(deploy-queue): exclusive advisory locks for writes to GITHUB_OUTPUT
 8:  667efa1 =  8:  667efa1 fix(deploy-queue): allow running with missing migrations
 -:  ------- >  9:  7f03960 impr(deploy-queue): test old and new versions against same database
 -:  ------- > 10:  0817dd6 impr(deploy-queue): run integration tests against newer schema
 9:  44b0c6d = 11:  4e51c78 chore(deploy-queue): bump version to 0.7.0

@jcgruenhage jcgruenhage force-pushed the jc.gruenhage/push-yxwprytzrkor branch from 4e51c78 to c6a0fa8 Compare November 18, 2025 12:56
@jcgruenhage
Copy link
Member Author

And as expected, testing against 0.5.1 failed because 0.5.1 refuses to work when there are unknown migrations applied to the database. See https://github.com/neondatabase/dev-actions/actions/runs/19466750059/job/55703743925.

I've removed the test for 0.5.1, but we'll have to make sure nothing is referencing 0.5.1 anymore before we can update to >=0.6.0. We've got 0.5.2 as an intermediate release to help with that.

@jcgruenhage jcgruenhage changed the title impr(deploy-queue): use GHA job URL as fallback deploy-queue v0.7.0 Nov 18, 2025
@Shugenya Shugenya self-requested a review November 18, 2025 14:19
Copy link
Contributor

@Shugenya Shugenya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!
I guess this means we can finally get rid of the "Get URL for current job" step

@jcgruenhage jcgruenhage merged commit c6a0fa8 into main Nov 18, 2025
13 checks passed
@jcgruenhage jcgruenhage deleted the jc.gruenhage/push-yxwprytzrkor branch November 18, 2025 14:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants