Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a scheduled workflow to synchronize branches from gitster/git #2

Merged
merged 3 commits into from
Aug 26, 2023

Conversation

dscho
Copy link
Member

@dscho dscho commented Aug 18, 2023

This ports over https://dev.azure.com/gitgitgadget/git/_build?definitionId=8 from an Azure Pipeline to a GitHub workflow.

@dscho dscho requested a review from webstech August 18, 2023 22:31
@dscho dscho self-assigned this Aug 18, 2023
Copy link
Collaborator

@webstech webstech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still in progress but I have to step away for a bit.

Comment on lines 66 to 90
} else if (c === 0 && sourceRefs[i].object.sha !== targetRefs[j].object.sha) {
// target ref needs updating
const sha = sourceRefs[i].object.sha
toFetch.add(sha)
refspecs.push(`+${sha}:${sourceRefs[i++].ref}`)
j++
} else {
i++
j++
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a little clearer on the increments?

              } else { // they are a match
               if (sourceRefs[i].object.sha !== targetRefs[j].object.sha) {
                  // target ref needs updating
                  const sha = sourceRefs[i].object.sha
                  toFetch.add(sha)
                  refspecs.push(`+${sha}:${sourceRefs[i].ref}`)
                } 
                i++
                j++
              }

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point!

dscho added 3 commits August 23, 2023 10:07
The Git maintainer publishes only the integration branches to `git/git`,
leaving the individual contribution branches in `gitster/git`.

We do want to make sure that the latter branches are available, and
synchronized, in `gitgitgadget/git`, to facilitate contributions to
patch series that are in flight.

So let's synchronize the branches, making sure that the integration
branches (`maint`, `master`, `next`, `seen`) are not synchronized from
`gitster/git`; They are synchronized via the `sync-ref` workflow
already.

Signed-off-by: Johannes Schindelin <[email protected]>
There are a lot of branches in `gitster/git`. A _lot_. So let's be nice
to GitHub and back off (but do try again) if encountering a 502 (which
indicates that GitHub is as overloaded as the rest of us).

Signed-off-by: Johannes Schindelin <[email protected]>
It does not make sense to try synchronizing in parallel.

I considered adding `cancel-in-progress: true`, but that strikes me as
wasteful: if we're already pushing an update, canceling it would require
the entire work to be done again, while chances are that any queued
run will likely find that everything is already up to date and no
further work needs to be done.

Signed-off-by: Johannes Schindelin <[email protected]>
@dscho dscho force-pushed the sync-gitster-git branch from e2422e2 to 525752b Compare August 23, 2023 08:11
@dscho
Copy link
Member Author

dscho commented Aug 23, 2023

@webstech here is another iteration of this PR. Range-diff to what you reviewed:

  • 1: 0627ed0 ! 1: f69d3e6 Add a scheduled workflow to synchronize branches from gitster/git

    @@ .github/workflows/sync-gitster-git.yml (new)
     +              const { data } = await github.rest.git.listMatchingRefs({
     +                owner,
     +                repo,
    ++                // We want to match `maint-*` as well as `[a-z][a-z]/*`
    ++                // sadly, this is not possible via GitHub's REST API,
    ++                // hence we do it below via the `filter()` call.
     +                ref: 'heads/'
     +              })
     +              return data
    @@ .github/workflows/sync-gitster-git.yml (new)
     +            const refspecs = []
     +            const toFetch = new Set()
     +            for (let i = 0, j = 0; i < sourceRefs.length || j < targetRefs.length; ) {
    -+              const c =  i >= sourceRefs.length
    ++              const compare =  i >= sourceRefs.length
     +                ? +1
     +                : j >= targetRefs.length
     +                  ? -1
     +                  : sourceRefs[i].ref.localeCompare(targetRefs[j].ref)
    -+              if (c > 0) {
    ++              if (compare > 0) {
     +                // no source ref => delete target ref
    -+                refspecs.push(`:${targetRefs[j++].ref}`)
    -+              } else if (c < 0) {
    ++                refspecs.push(`:${targetRefs[j].ref}`)
    ++                j++
    ++              } else if (compare < 0) {
     +                // no corresponding target ref yet => push source ref (new)
     +                const sha = sourceRefs[i].object.sha
     +                toFetch.add(sha)
    -+                refspecs.push(`${sha}:${sourceRefs[i++].ref}`)
    -+              } else if (c === 0 && sourceRefs[i].object.sha !== targetRefs[j].object.sha) {
    -+                // target ref needs updating
    -+                const sha = sourceRefs[i].object.sha
    -+                toFetch.add(sha)
    -+                refspecs.push(`+${sha}:${sourceRefs[i++].ref}`)
    -+                j++
    ++                refspecs.push(`${sha}:${sourceRefs[i].ref}`)
    ++                i++
     +              } else {
    ++                // the sourceRef's name matches the targetRef's
    ++                if (sourceRefs[i].object.sha !== targetRefs[j].object.sha) {
    ++                  // target ref needs updating
    ++                  const sha = sourceRefs[i].object.sha
    ++                  toFetch.add(sha)
    ++                  refspecs.push(`+${sha}:${sourceRefs[i].ref}`)
    ++                }
     +                i++
     +                j++
     +              }
  • 2: e2422e2 ! 2: 1fabadb sync-gitster-git: handle 502 gracefully

    @@ .github/workflows/sync-gitster-git.yml: jobs:
     -              const { data } = await github.rest.git.listMatchingRefs({
     -                owner,
     -                repo,
    +-                // We want to match `maint-*` as well as `[a-z][a-z]/*`
    +-                // sadly, this is not possible via GitHub's REST API,
    +-                // hence we do it below via the `filter()` call.
     -                ref: 'heads/'
     -              })
     -              return data
     -                .filter(e => e.ref.match(/^refs\/heads\/(maint-\d|[a-z][a-z]\/)/))
     -                .sort((a, b) => a.ref.localeCompare(b.ref))
    -+              for (let i = 1; i <= 10; i++) {
    ++              let attemptCounter = 1
    ++              for (;;) {
     +                try {
     +                  const [owner, repo] = repository.split('/')
     +                  const { data } = await github.rest.git.listMatchingRefs({
     +                    owner,
     +                    repo,
    ++                    // We want to match `maint-*` as well as `[a-z][a-z]/*`
    ++                    // sadly, this is not possible via GitHub's REST API,
    ++                    // hence we do it below via the `filter()` call.
     +                    ref: 'heads/'
     +                  })
     +                  return data
    @@ .github/workflows/sync-gitster-git.yml: jobs:
     +                  if (e?.status !== 502) throw e
     +                }
     +
    -+                const seconds = i * i + 15 * Math.random()
    ++                if (++attemptCounter > 10) throw new Error('Giving up listing refs after 10 attempts')
    ++
    ++                const seconds = attemptCounter * attemptCounter + 15 * Math.random()
     +                core.info(`Encountered a Server Error; retrying in ${seconds} seconds`)
     +                await sleep(1000 * seconds)
     +              }
  • -: ------- > 3: 525752b sync-gitster-git-branches: limit concurrency

Copy link
Collaborator

@webstech webstech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

@webstech webstech merged commit fa784e9 into gitgitgadget:main Aug 26, 2023
@dscho dscho deleted the sync-gitster-git branch August 27, 2023 06:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants