Only run benchmark/test suite once all the linting/formatting/etc passes

[sbryngelson]

Problem it solves

Run-running testing/benchmarking often.

Proposed solution

Use a yaml workflow addition that require linting/formatting, and other short checks to pass before running tests or benchmarks. e.g.,

# .github/workflows/workflow-b.yml
name: Workflow B

on:
  workflow_run:
    workflows: ["Workflow A"]
    types:
      - completed

jobs:
  job_in_b:
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    runs-on: ubuntu-latest
    steps:
      - run: echo "Workflow A finished successfully. Running workflow B."

[adbox53]

Hi, I’m a first-time contributor interested in helping with this issue as part of a class project.

From the description, it seems the goal is to ensure benchmarking and testing only run after linting, formatting, and other short checks have passed successfully. The YAML example provides helpful context for the proposed solution.

Could you please clarify what other short checks you envision being required along with linting and formatting? Also, is there any existing workflow setup I should be aware of or follow?

I’m happy to explore and propose a solution once I have some guidance!

[sbryngelson]

I'm going to take care of this task myself unless you beat me to it.

I envision all the short checks to happen before testing.

[Malmahrouqi3]

events-that-trigger-workflows
Looked into it briefly, so apparently workflow_run triggers per workflow listed and can't chain more than 3 events, i.e. if you list all quality workflows then you will call test.yml 11 times. The only way I can think of is by making two workflow groups Quality Check-> Test & Bench and if all quality tests pass then run test.yml & bench.yml. There might be an easier way tho.

quality.yml

for ["Lint Toolchain", "Lint Source", "Spell Check", "Pretty", "Source DRYness", "Lines of Code", "Coverage Check", "Cleanliness", "Documentation", "Deploy Homebrew Tap", "Homebrew Formula Test"]
Triggers all quality workflows and listen to them. Sample script for a single workflow being triggered and constantly monitored.

      - name: Dispatch and Listen
        uses: actions/github-script@v7
        with:
          script: |
            const tags = '${{ steps.tags.outputs.tags }}'.split(' ').filter(Boolean);
            for (const tag of tags) {
              await github.rest.actions.createWorkflowDispatch({
                owner: context.repo.owner,
                repo: context.repo.repo,
                workflow_id: 'workflow.yml',
                ref: context.ref.replace('refs/heads/', ''),
                inputs: { tag }
              });

              let runId = null;
              for (let i = 0; i < 60; i++) { // Try for up to 300 seconds
                const runs = await github.rest.actions.listWorkflowRuns({
                  owner: context.repo.owner,
                  repo: context.repo.repo,
                  workflow_id: 'workflow.yml',
                  event: 'workflow_dispatch',
                  per_page: 5
                });
                // Find the most recent queued or in_progress run
                const found = runs.data.workflow_runs.find(run =>
                  ['queued', 'in_progress'].includes(run.status)
                );
                if (found) {
                  runId = found.id;
                  break;
                }
                await new Promise(resolve => setTimeout(resolve, 5000));
              }
              if (!runId) {
                core.setFailed(`Could not find workflow run for tag ${tag}`);
                return;
              }

              let completed = false;
              while (!completed) {
                const run = await github.rest.actions.getWorkflowRun({
                  owner: context.repo.owner,
                  repo: context.repo.repo,
                  run_id: runId
                });
                if (run.data.status === 'completed') {
                  if (run.data.conclusion !== 'success') {
                    core.setFailed(`Workflow for tag ${tag} failed with conclusion: ${run.data.conclusion}`);
                    return;
                  }
                  completed = true;
                } else {
                  await new Promise(resolve => setTimeout(resolve, 15000));
                }

group.yml

name: Test & Bench

on:
  workflow_run:
    workflows:
      - "Quality Check"
    types:
      - completed

jobs:
  gate:
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    runs-on: ubuntu-latest
    steps:
      - run: echo "Quality Check finished successfully.
             Running testing & benchmarking."
  
  test-suite:
    needs: gate
    uses: ./.github/workflows/test.yml

  benchmarks:
    needs: gate
    uses: ./.github/workflows/bench.yml

[sbryngelson]

Thanks for the info. What is the status of your open PRs? We should push to finish those rather than starting new ones...

[Malmahrouqi3]

Currently gathering bench data ~28 runs across all releases.