Skip to content

Speedup and improve flaky detection #11

@AlexWaygood

Description

@AlexWaygood

On my average PR run of ecosystem-analyzer, I'm generally only interested in whether my PR adds/removes any non-flaky diagnostics. If a diagnostic is detected as being flaky by ecosystem-analyzer, I'd prefer it if the diagnostic could be excluded from the diagnostic summary table posted on PRs: currently we often get a summary table reporting that 40 invalid-await diagnostics were added, and it's only after clicking through to the separate tab that you can see that these were all diagnostics on prefect, and these diagnostics were all determined by ecosystem-analyzer to be flaky.

I also think we could easily speedup the process for flaky projects, which currently runs all projects in flaky.txt 10 times:

  • If there are no changes relative to main on the first run of ecosystem-analyzer on the PR branch, I think we can skip any reruns of ty on that project. There might be changes to which diagnostics are reported as flaky on the PR branch, but 10 runs isn't generally enough to make a concrete determination on that anyway (we do the weekly Wednesday workflow of ecosystem-analyzer with even more runs of ty for each project for this very reason), and again -- I'm usually only interested in non-flaky diagnostic changes on a PR comment
  • If after 2 or 3 runs, ecosystem-analyzer determines that all changes from the first run were flaky, I think it can short-circuit.

We probably want to retain an option to have the flaky detection run each project a fixed number of times, so that the weekly workflow can still provide a report of which projects are newly flaky or no longer flaky. But I think it's overkill for the normal PR workflow.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions