Potential Pipeline Metrics for Risk Assessment

[christopher-mohr]

In preparation of the "regulatory readiness" project at the upcoming nf-core hackathon, we've started brainstorming potential metrics that would potentially be needed/useful in that context.

• Maintenance

  • Number of solved issues -> aggregate from issue_stats
  • Median time to bug closure
    • median time to issue closure can be aggregated from issue_stats
    • issue labels are not properly captured, so we can't distinguish between bugs and other issues Capture issue labels #84
  • Number of releases Number of releases #83
  • Date of last release -> nfcore_pipelines.last_release_date
  • Has issue tracker -> all nf-core repos have an issue tracker
  • Has maintainer Number of maintainers #82
    • this is also a governance topic

• Popularity

  • Number of contributors -> aggregate from contributor_stats
  • Number of downloads clones -> aggregate from traffic_stats.clones_uniques
  • Number of Views -> aggregate from traffic_stats.views
  • GitHub Stars -> nfcore_pipelines.stargazers_count
  • GitHub Forks -> nfcore_pipelines.forks_count
  • Citations Get number of citations per pipeline #79

• Quality

  • Dependencies containerized and containers versioned Container stats #121
  • Test coverage of modules Module test coverage #80
  • Pipeline end-to-end tests
    • needs to be manually checked for the given set of input parameters, hard to flag this globally in a use-case agnostic manner
  • PR reviews mandatory? / 4-eye principle (nf-core guidelines reference) Collect PR review policies #89
    https://nf-co.re/pipelines.json
    -> This is also a governance topic.
  • Size of codebase Size of codebase #81

We plan to scan existing issues and PRs to identify which of these metrics are already being tracked or discussed. Feel free to comment, suggest additions, or flag any metrics that might be difficult to track or interpret.

CC @apeltzer @grst

[mashehu]

great idea. Has it been submitted to the hackathon projects list yet?

[apeltzer]

No :) I can do that!

[apeltzer]

Done!

[apeltzer]

Ok agreed discussion on nf-core hackathon in barcelona today was like this:

• Creating meta issues for services we use to query specific metrics

• Cross-ref individual metrics to these tickets, e.g. "Github stars" would cross-ref the "Github API Queries" ticket GitHub API Queries #76

• Repeat until we have at least a decent list of metrics of highest importance / agreement 🙂

• Milestone planning can then be done via the Github Milestones with the individual issues to put up for future implementation more effectively.