New RFC: Pipeline maturity model (Very Important Pipelines)

[JoseEspinosa]

Have you read the RFC docs?

• Yes, I have read and understood the RFC docs

Summary

The idea is that pipelines widely adopted in the community should adhere more strictly to the best practices promoted by nf-core, as they serve as templates and references for the development of the other pipelines.

Champion

@JoseEspinosa

Background & Motivation

Widely adopted pipelines in the community should stick to some strong recommendations, as they are critical for nf-core and should promote best practices for other pipelines. We therefore propose a tiering system for pipeline development: the more popular and widely used a pipeline becomes, the higher the expectations for following good software development life cycle practices.

Considerations for levels

We could consider including in the tier those pipelines that receive a lot of contributions (e.g. number of PRs). Other criteria for inclusion could include the number of stars in the repository, forks, age, number of issues, and website visitors. Or just decide it organically, taking into account several of the above criteria.

Naming

The name of the tier could be VIP (Very Important Pipelines)

Recommendations for VIP in terms of team

• Have a team of co-developers
• Have a GitHub team
• Have a team of CODEOWNERS
• Regular development team meetings (open to all)
• Use milestones to plan releases
• Have a separate developer Slack channel (separate from user questions)

Other highly recommended guidelines

• Regularly review default config usage
• Latest release to be within 2 tools / template releases (up to date)
• nf-test coverage > 80%
• Issue advisories when there have been known issues
• Have done a bytesize talk
• Should have a logo and stickers
• Should have a metro map
• Should have extended docs
• All outputs described (to be described in guidelins)
• Should have usage tutorials

Implementation

We will need to incorporate the docs into the website with the final proposal, and the core team will identify a handful of pipelines to be the first VIP pipelines (according to the criteria) and reach out to them.

Branding

An additional badge in the readme / website would be nice.

Goals

• Increase visibility of nf-core pipelines that are widely adopted in the community.
• Promote best development practices and enforce recommended guidelines for popular pipelines.
• Provide a model for other pipelines to align with and adopt these practices.

Non-Goals

• The tier system is not meant to devalue or discourage work on less widely adopted pipelines.
• Standard nf-core practices still apply to all pipelines; VIP adds extra expectations, not new requirements for everyone.
• Criteria (e.g. stars, PRs, forks) are indicative, not strict thresholds, and may evolve organically.

References

No response

[nf-core-bot]

Approval status: ✅ Approved

RFC has approvals from 7/7 required @nf-core/core quorum.

Review Status	Core Team members
✅ Approved	@JoseEspinosa, @mashehu, @nvnieuwk, @ewels, @edmundmiller, @maxulysse, @FriederikeHanssen
🕐 Pending	@mribeirodantas, @mirpedrol, @kenibrewer, @jfy133, @sateeshperi, @FranBonath, @christopher-hakkaart

[JoseEspinosa]

/approve

[mashehu]

/approve

one thing we might need to polish a bit, is what pipeline devs get out of it. All of them will help them in the long run, but i can understand that the immediate benefit is not clear and "just extra work". Maybe if they follow all the guidelines they get a badge on the website?

[nvnieuwk]

/approve

[ewels]

/approve

Only thing I'm missing here is the "how" - I guess this will be a case of some new docs at first, and core-team reaching out to a handful of select pipelines?

How do we ensure continued adherence?

Agree with @mashehu - an additional badge in the readme / website would be nice.

[pinin4fjords]

one thing we might need to polish a bit, is what pipeline devs get out of it. All of them will help them in the long run, but i can understand that the immediate benefit is not clear and "just extra work". Maybe if they follow all the guidelines they get a badge on the website?

Agreed. We don't really enforce a lot of existing regs, so I predict a low level of compliance here without incentives. Hoodie saying "Look how shiny my pipeline is"?

[JoseEspinosa]

I updated the proposal according to your comments @mashehu @ewels and @pinin4fjords

[edmundmiller]

/approve

[maxulysse]

/approve

[apeltzer]

x-ref nf-core/stats#73 - considering to use some of the metrics Jose mentioned here too in such an endeavour for "regulatory readiness".