Description
As part of the ongoing effort to improve consistency, quality and reliability, we have been adding new gates along the feature development path, such as writing a KEP, API review, production readiness review, conformance test requirements, etc. Because of this, we need to revisit our guidance for graduation criteria and perhaps the feature release process. The KEP template has a section for graduation criteria, and we have this set of guidance, but that's embedded in the doc about changing APIs and needs a broader focus.
As for feature release process, we have the KEP process. A quick check though shows our current process is not working as well as I think we'd like. Either that or there's a lot less going on that we think. Of about 133 KEPs, only 13 show as actually "implemented":
jbelamaric@jbelamaric:~/proj/gh/kubernetes/enhancements/keps$ grep -r status: sig-*/[0-9]*.md | wc -l
133
jbelamaric@jbelamaric:~/proj/gh/kubernetes/enhancements/keps$ grep -r status: sig-*/[0-9]*.md | cut -d: -f 3 | tr -d ' ' | sort | uniq -c
4
2 "False"
79 implementable
13 implemented
1 implemented)
1 proposal).
32 provisional
1 "True"
jbelamaric@jbelamaric:~/proj/gh/kubernetes/enhancements/keps$
When they are left in "provisional" and "implementable" state, even though at least some are certainly released in at least alpha or beta, it makes me wonder if we are properly reviewing the KEP graduation criteria as we merge/promote things. Maybe we are, but it's not clear, and I worry that we're not being very effective at reviewing the features and making sure that all the "i"s are dotted and "t"s are crossed.
So, the discussion I am trying to start is this:
- Do we need a clearer, more formal feature review process (presumably with the KEP as the primary artifact)?
- If so, who runs that? SIG-release? SIG-arch? Is it done by each SIG independently?
- Do we need clearer guidance for alpha/beta/GA that can then be used during that process to approve feature promotions?
- Who approves those?
As a starting point, some of the criteria for graduation at different levels could include:
- KEP review
- API review
- Completeness
- Production readiness review (and the criteria in there such as having playbooks, documented failure modes, etc.)
- Feature gating
- Level of unit and e2e test coverage
- Inclusion of conformance tests
- Inclusion of scalability tests
- Completeness and quality of documentation
- Upgrade / downgrade tests
- Real world usage statistics