[KEP-0] Purpose and process

[deepyaman]

What is a KEP?

KEP stands for Kedro Enhancement Proposal and is similar to a product requirement document commonly used in product management. The purpose of a KEP is to inform and involve the user community in major improvements to the Kedro codebase throughout the development process, to increase the likelihood that user needs are met.

KEPs should be used for significant user-facing or cross-cutting changes, not small incremental improvements. When in doubt, if a maintainer thinks a change needs an KEP, it does.

A KEP:

• Is a GitHub discussion of category "TSC Vote" proposing a major improvement or change to Kedro or a Kedro-maintained plugin
• Follows the template defined below
• Includes discussions about the proposal

Who participates in a KEP?

Any community member can help by discussing whether a KEP is likely to meet their needs, and by proposing KEPs.

Committers can help by discussing whether a KEP is likely to be technically feasible.

Maintainers can help by discussing whether a KEP aligns with long-term project goals, and by shepherding KEPs.

KEP Author is any community member who authors a KEP and is committed to pushing the change through the entire process. KEP authorship can be transferred.

KEP Shepherd is a Kedro maintainer who is committed to shepherding the proposed change throughout the entire process. Although the shepherd can delegate or work with other committers in the development process, the shepherd is ultimately responsible for the success or failure of the KEP. Responsibilities of the shepherd include, but are not limited to:

• Be the advocate for the proposed change
• Help push forward on design and achieve consensus among key stakeholders
• Review code changes, making sure the change follows project standards
• Get feedback from users and iterate on the design and implementation
• Uphold the quality of the changes, including verifying whether the changes satisfy the goal of the KEP and are absent of critical bugs before releasing them

What is the KEP process?

Proposing a KEP

Anyone may propose a KEP using the document template below. Please only submit an KEP if you are willing to help, at least with discussion.

If an KEP is too small or incremental and should have been done through the normal issue process, a committer should convert the discussion to an issue.

KEP document template

A KEP document is a short document with a few questions, inspired by the Heilmeier Catechism:

Q1. What are you trying to do? Articulate your objectives using absolutely no jargon.

Q2. What problem is this proposal NOT designed to solve?

Q3. How is it done today, and what are the limits of current practice?

Q4. What is new in your approach and why do you think it will be successful?

Q5. Who cares? If you are successful, what difference will it make?

Q6. What are the risks?

Q7. How long will it take?

Q8. What are the mid-term and final "exams" to check for success?

Appendix A. Proposed API Changes. Optional section defining APIs changes, if any. Backward and forward compatibility must be taken into account.

Appendix B. Optional Design Sketch: How are the goals going to be accomplished? Give sufficient technical detail to allow a contributor to judge whether it’s likely to be feasible. Note that this is not a full design document.

Appendix C. Optional Rejected Designs: What alternatives were considered? Why were they rejected? If no alternatives have been considered, the problem needs more thought.

Discussing a KEP

All discussion of a KEP should take place in a public forum. Any discussions that happen offline should be made available online for the public via meeting notes summarizing the discussions, to the extent possible.

During this discussion, one or more shepherds should be identified among maintainers.

Once the discussion settles, the shepherd(s) should call for a vote on the PEP moving forward in the #kedro-tsc channel on the Kedro Slack organisation. The vote should be open for at least one week and passes upon consensus (at least 3 +1 votes from maintainers and no -1 votes from maintainers). kedro-tsc should be notified of the vote result.

If there does not exist at least one maintainer that is committed to shepherding the change within a month, the KEP is rejected.

If a maintainer does not think a KEP aligns with long-term project goals, or is not practical at the point of proposal, the maintainer should -1 the KEP explicitly and give technical justifications.

Implementing a KEP

Implementation should take place via the standard process for code changes. Changes that require KEPs typically also require design documents to be written and reviewed.

[deepyaman]

I'm going to add some context in a comment:

• I've copied most of this verbatim from https://spark.apache.org/improvement-proposals.html; having reviewed a number of proposal processes across established projects, it seemed sufficiently lightweight and well-suited to adapt for Kedro's needs.
• Kedro technically already requires a 1/2 majority vote from maintainers for major changes. However, this process is not currently followed outside of Kedro-Datasets, so the new KEP process aims to formalize it.
  • I personally don't believe the process for adding new datasets needs to change at this time.
• I left the passing vote as "at least 3 +1 votes from maintainers" instead of "+1 votes from at least 1/2 of the maintainers," because it's difficult to get a meaningful review from 10 people, given the current size of the TSC.
  • This issue is apparent in the dataset approval process, where a few reviews are meaningful, and then there's an effort to get people to approve for the process' sake. The consensus process (with a -1 vote) and week-long voting period ensures 3 people can't just push through proposals.
• An approved KEP does NOT equal resource allocation or require anybody other than the shepherd(s) to work on the proposed enhancement.

[Galileo-Galilei]

Love the idea, and we definitely need a formal way to suggest enhancements. The template seems a bit generic to me, but it can be adapted on the long run eventually, and of it's good for spark It should be for us too.

I have some doubts about the shepherding process (not exactly sure of what it would imply to have someone who opt for shepherding within 1 month especially if there are no allocated resources) and I try to mentally imagine how it would have gone with my own 'universal deployment' serie of issues but once again, I'm definitely in favor to try it out.

[deepyaman]

The template seems a bit generic to me, but it can be adapted on the long run eventually, and of it's good for spark It should be for us too.

I agree that it's generic—NumPy and Kubernetes have more tailored processes, for example—but I think it's also not too complex, and, as you said, can be tailored to our needs down the line. I think having clear proposals to vote on is a first step in the right direction regardless. :)

I have some doubts about the shepherding process (not exactly sure of what it would imply to have someone who opt for shepherding within 1 month especially if there are no allocated resources) and I try to mentally imagine how it would have gone with my own 'universal deployment' serie of issues but once again, I'm definitely in favor to try it out.

In my view, the main benefits of the shepherding process are:

1. Significant proposals originating outside the TSC MUST find support from a TSC member. A recent example that comes to mind is Runner Component: Interactive Execution Workbench kedro-viz#2483. While I think it sounds like a great idea personally, and I am a TSC member, I'm not familiar enough with Kedro-Viz to sign up to shepherd such a proposal. While @AnselmAdrian has expressed interest in writing up and refining a design proposal, under this process he would need to get the buy-in from a TSC member who would ultimately take responsibility for making sure it's implemented in the right way for Kedro-Viz, reviewing the code, etc., even if Anselm was the one doing a lot of the code contribution. (Hope you don't mind me using your work as an example. :)) If he made the proposal and wasn't able to find somebody on the TSC to be the shepherd, we could reject the proposal—not because it's bad, but because we don't want to have a lot of proposals that people think are good ideas but nobody wants to commit to.
2. For something like your Universal Deployment series, you could be the shepherd. It doesn't mean work needs to get done or even start in the month. That said, I think the NumPy process has some good guidance: "The more focused the NEP, the more successful it tends to be. If in doubt, split your NEP into several well-focused ones." In my mind, a KEP would still represent a concrete technical proposal; something like Universal Kedro Deployment (Part 4) - Embedding kedro pipelines in third-party applications  #3540 could be broken into several KEPs. NumPy does have multiple types of NEPs, including Provisional status NEPs that can delegate work to future NEPs, but I think this may be overly much to define now.

[Galileo-Galilei]

Hard agree. A clear proposal on the broken voting process is necessary for the project, and all your comments make sense. This proposal will evolve along time but this is an excellent starting point.

[Joseph-Perkins]

The process makes logical sense to me! 👍

The only question I have remaining is around the time frames - which might end up being a bit short? For example if a Maintainer who intended to vote -1 on a KER happened to be on holiday and out of contact for 1 week (which seems possible), then a vote might sneak through despite their intended veto? haha

Should the voting period perhaps be extended to longer than a week?

[deepyaman]

The only question I have remaining is around the time frames - which might end up being a bit short? For example if a Maintainer who intended to vote -1 on a KER happened to be on holiday and out of contact for 1 week (which seems possible), then a vote might sneak through despite their intended veto? haha

I'm open to whatever seems reasonable! 2 weeks? I increased it from 72 hours to 1 week from the SPIP process already for exactly this reason.