More thorough contribution guideline

[logan-keede]

I am opening this discussion to discuss about how to approach refactoring and perhaps changes in general to make it easier for downstream repos and be more efficient with review process.

This came up while discussing my GSoC 2025 proposal for "Optimizing compile time and binary size" with @ozankabak which expects a large amount of refactoring.

After some research, I found that almost no Open Source Repository has something like Refactoring Guideline and it is reasonable generally it is not needed, general contribution guideline is enough. However, Datafusion is perhaps a bit too refactoring happy/needy.
DataFusion :-

A repo with 17 times more commit then datafusion:-

Perhaps a direct comparison is not fair, because we do need refactoring. So the best we can do is to make it easier for everyone.

Proposed Solution

1. Make a feature branch, Do all the Major refactoring there publish a Roadmap on Why this refactoring/change is necessary and what does it change. This is perhaps more useful for refactoring Epics like [Epic] Split datasources out from datafusion crate (datafusion/core) #14444.
   suggested by @ozankabak over discord

2. Use 'cargo-semver-checks' to detect unintentional API breakages. Smallest things can break APIs in ways we can not predict. Here is an article about this.

3. add do's and don'ts in Guideline. Start with a tentative version and refine it over time.
   DataFusion already has a Contribution Guideline, which explain the general style with which we handle PRs and Issues but it does not go into great detail what to do and to not do. While this is not a big problem(if a problem at all) for more experienced member of community it is still good highlight Good and Bad Practice for the newer members.

This also make sure that we have a DataFusion way of dealing with problems and make sure that there is no unexpected or uninformed(as much as possible) API changes/breaking. It will also save some reviewing bandwidth as reviewer will not have to explain same old common reasons for rejection again and again.

It will be valuable to collect community's ideas on this and reviews of downstream maintainers on what kind of Datafusion issues they face that can be avoided through better policy in this discussion.

[ozankabak]

Thank you @logan-keede for this. There is indeed a lot of refactoring going on and I think we can do much better w.r.t. how we approach refactoring. A few thoughts:

• We don't do a good job at managing downstream effects of refactors (for forks, crate-level users and others): Refactors that seem innocuous/sensible for DF core can turn out to be bad ideas in a larger context. It would be great if we had a "protocol" for major refactor proposals; something that flows like:
  1. Start with a proposal,
  2. Create a feature branch containing the core ideas in the refactor for downstream projects to experiment with,
  3. Collect feedback from downstream projects to reveal any possible design issues,
  4. Publish a roadmap and upgrade guide
  5. Proceed with the implementation on main.
• We should channel our contributors' time to highest-priority items; i.e. direct their efforts into missing features, increasing test coverage and regression fixes instead of refactors that only improve the status quo slightly. I think we have a lot of refactors because it is "easy" to get refactors merged -- low-impact refactors are not that hard to create a PR for, but get merged without much scrutinization.

[logan-keede]

iii. Collect feedback from downstream projects to reveal any possible design issues

Do we have any communication channel for collecting feedback, or announcing feature branch?

@alamb What are your thoughts on this, maybe we can try out 'feature branch' approach with remaining work in #14444?

[ozankabak]

iii. Collect feedback from downstream projects to reveal any possible design issues

Do we have any communication channel for collecting feedback, or announcing feature branch?

We don't -- we announce things that can affect downstream in a few places simultaneously (Slack, Discord, e-mail list).

@alamb What are your thoughts on this, maybe we can try out 'feature branch' approach with remaining work in #14444?

I think #14444 already made a lot of progress and significant parts of it merged to main, so I don't think it would be a good vehicle to test a new methodology, but let's hear from @alamb.

[alamb]

We do have this guide for API stability (aka trying to reduce breaking changes) https://datafusion.apache.org/contributor-guide/api-health.html

Use 'cargo-semver-checks' to detect unintentional API breakages. Smallest things can break APIs in ways we can not predict. Here is an article about this.

This is a great idea. The other thing we have been trying to do more recently is test with downstream crates before we make a release, to catch regressions

• Release DataFusion 47.0.0 (April 2025) #15072

• I agree that trying to do long lived feature branches is tough -- and something like [Epic] Split datasources out from datafusion crate (datafusion/core) #14444 will be especially tough as it it will generate many conflicts

[alamb]

In general I think there will always be a tension between:

1. What is good for people developing DataFusion (aka making improvements as few constraints as possible)
2. What is good for specific projects (aka making API changes as slowly / minimally as possible)

[alamb]

Use 'cargo-semver-checks' to detect unintentional API breakages. Smallest things can break APIs in ways we can not predict. Here is an article about this.

@logan-keede could you potentially file a ticket with this idea (adidng a cargo semver CI check?)

[logan-keede]

Filed #15408.