Proposal for change to Cats 2.0 plan

[kailuowang]

UPDATE: Closed for now to be replace with a revised draft incorporating feedbacks.

@LukaJCB and I would like to propose a new plan for Cats major version release. We think this plan will ease our future evolution pain/anxiety, bring high value features sooner without negative impact on the user side.

Cats is in a similar situation as Scala stdlib as that it's relatively stable now and widely used by the ecosystem and thus must remain binary compatible for extended period. Scala stdlib, as well as the Scala lang, gradually evolves with breaking changes thanks to the cross releasing machinery (i.e. the Scala version suffix of library artifact names) in the Scala ecosystem. By aligning Cats' breaking change release cycle with the Scala release cycle, Cats-core can evolve with breaking changes this way as well.
By "aligning", We mean specifically:

1. timing alignment
2. All cats-core releases for a certain Scala major version guarantee to be BC
3. Introduce BC-breaking changes only for a new Scala major version.

More concretely speaking, we propose that Cats 2.0 be aligned with Scala 2.13, with all the would-be-breaking changes only on cats-core_2.13. "Would-be-breaking" because there is no previous 2.13 release to be breaking with. cats-core_2.12 (regardless of which Cats major version it is) remains binary compatible (until we drop support for Scala 2.12). Cats 3.0 will be aligned with Scala 2.14 in the same fashion, Cats 4.0 with Scala 3.0, etc.

Like Scala std lib, to maintain multiple incompatible major versions, we'll need multiple parallel source trees. In the meantime we'll try maintain source compatibility across different Scala versions Cats source. To ensure this, we will maintain multiple scala version specific Cats source trees in a single branch but in different source folders, while having our tests cross build. It should be fine if we change our mind and switch to use git branch based parallel source trees later.

In regards to the incoming 2.0 release, we have [a list of things]((https://github.com/typelevel/cats/milestones/3.0) that we originally scheduled for 3.0. We want to take this opportunity to move some of them to 2.0 release on 2.13 source tree. (On the other hand if we miss this time window, it will have to wait until Scala 2.14.)
The criteria for selecting items to move is:

1. High value
2. The API of which has been stable
3. Source compatible and thus cross build friendly to users

One of such high value items would be new type based zero cost monad transformers. Another important factor here is that **there are few applications running Scala 2.13 on production at the time, this should give us some time to test and fix the potential issues brought by these 2.13 only changes.

We estimate the effort to likely take 2-3 weeks, maybe quicker if we can recruit some contributors to parallelize it.

There are two separate decisions in this proposal:

1. Shall we align our breaking change release cycle as Scala major version and evolve likewise taking advantage of the Scala version suffix based release machinery. The alternative will be a breaking Cats release somewhere in the middle of the Scala major version circle and force the whole ecosystem to upgrade (probably under a different root package name as well as publishing org to avoid conflicts)
2. Shall we start this on Cats 2.0 and spend a couple of weeks to roll in some of the high value Cats 3.0 items.

We need to move quickly as the ecosystem is waiting on Cats 2.0

[kubukoz]

I'm mildly 👎 because the whole ecosystem will be blocked for at least a month (I assume it'll take as much to get a release). I'd rather get 2.0 as it is now and start working on 3.0 that'd be released ~6mo from now, with all the breaking changes that were planned for the whole plan. Just my 5 cents.

[kailuowang]

@kubukoz thanks for your feedback. I thought about this idea but the issue is that once an 2.0 for Scala 2.13 is out it is much harder to break it, you will have to chase to upgrade the whole ecosystem. The much easier way is to sync it with Scala release. And breaking a major release in just 6 months might not be well received.
Another thing is that 2.0 is blocked by a Scalatest anyway (admittedly there is less ideal work around which involves duplicate code to a new separate repo).
If we can get enough volunteers to contribute to this effort I think we can get it done in less than a month.

[erwan]

I agree that it's a good opportunity to break compatibility now, when people are upgrading all their libraries for 2.13.

If cats breaks compatibility 6 months from now, people will have already done their big migration for 2.13 and this will be another round of upgrading several libraries at once (all libraries depending on cats). I suspect that many projects ("user" projects, i.e. applications) would then put off the upgrade of the typelevel ecosystem.

[erwan]

just to be clear, @kailuowang , which of the 3.0 tickets would be candidate to be moved to 2.0?

[travisbrown]

As you know, I think this is a move in the wrong direction, for several reasons:

1. Extending bincompat guarantees prioritizes the needs of one set of users (commercial adopters with large codebases) over others (new users, new contributors, maintainers), and one set of values (low-cost adoption) over others (innovation, consistency, pedagogical usefulness, elegance).

2. Maintaining consistency across multiple source trees is going to be difficult and expensive. The Cats codebase is already crammed with inconsistencies (e.g. whether any given *Functions or *Instances or *BinCompat trait is part of the public API or not is more or less random), and version-specific source trees is going to make the problem much, much worse.

3. It's a major break with Scala library convention and user expectations. I answer a lot of Stack Overflow questions, and I know this change will result in hundreds of awkward "if you're on Cats 2 and Scala 2.13 or Cats 3 and 2.14, then it works like X, but on Cats 2 and 2.12 or Cats 1, then it's Y" explanations.

4. It imposes new costs on other library maintainers. I personally want to minimize version-specific code in my projects, and this change is extremely likely to mean I'll need more version-specific code, which in turn means I'm less likely to decide to depend on Cats in my libraries.

5. It imposes new expectations on other library maintainers. I make zero bincompat guarantees in Circe, for example, because for me innovation and elegance are more important than minimizing the cost of adoption. I don't want pointless or arbitrary bincompat breakage, but I do want adopters who are willing to invest in the health of the ecosystem by keeping their dependencies up to date. This proposal rewards the opposite attitude.

I know this isn't a popular position, but I feel like a lot of these important decisions about the direction and priorities of the project are being made too quickly and without enough discussion, and I'd like for us to be sure that this is really what the community wants before putting even more constraints on ourselves.

[travisbrown]

(For what it's worth I also think this proposal combines too many different kinds of things—e.g. I'm like 👎👎👎👎👎 on permanently syncing Cats major releases to Scala versions, but only like 👎👎 on completely separate 2.13 source for 2.0.)

[djspiewak]

I'm also 👎 on this, for slightly different reasons.

In my experience, breaking upgrades are best ingested one at a time. Nothing makes a breaking change (like a major Scala upgrade) quite as bad as when everything has also broken at the same time. What I mean by this is that, for a given version of Cats that is compatible with a given version of Scala, it should be (if at all possible!) the case that that same version of Cats is also compatible with the next version of Scala, so that I can upgrade Scala by itself without worrying about the rest of my dependencies shifting at the same time.

Just look at the hell that has been wrought by the ScalaCheck decision to not publish 1.13 for Scala 2.13, particularly given that 1.14 also dropped support for 2.11. Let's not make more of that unless we absolutely have to. The best time for Cats (or any other library) to release a breaking change is mid-cycle on Scala upgrades (in other words, as far away as possible from being synced with Scala's breakage), and ideally the old version would be backport-published for the next major Scala release before obsolescence, just to ensure that the upgrade path is as smooth as possible.

Materially separate APIs for Scala version 2.n and 2.(n+1) is even worse, I fear. While this is technically possible, it really breaks the implicit contract regarding cross publication versioning that the entire community has adhered to. Namely, that a given major/minor version of a library will be identical on Scala 2.n and 2.(n+1) when cross-built for both. The reason this has been an implicit contract is because it makes Scala upgrades easier, which is an important thing to optimize for since Scala upgrades are the most breaking thing in our entire ecosystem right now, and therefore also usually the hardest thing to upgrade. If we break that contract, it will make it immensely difficult to use Cats across Scala versions. Even worse, because Cats is extremely close to the root of the dependency hierarchy, it will force all of the other intermediate libraries (i.e. any library that builds on Cats) to do the same thing, which effectively perpetuates the breakage and magnifies the impact. This is worse for everyone.

In my opinion, conditional compilation (i.e. separate source directories) should only ever be used for compatibility shims and bincompat tricks. Never ever for the effective public API. Never for semantics.

Regarding delaying 2.0 to squeeze in more changes… As exciting as some of these changes are, the whole ecosystem is waiting on Cats. Think of how annoying it is that we're all still waiting on ScalaTest. Delaying 2.0 would multiply that annoyance for pretty much everyone. Breaking more things in 2.0 would cause the upgrade to be even harder, and rapidly snowball Scala the 2.12 -> 2.13 migration into something approximating Python 3, or at the very least the Scala 2.7 -> 2.8 migration.

Speaking of the 2.7 -> 2.8 migration… That particular upgrade was utter hell for anyone using Lift. The reason it was hell was Lift did exactly what you're proposing: version 1.x was only available for 2.7, while version 2.x was only available for 2.8. You had to upgrade everything all at once, and the set of breaking changes was massive. This was on top of a very large set of breaking changes in Scala itself. I did the majority of the work for that upgrade for Novell (the company I was working for at the time), and it took me about 7 very frustrating months to get through everything and make it actually work, and that was on a codebase which was only about 30,000 lines of Scala. I'd really like to not make 2.12 -> 2.13 a repeat of that experience, or even worse given that the ecosystem is an order of magnitude more complicated than it was ten years ago.

In my view, the ideal solution would have involved releasing something in the Cats 1.x line for 2.13. This is obviously impossible due to ScalaCheck, and so the next best thing is to release 2.x as quickly as humanly possible, and keeping it as close as possible to 1.x. The only binary breakage should be a) that which is required by ScalaCheck (and transitive), and things that are absolutely urgent to squeeze in.

In general, I'm 👎 on the whole proposal.

[kailuowang]

just to be clear, @kailuowang , which of the 3.0 tickets would be candidate to be moved to 2.0?

@erwan
We haven't finalized the list yet. Here are some candidats

• Move Binary Incompatible instances and syntax into main traits Move Binary Incompatible instances and syntax into main traits for 2.0 #2415
• Use an abstract type member for the parallel type of the Parallel type class Use an abstract type member for the parallel type of the Parallel type class #2233
• Remove redundant Monad requirement when Parallel is provided Remove redundant Monad requirement when Parallel is provided #2180
• newtypes based data type (e.g. MonadTransformers, NonEmptyXXXs) Consider adding some form of newtypes to cats #1800 (this one can be break into smaller issues)

[kailuowang]

@travisbrown @djspiewak looks like the plan was not very clear in the detail and there is certainly some big devil there. I am going to spit out all the detail in the top post, which I believe will address most of your concerns.

[kailuowang]

I am closing this for now to avoid further confusion. Will restart the conversation once I have a clarified version.