Please commit to a 1.0 or list reasons preventing it

[Datseris]

Hi,

this package by now is used by lots of people, and probably also in other packages. The constant release of 0.x versions makes handling compatibility really unpleasant. At least for me in ClimateBase.jl the DimensionalData.jl has been the only main dependency constantly causing compatibility problems. This is because Julia treats 0.x releases as breaking, so every 3 months or so my research project files will not be updatable unless I make a PR in ClimateBase.jl to update the DimensionalData.jl.

Given the large amount of users and usecases, I believe stability is warranted and deserved even.

Why has there been no 1.0 yet? Can the lead devs provide a checklist of the reasons preventing this? Or, better for the users at least, maybe just commit to a 1.0 and only merge in breaking releases if they really warrant a huge ecosystem disruption?

The package is now in 0.29, meaning it had 28 breaking releases. I would argue, for the number of users and usecases of this package, this is really a lot. Perhaps it is time to acknowledge that there is a meaningful need for a long-term stable version.

[rafaqz]

@Datseris please take it easy with demands like this, a simple question would be fine. We work on this package in our spare time and honestly I am pretty burned out from issues written in this style.

Also I am only ever breaking things when adding new functionality that users have explicitly requested. Other people want breaks. So what do we do. We are trying pretty hard to make a great package here.

But, I'll attempt an explanation. Dimensional data.jl is pretty complicated package, and compared to Dataframes it's a new and unclear design space. Xarray in python continues to make small breaking changes, just with deprecations instead of semver to protect you.

We rarely break core functionality, but small technical breaks are almost unavoidable, especially for DimStack, DimSelectors etc, groupby and other things that a difficult to design right the first time. I just had to change similar in DimStack I wrote badly three years ago, not understanding the design space.

But we are doing some concrete things to reduce breaks.

• The new DimTree is included without semver guarantees so changes to it won't be breaking, and new features will be treated like this from now on until they are stable.
• There is a breaking branch where all breaking changes accumulate until we are ready for a release. This is reducing breaks a lot. But not totally. Let's aim for no more than twice a year?

We can probably do more. We could make a core package and move DimArray, Lookups and Dimensions into it, leaving other less stable things in DD.

But, TLDR xarray is still making small breaks after many years of development. We are similarly unlikely to never need breaks, unless we just stop working on the package completely. 1.0 won't change that

[rafaqz]

To discuss something concrete, I'll cover the breaking changes on the way #946

Multiple slack etc questions have asked why DimArray{NameTuple} doesn't work like a regular array in the Tables.jl interface. The answer is because it needs a specific override that no-one wrote before, and it really should work like a regular array. So this PR is a minor change, but technically breaking.

#839

This one is tiny (types are detected as categories), but still technically breaking. We would only make the change if paired with other more important things.

#876

This one is for the widely asked for (#874 #973 #865) combine method that uncovered inconsistency in similar on DimStack and DimArray that needed to be fixed for consistent behavior accross both types.

#903

This one closes issues from multiple user complaints that set is not currently good enough. And it isn't, but it will be pretty nice, and facilitate pretty nice declarative programming styles after this change.

#926

And probably, we will change metadata to be DataAPI.jl compatible, which is also slightly breaking. Rolling this change with the others will delay the release by a month or two, but may be worth it to reduce breaks.

These are the kinds of issues and feature requests that lead to breaking changes. To me most of these are pretty defensible and provide real benefits to users.

Its hard to be sure there are not a bunch more of these out there, they are certainly decreasing but there's a long tail.

[Datseris]

Thanks for the reply @rafaqz . I am sorry about my tone which was innapropriate or sounded demanding. To my defense, a sentence that starts with "please do something" is more of a plea for help than a demand. From my perspective, comes from how often my projects have compat problems that cause project-wide downgrades, sometimes associated with Makie, or some other fundamental package like Static/Struct arrays. Perhaps you are not aware of what kind of downgrades can be caused when DD.jl is not the "top-end" dependency, but rather a downstream dependency? This is not a problem of a particular breaking change such as the issue I opened in #976 . Rather this is a problem of the way Julia handles compatibility, which treats 0.x releases as backwards-incompatible, and the fact that DD.jl only changes its compatibilities to other packages in its latest releases. This last part is normal, but when the latest releases are often and "formally" breaking, it causes problems. For example, in an ongoing project, I just added ClimateBase.jl. This caused the downgrade StructArrays v0.7 -> v0.6.18. Give how "deep" in the dependency hiearachy this package is, this got me very worried and I had to fix it. Trying to re-establish compatibility meant that I had to change ClimateBase.jl compat to latest DD.jl - there was no other way. In another project, adding ClimateBase.jl caused downgrades in the prevous major version of the whole Makie ecosystem, due to deeper dependencies. In this scenario it was simply easier for me to just go with the previous "major" version of Makie in this other project instead of figuring out the deep dependency that was causing incompatibility.

In any case, I am sorry and I will be careful with my tone from now on. I was very frustrated. I appreciate you taking the time to reply despite the insult.

Given what you said though, there are still things to resolve. For example

But, TLDR xarray is still making small breaks after many years of development. We are similarly unlikely to never need breaks, unless we just stop working on the package completely. 1.0 won't change that

This is an incredible claim. I can't believe that a package used by most meteorological agencies in the world routinely disrupts the wide ecosystem. Wouldn't the huge community of both developers and users that depend on this package be outraged by this? I am sorry, but such a claim must be accompanied by strong proof, but you haven't provided it yet. Perhaps you can list where in the changelogs are these breaking changes ? I got curious after your post but the first page of the release notes I couldn't find anything obviously breaking.

We rarely break core functionality, but small technical breaks are almost unavoidable, especially for DimStack, DimSelectors etc, groupby and other things that a difficult to design right the first time.

Right, but because of the way compatibilities work, it doesn't matter if the change is small, or miniscule, or it doesn't affect practically anyone. If it is released as breaking it is breaking with respect to compat, and it means that everyone that depends on this package needs to (a) resolve the compats if they have to, and (b) check the changelog to see what the breakage was.

As I stated in the opening post, this has occurred so far 28 times, at least according to the version number. That's a lot, given the widespread usage. (I assume the package has widespread usage, as I am not a developer I haven't checked this)

We could make a core package and move DimArray, Lookups and Dimensions into it, leaving other less stable things in DD.

This to me sounds like a fantastic idea, provided this package would be a direct 1.0 release. What does Rasters.jl depend on? Only this core? This is one more reason to split pacakges: how large part of a functionality does a downstream package depend on?

You can also check how I handle releases in DynamicalSystems.jl. It is also a large package and it has many completely disparate parts. My approach (which is also disclosed to users https://juliadynamics.github.io/DynamicalSystemsDocs.jl/dynamicalsystems/dev/#Version-numbers) is that if a change is affecting only a single function, it won't trigger a major release of DynamicalSystems.jl. This would be a huuuuuuge disruption to thousands of users and it is simply not justified. The change of course is communicated in the changelog, and is also mentioned as breaking. It also helps that DynamicaLSystems.jl is composed of many sub packages with their own versions and release notes.

For example, in the Attractors.jl subpackage there is an upcoming "formally breaking" change: we are changing the default keyword argument of one function/data structure. Technically it's breaking. Will it affect anyone? It is likely that a small amount of people will be affected, can't know for sure. What I can say for sure that releasing a version v4.0 instead will affect everyone.

[rafaqz]

Yeah, Semver is a burden. especially this problem where small technical breaks that hardly effect anyone still need a breaking change. But discussing this with python people, the scramble they have to do to fix live bugs sounds worse to me.

This is an incredible claim. I can't believe that a package used by most meteorological agencies in the world routinely disrupts the wide ecosystem.

I never said anywhere that they disrupt the ecosystem. But they quite clearly make breaking changes:
https://docs.xarray.dev/en/stable/whats-new.html
https://docs.xarray.dev/en/stable/whats-new.html#id45
https://docs.xarray.dev/en/stable/whats-new.html#id6
https://docs.xarray.dev/en/stable/whats-new.html#id314 (edist: better links direct to breaks)

(actually multiple breaking changes in 2025 already, 1 in Jan and two in March - we haven't had a breaking change since November 2024)

Usually small breaks in the same order as the breaks here. They are less disruptive in some ways because semver isn't followed, but more disruptive in others as they can break real code without as much protection from the package manager. You are experiencing our disruptions the most as a dependent developer having to work around our semver enforcing package manager. And I get it, its annoying. But not massively, Makie has a similar affect on DD and it really does make core breaking changes.

if a change is affecting only a single function, it won't trigger a major release of DynamicalSystems.jl

It is of course a little contentious to break semver like this. I don't think we can do that here, and I have gotten in trouble for doing this elsewhere in the past.

This to me sounds like a fantastic idea, provided this package would be a direct 1.0 release. What does Rasters.jl depend on?

Yeah, its possible but probably a bunch of work to decide what is really stable. Unfortunately Rasters uses AbstractDimStack and AbstractDimIndices objects and causes some of the changes to them... so it would have to depend on DD proper.

[rafaqz]

StackOverflow answers have very strong opinions about not allowing minor breaks without semver bumps:
https://stackoverflow.com/questions/71813328/is-a-small-breaking-change-enough-to-do-a-major-version-upgrade-while-using-semv

Maybe too strong, but that does seem to be the general consensus, supported by a very strong MUST in the spec. Following that would at present make it pretty hard to have less than one or two breaks a year in DimensionalData.jl.

[Datseris]

StackOverflow answers have very strong opinions about not allowing minor breaks without semver bumps:
https://stackoverflow.com/questions/71813328/is-a-small-breaking-change-enough-to-do-a-major-version-upgrade-while-using-semv

This links to a discussion where there are 3 posts and a total of 10 upvotes. I don't think this is indicative for preferences. Besides, context matters. What software are we talking about? How large? What change are we talking about ? What does "breaking" mean? Code giving identical answer? Code giving slightly different answer? If we talk about a massive simulation framework like Agents.jl, what does "breaking" mean? Code that didn't error now errors? Or code that after doing a massive 20-file simulation, one of the 1000s observables has changed value by epsilon? "Breaking" is on the eye of the user more than it is on the eye of the developer. Yet, developers get to state when something is breaking. Without context it is hard to say. This is one of the main disadvantages of semver.

That is why I think semver has difficulties to deal with large, and complex software composed of many individual parts. I think Chris Rackauckas has said similar things for how they manage the diff eq ecosystem. For example, according to SemVer if you have packages re-exporting other packages, then any time the smaller package has a breaking change, the large must have as well. When you have many packages, this quickly gets out of control and you end up having major releases every couple of releases.

I am not opposing your decision, to be clear, just stating how I view things as a developer hoping to satisfy my users. For me the main avenue to solve this is communication. What does changes to my software mean for my users? Where can they find them? Then they can decide for themselves if they are affected. This excludes changes where I can confidently say "everyone is affected", in which case I release a new major version.

Compartmentalization is another way to deal with this. Splitting into smaller packages and including them into one large. But then if you re-export them, you haven't solved anything if following semver to the letter.

Perhaps a post in the Julia discourse is also warranted so we see how developers we know think of this. I am sure we will get more responses than this stack overflow post haha

[rafaqz]

Ok, but its still clear that your strategy is actually not widely accepted, but likely the opposite. And it will certainly have its detractors.

As for the other strategies you mention, we already don't pass on breaking changes in some dependencies, like Makie, as its just not worth it. Neither are changes to plots styling. Small floating point changes of course also wont be breaking.

But changing the return value of similar can actually break real functionality. Even if it never really made sense. It will likely break Rasters.jl, and we need protection against that.

[Datseris]

Ok, but its still clear that your strategy is actually not widely accepted, but likely the opposite. And it will certainly have its detractors.

Perhaps. But for whom are we talking about? and who are we caring about? Whether the strategy is "accepted" by other developers that don't use the library? Or people in stack overflow? Why would this matter...? All I know is that my strategy has been very successful with the people it targets: the users of DynamicalSystems.jl. I've never had someone complain about the release process of the library despite its "unconventional" strategy. I guess the success is also due to every "breaking" change that doesn't get a major release is instead having a deprecation (which, in this sense, DYnamicalSystems.jl is very much compatible with semver or other rigid systems...) . xarray also follows an unconventional strategy but it doesn't appear to be a big issue? I don't use it so I don't really know.

In any case, I've mentioned the release strategy I follow in DynamicalSystems.jl merely as an example. It was never my goal to imply that this strategy should be followed here, nor that it is the best strategy.

For me, the take-home message is that releases and breakages are more about communication to the people that care about them, than following a rigid system. In the end of the day, no matter what release process we follow, breakages will occur anyways because we can't foresee everything or test everything exhaustively. That's natural.

This conversation got me curious. I'll make some sort of poll or discussion on discourse to see what are the typical practices in various Julia ecosystems. I realize now I honestly don't know...

[tiemvanderdeure]

It seems to me like this issue is more of a broader discussion about the use of semver than about this particular package.

I understand that breaking releases can be frustrating to deal with, but I don't think it's entirely reasonable to put the blame on DimensionalData, is a 1.0 release a magic bullet.

Making breaking releases for even tiny breaking changes is (I would say) the accepted standard in Julia. Just an example I noticed recently is the 2.0 release of SpecialFunctions (see the discussion in JuliaMath/SpecialFunctions.jl#297), but there are many more.

There are also lots of well-established and much-used packages out there that do not have a 1.0 release, even though breaking releases are rare. Of course StatsBase is one of them (on 0.34). Others are Makie (on 0.22), Distributions (on 0.25), CategoricalArrays (0.10), and there are many more.

There are other packages that take the strategy of releases a 1.0 faster, but then they still make breaking changes. ModelingToolkit is on version 9, and there are lots of other SciML packages like this. If DD were to release a 1.0, then we'd probably need a 2.0 at some point.

I also think you might be exaggerating a little bit on the number of breaking releases DD has. I just looked for the last year and we had 0.27 in April, then 0.28 in September, then 0.29 in November. So 29 is a high number, but the breaking releases have been slowing down a lot, and I doubt you've had to update your package 28 times.

I've also personally had some problems with compat bounds in DD (in particular the 0.34 StatsBase requirement), but I think it's just the price we pay for the package system.

Maybe we could look at some of the compat bounds and see if some of them could be wider. Otherwise I think the breaking releases will keep slowing down as this package matures, the bugfixes will keep coming, and maybe one day we'll end up with a version that's as ridiculous as the one Distributions has (currently v0.25.119)