Is there a plan for stable/1.0 version, or at least LTS?

[stephanemagnenat]

Hello,

nalgebra is becoming a key foundation for advanced domain-specific crates, such as rapier or cv. However, because nalgebra has no stable version, it makes it hard to mix crates that use a different version of nalgebra. If we imagine to build an augmented-reality game that uses both rapier (depending on nalgebra 0.32) and cv (depending on nalgebra 0.30.1), we face a mismatching set of nalgebra versions.

I understand that, being an advanced math library, nalgebra wants to keep up to date with modern Rust features and I can very well imagine why the authors do not feel it is ready for 1.0 version. Nonetheless, given the widening of its adoption as one of the core math library in the Rust ecosystem, as a user the variety of nalgebra versions third party crates depend on is a clear source of friction.

So maybe there is an intermediate way, such as a LTS (or rather, Mid Term Support) version, so that one can provides PR towards ecosystem crates in order to let them converge to the same nalgebra version, at least for a while. The goal would be that while they do not depend all the time of the same version, there are at least some times they do.

[sebcrozet]

Hi!

I don’t see nalgebra reach the version 1.0 as long as the more fundamental math crates it depends on (in particular rand and num) don’t reach 1.0.

I’m not completely sure of the benefit of a mid-term-support version. A mid-term-support version would imply merging fixes to older version of the crate, which can become a significant maintenance burden. Updating a crate to the latest nalgebra is rarely painful now that it as reached a somewhat stabler API state (until the next big rustlang features like specialization and complete const-generics get stabilized). So it feels cleaner for the community to just update to the latest version.

[Ralith]

Crates which require a stable API should not expose nalgebra in their interface. In general, exposing third party crates in your public API always makes stability very difficult. The purpose-built mint crate is a good foundation for a more stable low dimensional linear algebra interface, which could e.g. painlessly have different versions of nalgebra on either side.

[Andlon]

I think this is easily the biggest problem with nalgebra right now, and I think we should consider alternatives to the current status quo. I think @sebcrozet and @Ralith both severely underestimate the issue here, and I'll explain why.

First, @sebcrozet suggests that it's cleaner for the community to "just update". The problem is that you need all crates to agree on the same nalgebra version, which is a significant hurdle for downstream applications or libraries that rely on several crates that depend on nalgebra. Although you can send a PR to the crate's repository, there's no guarantee this PR will be merged in a timely manner, or at all. In the meantime, you might have to work with a separate fork. If you are also publishing a library, however, then you cannot just rely on some patched version in a git repository. You'd have to publish your fork, which you really don't want to do.

This is not just theoretical; I've run into these kind of issues several times, and it's been pretty painful. Although thankfully in several cases I had the luxury of having control over several of the crates in question.

With this in mind, @Ralith is completely correct that in order to avoid this situation, it's best not to use nalgebra in the public API. This might work out if you're only working with a small subset of nalgebra's data types. However, if your crate needs to work with a substantial subset of nalgebra, such as both fixed-size and dynamic types, and possibly geometric types too, replicating the necessary API surface is not only terribly laborious to the point that it might be practically infeasible, it also leads to a drastic loss in usability of your API compared to using native nalgebra data types directly.

I don't think we can comfortably reach 1.0 any time soon. In addition to what @sebcrozet mentioned, there are too many design issues that have yet to be addressed for this, in my opinion. I also don't think the project has the resources (which are stretched way too thin already) to maintain some kind of LTS scheme.

Instead, what I think might be feasible, is to split up nalgebra into several sub-crates, which ultimately depend on an nalgebra-core crate which contains the core data structures and std::ops implementations etc. This means that a substantial portion of nalgebra's API would still have to reside inside nalgebra-core, but hopefully this would be the parts of the API which are evolving the most slowly, hopefully only very rarely requiring breaking changes. Crate authors could then only expose types from nalgebra-core in their public API but be free to use the remaining API from nalgebra internally. This allows multiple different versions of nalgebra to interoperate as long as they interface through data types from the same version of nalgebra-core.

This means that much of nalgebra's API would have to be moved to extension traits, which has its pros and cons. I'd like to mention here that one possible benefit of this is faster compilation, since independent sub-crates that depend on nalgebra-core but not each other would be able to compile in parallel.

I don't know if this is feasible, practical, or worth the trade-offs. It would also involve a very substantial amount of work that I'm not sure if anyone is willing to take on. But it's something I think we should consider and discuss.

[sebcrozet]

First, @sebcrozet suggests that it's cleaner for the community to "just update". The problem is that you need all crates to agree on the same nalgebra version, which is a significant hurdle for downstream applications or libraries that rely on several crates that depend on nalgebra. Although you can send a PR to the crate's repository, there's no guarantee this PR will be merged in a timely manner, or at all. In the meantime, you might have to work with a separate fork. If you are also publishing a library, however, then you cannot just rely on some patched version in a git repository. You'd have to publish your fork, which you really don't want to do.

I might be missing something but I’m having a hard time seeing the difference between updating from 0.1.0 to 0.2.0 and updating from 1.0.0 to 2.0.0. All the issues you mentioned will be the same. Changes that require a version bump from nalgebra 0.1 to 0.2 are breaking change. Changes that require a version bump from nalgebra 1 to 2 are the same: breaking changes.

So nalgebra being at version 1 or 0.1 doesn’t change anything to the frequency at which the aforementioned version mismatch problems can happen.

This means that much of nalgebra's API would have to be moved to extension traits, which has its pros and cons.

In the early days of nalgebra, a large piece of its API was based on extension traits. That was widely disliked by the community; so I wouldn’t take that route again. Though, admittedly, compilers and editors were way less smart than they are now. So perhaps that would be less painful today.

[Andlon]

I might be missing something but I’m having a hard time seeing the difference between updating from 0.1.0 to 0.2.0 and updating from 1.0.0 to 2.0.0. All the issues you mentioned will be the same. Changes that require a version bump from nalgebra 0.1 to 0.2 are breaking change. Changes that require a version bump from nalgebra 1 to 2 are the same: breaking changes.

So nalgebra being at version 1 or 0.1 doesn’t change anything to the frequency at which the aforementioned version mismatch problems can happen.

I agree with everything you've said here, I'm absolutely not in favor of publishing 1.0 any time soon. I'm a bit confused, I think I must have explained myself poorly since this seems orthogonal to what I tried to communicate. To take a more concrete example, say you have a crate mycrate that depends on crates a, b and c, and they all depend on nalgebra:

mycrate: nalgebra 0.32
   - a: nalgebra 0.32
   - b: nalgebra 0.32
   - c: nalgebra 0.32

Everything is good. Now, let's say there's a new breaking change of nalgebra, which we'd like to start using. In the meantime, crate a upgrades as well. So now we have:

mycrate: nalgebra 0.33
  - a: nalgebra 0.33
  - b: nalgebra 0.32
  - c: nalgebra 0.32

since all of these crates use nalgebra for their data structures in their public API, we can't get this to compile anymore. So we're stuck. We send a PR to crate b and c to upgrade, but it turns out that either they're not responsive or for some reason not able to upgrade (yet). Time passes, in which we still cannot use nalgebra 0.33, and there's a new nalgebra version 0.34. Now suddenly c upgrades but the maintainer of a is taking a break. So we have

mycrate: nalgebra 0.34
  - a: nalgebra 0.33
  - b: nalgebra 0.32
  - c: nalgebra 0.34

so now we're spread across 3 different versions, with no solution in sight.

This is a tough problem that we can't outright fix as long as we need to make breaking changes (and there will be more), but maybe there's a way we can reduce the impact of breaking changes, hence why I brought up my suggestion of exploring the option of separating out a nalgebra-core crate.

In the early days of nalgebra, a large piece of its API was based on extension traits. That was widely disliked by the community; so I wouldn’t take that route again. Though, admittedly, compilers and editors were way less smart than they are now. So perhaps that would be less painful today.

I know, and I remember. There were two problems with this, as I recall:

The first one is tooling, as you said. There was no good auto-import of traits. But we have that now, in the form of rust-analyzer and IntelliJ Rust.

But I remember the main reason rather being that nalgebra in those days using very granular traits. For example, there was a separate trait for Dot, for Transpose and so on. What I'm suggesting is to still leave many things as direct impls, and use "larger" extension traits. That said, due to the complexity of nalgebra's generics, it might be that many traits would still have to be granular in order to replicate the often complicated type bounds currently used in impl blocks. I'm not sure. I must also say that I'm not convinced myself of this path, it would have to be explored!

[Ralith]

However, if your crate needs to work with a substantial subset of nalgebra, such as both fixed-size and dynamic types, and possibly geometric types too

Perhaps it would be worth exploring a way to abstract over these, too, in a stable interface crate similar to mint? Explicit conversions avoid trait problems, and while not pleasant are a relatively small price to pay for stability.

Instead, what I think might be feasible, is to split up nalgebra into several sub-crates

For comparison, tokio had a structure like this pre-1.0 but dropped it due to the high maintenance load it incurred.

There was no good auto-import of traits. But we have that now, in the form of rust-analyzer and IntelliJ Rust.

I am uncomfortable with design decisions that rely on fragile heavyweight development tools to be usable.

[stephanemagnenat]

A question: In the development of nalgebra, does it happen that sometimes breaking changes bring big new features or cleanups but other times they are minor but necessary for the internal evolution of the library?

If so, my idea of "Medium Term Support" was that when the authors of nalgebra release a new version that they deem important for the ecosystem, they could mark it as such in order to inform the user libraries that it is worth updating to that version of nalgebra. That way, libraries using nalgebra would not need to update their dependency to nalgebra too often, which in the end is the source of the issue we discuss here.

[Andlon]

@stephanemagnenat: Perhaps I don't fully understand your proposal, but I don't really see how to do this in practice. It's up to the user to decide on what they think is important enough to warrant an upgrade, I don't really see a reasonable way for us to make this decision. Releases basically always come with something useful to users, I can't think of a release that was only "necessary for the internal evolution of the library" 🤔

[Andlon]

I think that, generally, what might help would be to triage issues and clearly document outstanding design issues that would need breaking changes, and actively work towards resolving these issues (perhaps tracked by a separate milestone). After that point, we could hope that breaking changes would be much less frequent.

The bigger problem here is that, I think, nobody currently has the capacity to take on this kind of work.

[stephanemagnenat]

The bigger problem here is that, I think, nobody currently has the capacity to take on this kind of work.

I agree with your triage idea and this assessment about capacity. I believe that given the importance of nalgebra for the community there is a need of some funded project management work. I have no idea if the funded companies who are actively using nalgebra could help here, but maybe it is worth asking?

[Andlon]

Perhaps it would be worth exploring a way to abstract over these, too, in a stable interface crate similar to mint? Explicit conversions avoid trait problems, and while not pleasant are a relatively small price to pay for stability.

We've kind of loosely explored this in the past on Discord, and I/we couldn't really come up with anything acceptable 🤔 I think the problem is that you need exactly nalgebra's model in order to be able to convert correctly in a generic context (in part because we have no specialization), in which case it's pointless since you anyway need to be 1:1.

Perhaps you could move only those core traits/types (like Dim / DimName, allocators and so on) to a separate crate that nalgebra would also rely on... Not sure if this is feasible, and if it would be enough. Maybe worth thinking about though.

For comparison, tokio had a structure like this pre-1.0 but dropped it due to the high maintenance load it incurred.

Interesting. I don't know the particular case of tokio, but splitting up a project into sub-crates is an established (although somewhat annoying) way to reduce compile times. I think in nalgebra's case, there may be some natural boundaries. For example, matrix decompositions would be a natural candidate for a separate crate. Similarly, geometry stuff could probably also reside in a crate, and so on.

But yeah, it does add a non-negligible maintenance burden over a monolithic crate, this is true.

I am uncomfortable with design decisions that rely on fragile heavyweight development tools to be usable.

I was not clear here. You'd collect everything into a prelude and just type use nalgebra::prelude::*. Preludes work well these days and used by many libraries.

[Andlon]

Here's a variant of the preceding ideas that may perhaps be more feasible to implement. Let's for the moment only focus on the issue of having the Matrix type be more "stable".

nalgebra::Matrix currently looks like this:

#[repr(C)]
pub struct Matrix<T, R, C, S> {
    pub data: S,
    _phantoms: PhantomData<(T, R, C)>,
}

Now, to begin with, I was never sure why data was public. I think perhaps it should not be, but I guess we can ignore that for now. Imagine that we replace the definition of Matrix with

#[repr(transparent)]
pub struct Matrix<T, R, C, S> {
    inner: nalgebra_core::CoreMatrix<T, R, C, S>,
}

where nalgebra_core::CoreMatrix is exactly the current nalgebra::Matrix. nalgebra-core would contain everything needed for the definition of CoreMatrix, including presumably the allocator and dimension machinery and so on.

However, CoreMatrix would not have any methods implemented, nor any operators. It exists strictly only to represent the data, not to operate on it. Methods and ops are all implemented on nalgebra::Matrix, just like now. But, crucially, nalgebra can evolve separately from nalgebra-core, and libraries that use nalgebra::Matrix in their public API can instead use nalgebra-core::CoreMatrix, and nalgebra would have methods to convert to/from CoreMatrix. Hopefully then we could bring nalgebra-core to a state where breakage is very rare. From a user of nalgebra, this has no impact - I believe everything would work just like they do now.

There may be some obvious problems with this proposal that I have missed, and I have not yet had the chance to think it through in detail, but it seems like a potentially interesting direction to take, so I wanted to bring it up here.