Stacked Borrows UB in MatrixViewMut sub-view creation

[pandashark]

What's happening

When you create a sub-view from a MatrixViewMut (e.g. via rows_range_mut), Miri flags UB under Stacked Borrows. The culprit is &mut self.data in the matrix_view_impl! macro — it performs a Unique retag that kills sibling pointers derived from the same parent ArrayStorage.

Found this while digging into #1520. The axcpy_uninit fix in #1587 addressed the immediate problem, but this one sits deeper in the view infrastructure.

Where it happens

The matrix_view_impl! macro (src/base/matrix_view.rs, around line 819) generates rows_range_mut and friends. They go through &mut self.data on ViewStorageMut, which under Stacked Borrows retags with Unique over the entire pointer provenance — wiping out any sibling borrows from the same allocation.

In practice this fires whenever columns_range_pair_mut is followed by sub-view ops on the resulting views:

let (mut col_a, col_b) = matrix.columns_range_pair_mut(0, 1..2);
let sub = col_a.rows_range_mut(..);
// ^^ &mut self.data retags Unique, invalidating col_b's provenance

Every decomposition that uses columns_range_pair_mut + sub-views hits this path.

Does it matter?

• Stacked Borrows: yes, UB
• Tree Borrows: no violation — doesn't invalidate siblings on mutable retag
• Real-world: no known miscompilations

Repro

Minimal reproducer — any SMatrix decomposition triggers this:

fn main() {
    let mat = nalgebra::Matrix2::<f32>::identity();
    let _ = mat.cholesky();
}

# Stacked Borrows (Miri default) — fails:
cargo +nightly miri run

# Tree Borrows — passes:
MIRIFLAGS="-Zmiri-tree-borrows" cargo +nightly miri run

Note: requires #1587 to be merged first, otherwise Miri will hit the axcpy_uninit slice aliasing UB before reaching the view system issue.

Possible fixes

1. Skip &mut self.data in sub-view creation — go through the raw pointer directly without an intermediate mutable reference to the storage
2. Restructure ViewStorageMut — hold a raw pointer instead of a borrow, so sub-view creation doesn't retag
3. Target Tree Borrows — document that nalgebra assumes Tree Borrows semantics and move on

(3) is the pragmatic call given Tree Borrows is where things are headed, but (1) or (2) would make it sound under both models.

Related

• Undefined Behavior in some decomposition methods on SMatrix #1520 — original Miri report
• fix: resolve Miri UB in SMatrix decompositions #1587 — axcpy_uninit fix

[Alouzious]

Hi @pandashark I'd like to work on this issue. Could someone assign it to me?

I've gone through the issue description and understand the root cause, the &mut self.data retag in matrix_view_impl! invalidating sibling pointers under Stacked Borrows. I'm planning to explore Option 1 (bypassing the intermediate &mut and going through the raw pointer directly) as a starting point.
Happy to discuss the approach before submitting a PR.