Skip to content

Fix tied nearest neighbors merging a clade twice - #356

Merged
josevalim merged 3 commits into
elixir-nx:mainfrom
RicardoSantos-99:fix-hierarchical-tied-chain
Aug 22, 2026
Merged

Fix tied nearest neighbors merging a clade twice#356
josevalim merged 3 commits into
elixir-nx:mainfrom
RicardoSantos-99:fix-hierarchical-tied-chain

Conversation

@RicardoSantos-99

@RicardoSantos-99 RicardoSantos-99 commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Hierarchical.fit can merge a clade twice when nearest neighbor distances tie. The result is a malformed tree: a clade ends up as the child of two different parents, the sizes stop adding up, and the root no longer gathers every point.

The nearest neighbor chain is only guaranteed to terminate on a mutual pair while the distance along it strictly decreases, and that is also what stops it from ever revisiting a clade it already holds. Ties break the guarantee. The chain ended only when argmin happened to return the previous entry, so on a tie it could walk past that entry and back onto a clade already on the chain. The duplicate then outlives the merge that consumes its other occurrence and is merged a second time, after it is already dead.

The chain now ends whenever the previous entry is a nearest neighbor of the tip, not only when it is the one argmin returns, which restores the strict decrease. Comparing the distances alone would miss a NaN, which is never equal to itself, so either condition ending the chain is enough. Writing past the end of the chain buffer is guarded too, since Nx.indexed_put clamps an out of range index rather than raising.

Introduced in #355. Measured over 200,000 random fits on integer coordinates, where distances tie constantly: 38 malformed trees before, 0 after. The regression test fails on the current implementation.

linkage: :ward also raises on the binary backend, and is fixed here too. Merged clades are masked out of the matrix rather than blanked, so their rows keep stale values that ward's update still reads, and its Lance-Williams term can come out negative for a dead column. Those columns are never used, but the square root of a negative raises where EXLA gives a silent NaN, so the term is clamped at zero. Also introduced in #355: 17 crashes in 600 random fits before, 0 after.

The chain guard also covers a third case from #355: an infinite coordinate puts both infinity and NaN in the distance matrix, the chain then extends without ever finding a mutual pair, and writing past the end of its buffer raised index 4 is out of bounds on every linkage. It now reports the merges it could not make, as the moduledoc describes.

The chain only cycles back on its immediate predecessor while the distance
along it strictly decreases. Ties break that: argmin can return a different
clade that is tied for nearest, the chain walks back onto an entry it already
holds, and the duplicate outlives the merge that consumed it. It is then
merged a second time, after it is already gone, so a clade ends up under two
parents, the sizes stop adding up, and the tree never closes over every point.

End the chain whenever the previous entry is a nearest neighbor of the tip,
not only when it is the one argmin returns.
…tive

Merged clades are masked out of the matrix rather than blanked, so their rows
keep stale values that ward's update still reads. The term can then come out
negative for a dead column, which raises on the binary backend and is a silent
NaN under EXLA.
An infinite coordinate puts both infinity and NaN in the distance matrix, so
the chain can keep extending without finding a mutual pair. Walking past the
end of its buffer raised instead of reporting the merges it could not make.
@josevalim
josevalim merged commit 879b248 into elixir-nx:main Aug 22, 2026
2 checks passed
@josevalim

Copy link
Copy Markdown
Contributor

💚 💙 💜 💛 ❤️

@RicardoSantos-99
RicardoSantos-99 deleted the fix-hierarchical-tied-chain branch August 22, 2026 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants