`metric="cosine"` and `metric="arccos"` are in `FAST_METRICS` but cannot be dispatched — `ValueError: Unrecognized metric`

[matteodellamico]

[LLM usage disclaimer: I found the bug with Claude Opus and used it to draft the report; I verified it and edited the report manually.]

cosine and arccos are the only two metrics added to FAST_METRICS by hand, and they are exactly the two that algorithm="best" cannot route anywhere. Any call with either metric raises ValueError: Unrecognized metric 'cosine' instead of clustering.

Reproducer

import numpy as np, hdbscan

X = np.random.default_rng(0).random((200, 8))
hdbscan.HDBSCAN(metric="cosine", min_cluster_size=5).fit(X)

ValueError: Unrecognized metric 'cosine'

Same for metric="arccos". Reproduced on the released 0.8.44 and on current master (hdbscan/hdbscan_.py unchanged in the relevant region as of 2026-08-20).

Cause

hdbscan_.py:

FAST_METRICS = KDTREE_VALID_METRICS + BALLTREE_VALID_METRICS + ["cosine", "arccos"]

and in the algorithm == "best" branch:

if issparse(X) or metric not in FAST_METRICS:
    ... _hdbscan_generic ...          # cosine is IN FAST_METRICS, so skipped
elif metric in KDTREE_VALID_METRICS:
    ...                               # cosine is not
else:  # Metric is a valid BallTree metric
    ...                               # cosine is not — and this branch assumes it is

Membership in FAST_METRICS is used to mean "this metric does not need the generic path", so listing cosine/arccos there routes them past _hdbscan_generic — the only path that could actually handle them — and into a branch whose comment asserts a BallTree metric. BallTree then rejects it:

>>> "cosine" in hdbscan.hdbscan_.FAST_METRICS
True
>>> "cosine" in hdbscan.hdbscan_.KDTREE_VALID_METRICS
False
>>> "cosine" in hdbscan.hdbscan_.BALLTREE_VALID_METRICS
False

So the two metrics are simultaneously declared fast and supported by no fast path.

Impact

Cosine is the standard choice for text, embeddings and any L2-normalised representation, so this removes a common use case. The failure is at least loud rather than silent, but the error names the metric rather than the dispatch, which sends users looking in the wrong place — the metric is recognised by the library, it just cannot be reached.

The usual workaround — L2-normalise the rows and pass euclidean, since d_euclidean = sqrt(2 · d_cosine) on the unit sphere — is not quite a substitute, which is worth stating precisely because it is easy to assume it is.

The map is monotone increasing, so it preserves the ordering of distances, and therefore the k-nearest neighbours, and therefore the core distances (core_euc = sqrt(2 · core_cos)). Since max commutes with a monotone map, mutual reachability transforms the same way, and Kruskal/Prim depend only on the ordering of edge weights. The MST edge set and the whole single-linkage hierarchy are therefore identical; only the merge heights are rescaled.

EOM is where it stops. compute_stability accumulates

stability[parent] += (lambda_ - births[parent]) * child_size

with lambda = 1/d — a weighted sum of differences of lambda. That makes the invariance condition exactly affine: under lambda -> a*lambda + b the offset b cancels inside every (lambda_ - births) term, and the scale a > 0 multiplies every cluster's stability equally, so the comparison EOM actually makes — a node against the sum of its descendants — is unchanged. Affine rather than linear is the point: an offset is harmless, and it is the curvature that does the damage.

1/sqrt(2d) is not affine in 1/d, so the stability sums are reweighted non-uniformly across the tree and the selection can move. cluster_selection_epsilon is likewise in distance units and would need transforming.

Measured, with algorithm="generic" so both metrics run the same code path: over 80 configurations spanning varied-density blobs, nested core-plus-halo, and heavy-tailed data, 3 gave different flat clusterings, the worst at 0.816 AMI agreement with 3 clusters against 2. On data with clean, unambiguous structure the two agree exactly (eom_invariance_easy.py, 6/6 identical). So the workaround is safe when the
clustering is obvious and silently is not when the selection is marginal — which is precisely when a user is relying on the tool's judgement.

Suggested fix

Drop the two metrics from FAST_METRICS, which has the same effect through the existing first branch and removes the contradiction at its source.

For comparison, sklearn.cluster.HDBSCAN (1.9.0) handles this correctly: algorithm="auto" with metric="cosine" succeeds by falling back to brute force, and the explicit tree algorithms raise a message that names the real problem and the remedy —

ValueError: cosine is not a valid metric for a KDTree-based algorithm.
            Please select a different metric.

Environment

• hdbscan 0.8.44 (also checked against master, 2026-08-20)
• Python 3.14, numpy 2.5.0, scikit-learn 1.9.0, Linux x86-64