Add many-way coauthor-corroborated topic split to predict

[atalyaalon]

Alternative to #74. Same goal (split over-merged same-name clusters using embedding topics gated by coauthor disjointness) but handles many identities per cluster, not just a 2-way split.

Motivation

Same-name authors who work in different fields — e.g. Erick Matsen IV (computational biology) vs his father Frederick Matsen III (orthopedic surgery), or a "N. Harel" block holding a dozen unrelated people — get over-merged. Their papers' SPECTER embeddings sit close in absolute cosine (most academic titles are ~0.7–0.9 apart), so the pairwise model treats them as compatible and average-linkage fuses them. The separating signal is relative: each person's papers form a tight embedding sub-cloud with a distinct centroid.

Change

Post-clustering pass (s2and/topic_split.py): over-segment each cluster's embeddings with agglomerative cosine clustering (no fixed cluster count, so it adapts to however many identities are present), then re-merge sub-groups that share ≥2 full-name coauthors. Genuine single authors carry recurring coauthors across their own topics and re-merge; distinct same-name people have disjoint networks and stay apart. Full names (not initial+surname blocks) avoid common-name collisions like "j lee" spuriously gluing distinct people. Affiliation is not used — shared geography tokens cause false overlap. Only ever splits within a cluster, never merges, so it cannot lower recall on already-separated clusters.

Wired into the full-block predict() path, default on, configurable via Clusterer.topic_split_* attributes, read through getattr so older pickled bundles fall back to defaults.

Results (s2and-mini, clean integrated predict, OFF vs ON)

dataset	F1 OFF	F1 ON	Δ
arnetminer	0.894	0.895	+0.001
inspire	0.979	0.973	−0.006
kisti	0.958	0.956	−0.002
pubmed	0.940	0.936	−0.004
qian	0.955	0.949	−0.006
zbmath	0.966	0.966	0
mean	0.9487	0.9458	−0.0028

Pathological blocks (best-cluster purity, recall held):

block	OFF	ON
f_matsen	0.49	0.71
n_harel	0.03	0.58 (multiple N. Harel identities separated)

vs #74 (2-means + silhouette)

	#74 (2-way)	this (many-way)
f_matsen	0.70	0.71
n_harel	0.03 (unchanged)	0.58
s2and-mini mean B3 F1	−0.0024	−0.0028

Many-way fixes the many-identity case (#74's 2-means silhouette gate never fires on the n_harel 268-blob) at marginally higher benchmark cost. Pick one.

Scope / notes

• Full-block predict() only — not predict_incremental.
• tests/test_topic_split.py added (6 tests).
• Unrelated pre-existing failures (rust-extension version, stale v1.2 eps fixture) are not touched here.

[sergeyf]

I tried this approach on my labeled data in some real blocks:

Smallest full big-block by dry-plan pair count was s_gupta:

• s_gupta: 112,691,163 pairs
• j_smith: 134,686,084
• a_khan: 137,178,268
• larger: a_silva, s_park, h_wang, s_lee

I ran s_gupta at eps=0.65 with baseline plus PR #75-style topic split. Run completed with EXIT_CODE=0 in about 68.5 minutes.

Key weight_query_class_balanced result:

variant	positive_recall	false_split_rate	negative_separation	false_merge_rate	accuracy
baseline	0.883749	0.116251	0.953882	0.046118	0.917436
topic_split	0.850577	0.149423	0.958697	0.041303	0.902511
delta	-0.033172	+0.033172	+0.004815	-0.004815	-0.014926

So on the smallest block, #75-style splitting does reduce false merges, but it gives up noticeably more recall than it gains in false-merge reduction.

Topic split stats:

• input clusters: 3,404
• output clusters: 4,078
• input clusters split: 370
• splits made: 674
• changed signatures: 3,316
• topic split time: 4.06s

Outputs are in [eps_overall.tsv](scratch/topic_split_eps065_smallest_block/s_gupta_20260629_1015/eps_overall.tsv), with run log at [eps_sweep.log](scratch/topic_split_eps065_smallest_block/s_gupta_20260629_1015/eps_sweep.log).