[FEATURE] Affiliation normalization / fuzzy clustering — re-enable Author Affiliation facet

[SteveLLamb]

Summary

Normalize / fuzzy-cluster the ~6,936 distinct raw affiliation strings now in the registry (post-#1196) down to a canonical-name index of ~1,000–2,000 actual institutions, then re-enable the Author Affiliation facet on the doc-list browse.

Rationale

PR #1196 backfilled authors[].affiliation from _source/SMPTE/HIGHWIRE/ NLM <aff> blocks across ~9,800 SMPTE journal-article docs. The raw strings are verbatim from the source XML — every spelling/punctuation variant survives:

• "RCA Manufacturing Co., Camden, N. J."
• "RCA Manufacturing Co., Camden, N.J."
• "RCA Manufacturing Co., Camden, NJ"
• "RCA Manufacturing Co."

→ 4 facet buckets for what's clearly one institution. Multiplied across the corpus this lands the affiliation facet at ~7k distinct buckets — unusable as a picker (even ~500 would be too many).

#1196 ships with the affiliation facet disabled ([build.search-index.js], [docList.js]) — search index + URL filter still work against the raw strings, but the picker UI is hidden until this issue is resolved.

Implementation thoughts

Strategy: display-time normalization, source verbatim

Keep authors[].affiliation as the raw source string per doc (provenance — never lose what the XML said). Add a canonical-name layer the facet builder uses to bucket.

Normalization steps (cheap to expensive)

1. Whitespace + punctuation normalize (free — Levenshtein 0 collapses): strip trailing ., normalize internal whitespace, fold Co. ↔ Co ↔ Company, Inc. ↔ Inc ↔ Incorporated, Dept. ↔ Department, etc.
2. Strip city/state/country trailing tokens for short-form canonicalization: "RCA Manufacturing Co., Camden, N. J." → "RCA Manufacturing Co.". Keep the long form for display when the short form has multiple sources, but bucket on the short.
3. Acronym expansion lookup (refMap-style): "MIT" ↔ "Massachusetts Institute of Technology", "USC" ↔ "University of Southern California". Small curated map.
4. Levenshtein-based fuzzy match within publisher-clustered subsets to catch typos / minor spelling drift. Threshold tuned per corpus size.
5. Optional ROR/Wikidata cross-reference — too heavy for v1 but worth knowing the data exists.

Where the normalization lives

• New helper src/main/lib/affiliationNormalize.js — pure function from raw → canonical key + display string.
• build.search-index.js re-enables facets.affiliations bucket by emitting canonical → count (raw strings preserved per-doc on idx row for the URL filter and full-text search).
• facets.affiliationLabels map emitted alongside the bucket so the picker shows the canonical name, not the raw form.
• docList.js re-enables the picker entry — same facetLabel() pattern as ICS codes.

Picker UX

Even after normalization, ~1,500 buckets is still a lot for a vertical accordion. Worth pairing with a min-count threshold (only show institutions with ≥5 docs in the picker; long-tail still URL-filterable). Mirrors the existing min-bucket-threshold pattern on keywords if there is one — if not, this issue adds it.

Acceptance

• Affiliation normalization helper lands with whitespace+punctuation rules (the cheap win)
• Acronym map curated for the top ~50 institutions in the SMPTE corpus (Kodak, RCA, Bell Labs, Sony, NHK, BBC, etc.)
• Levenshtein fuzzy cluster pass against the normalized strings; threshold tuned
• Distinct canonical institutions drops from ~7k → ~1–2k
• Picker re-enabled with min-count threshold; renders ≤500 buckets typically
• Search / URL filter still works against raw and canonical forms
• Sample-verify: ?f.affiliations=RCA+Manufacturing+Co. lands on every variant doc

Related

• #1196 — authors object-form migration + HIGHWIRE affiliation backfill (where the raw strings come from)
• src/main/scripts/build.search-index.js — facet builder; affiliation bucket commented out pending this work
• src/site/js/docList.js — picker; affiliation entry commented out pending this work

[SteveLLamb]

Adding scope note from PR #1213 follow-up:

Affiliation rendering was set to plain text on both doc detail page and doc index card preview (commit 2c8e55a) — no click-to-filter link wrapping. The raw 6,936-string affiliation space made the URL filter both unhelpful (zero/one match per click) and brittle (commas + spaces in the raw string round-trip badly through ?f.affiliations=... even with encodeURIComponent).

When this issue lands a canonical bucket (~1–2k institutions), the link-back can return cleanly:

• Re-enable the affiliation chip / card link with href="?f.affiliations={{canonicalKey}}" against the normalized canonical key (not the raw string)
• The picker re-emerges driven by the same canonical buckets
• Search-as-text continues to work against the raw per-author strings as a fallback

Tracking it here so we don't lose the UX piece if/when the normalization is done.