Disorder browser search lacks relevance ranking

[kevinschaper]

Disorder browser search lacks relevance ranking

Problem

Searching for a disorder by name in the browser app (app/index.html) returns results in alphabetical order rather than by relevance. This means exact or near-exact name matches can be buried below unrelated disorders that happen to mention the search term somewhere in their descriptions, phenotypes, or other fields.

Example: Searching "Parkinson" returns Parkinson's Disease as the 4th result, behind other disorders that merely reference "parkinson" or "parkinsonian" in their pathophysiology or phenotype descriptions.

Root cause

The current search implementation is boolean-only:

• Tokenizes all searchable fields into a flat Set per record
• Checks whether all query tokens appear as substrings of any record token
• Matching records are then sorted alphabetically by name (or by date if selected)

There is no scoring or ranking — a match in the name field is treated identically to a match buried in a phenotype list or description paragraph. There is no prefix match advantage, no term frequency weighting, and no field boosting.

Expected behavior

• Searching "Parkinson" should return Parkinson's Disease as the Bump actions/checkout from 4 to 6 #1 result
• Name matches should rank above matches in descriptions or other fields
• Prefix matches (e.g., "Parkin") should rank the matching disorder first
• Multi-word queries like "Sickle Cell" should rank Sickle Cell Disease first
• When a search query is active, results should default to relevance sort rather than alphabetical

[github-actions]

Summary & Implementation Notes

This is a UX bug in app/index.html. The root cause is confirmed by reading the code directly.

Current code path

Method	What it does
buildSearchIndex() (line 520)	Tokenizes all searchable fields into a single flat Set<string> per record. Name, phenotype list, and long description tokens are indistinguishable.
filter() (line 558)	Boolean-only: keeps records where every query token matches some token in the flat set. No score is produced.
sortResults() (line 587)	Sorts by name-asc (default) or by date. When search is active, results still go alphabetical — there is no relevance option.

---

Suggested fix (all within app/index.html, no build step needed)

1. Replace the flat token set with a field-keyed index

buildSearchIndex() {
    const searchableFields = this.schema.searchableFields || [];
    // Field → weight map (higher = more important)
    const fieldWeights = { name: 100, synonyms: 60, phenotypes: 20, description: 5 };

    this.data.forEach((record, idx) => {
        const byField = {};
        searchableFields.forEach(field => {
            const value = record[field];
            const tokens = new Set();
            if (Array.isArray(value))
                value.forEach(v => this.tokenize(String(v)).forEach(t => tokens.add(t)));
            else if (value)
                this.tokenize(String(value)).forEach(t => tokens.add(t));
            byField[field] = tokens;
        });
        this.searchIndex.set(idx, byField);
    });
}

2. Add a scoring function

scoreRecord(record, queryTokens) {
    const fieldWeights = { name: 100, synonyms: 60, phenotypes: 20, description: 5 };
    const byField = this.searchIndex.get(record._idx);
    let totalScore = 0;

    for (const qt of queryTokens) {
        let tokenMatched = false;
        for (const [field, tokens] of Object.entries(byField)) {
            const w = fieldWeights[field] ?? 1;
            for (const rt of tokens) {
                if (rt === qt) { totalScore += w * 2; tokenMatched = true; }          // exact
                else if (rt.startsWith(qt)) { totalScore += w * 1.5; tokenMatched = true; } // prefix
                else if (rt.includes(qt)) { totalScore += w; tokenMatched = true; }        // substring
            }
        }
        if (!tokenMatched) return -1; // must match every query token (AND semantics preserved)
    }
    return totalScore;
}

3. Make filter() attach scores and auto-switch sort

filter() {
    let results = this.data.map((r, i) => ({ ...r, _idx: i }));

    if (this.searchQuery) {
        const queryTokens = this.tokenize(this.searchQuery);
        results = results
            .map(r => ({ ...r, _score: this.scoreRecord(r, queryTokens) }))
            .filter(r => r._score > 0);
    }

    // … facet filtering unchanged …
    return results;
}

4. In sortResults(), add a relevance case

sortResults(results) {
    const sorted = [...results];
    if (this.sortBy === 'relevance') {
        sorted.sort((a, b) => (b._score ?? 0) - (a._score ?? 0) || this.compareNames(a, b));
        return sorted;
    }
    // … existing name / date cases unchanged …
}

5. Auto-switch to relevance when search is active

In bindEvents() (line 849), inside the debounced input handler:

this.searchQuery = e.target.value;
if (this.searchQuery && this.sortBy === 'name-asc') this.sortBy = 'relevance';
if (!this.searchQuery && this.sortBy === 'relevance') this.sortBy = 'name-asc';
this.render();

And add the option to the sort <select>:

<option value="relevance" ...>Relevance</option>

---

Expected outcome after fix

• Searching "Parkinson" → Parkinson's Disease scores name-exact match × 100 × 2 = 200, while a record that only mentions "parkinsonian" in a description gets at most description-substring × 5 = 5. Parkinson's Disease ranks first.
• Prefix queries ("Parkin") and multi-word queries ("Sickle Cell") benefit from the same tiered scoring.
• Sort control auto-selects "Relevance" when a query is typed; reverts to "Name (A-Z)" when cleared.
• The field weight constants are easy to tune in one place.

The change is entirely self-contained in app/index.html — no schema changes, no Python pipeline changes, no new dependencies. The buildSearchIndex now builds a richer data structure but the memory footprint is negligible for the current KB size (~55 entries).

@dragon-ai-agent please implement the relevance ranking fix described above in app/index.html. The key changes are: (1) replace the flat token Set in buildSearchIndex with a per-field token map, (2) add a scoreRecord method using field weights (name=100, synonyms=60, phenotypes=20, description=5) and bonus multipliers for exact/prefix/substring matches, (3) update filter() to attach _score to results, (4) add a relevance sort option in sortResults() and the sort <select>, and (5) in bindEvents() auto-switch sort to relevance when a query is entered and back to name-asc when cleared. The AND semantics (every query token must match somewhere) should be preserved. Make a PR.