Eliminate Large Data File Redundancy

[turbomam]

Overview

Remove redundant code that scans the large fitness dataset (4.9M cells) when pre-filtered data is available.

Current Problems

• find_essential_genes() and find_growth_inhibitor_genes() scan all 4,440 genes × 1,117 conditions
• Recreating work already done in pre-filtered pairs data
• Poor performance for threshold-based operations

Redundant Code Patterns

BAD: Scans 4.9M cells
for gene_id, gene_info in fitness_loader.genes.items():
fitness_data = fitness_loader.get_gene_fitness(gene_id, condition_filter)
# Check if any conditions meet threshold...

GOOD: Uses pre-filtered significant effects
conditions = pairs_loader.get_conditions_for_gene(gene_id)

Already filtered to |value| > 2

Functions to Refactor

• find_essential_genes() → Use pairs_loader + filter by positive values
• find_growth_inhibitor_genes() → Use pairs_loader + filter by negative values
• Any other threshold-based discovery → Use filtered data first

Performance Impact

• Current: 4.9M cell scans for each discovery operation
• Proposed: ~40K pre-filtered pairs (100x faster)

Implementation Tasks

• Refactor find_essential_genes() to use filtered data
• Refactor find_growth_inhibitor_genes() to use filtered data
• Document data usage strategy in code comments
• Add performance tests comparing approaches
• Update function docstrings

Dependencies

Should be implemented after centralized metadata registry (Issue #21).