res['numeric_count'] calculation in Hierarchical Tokenizer

[zilinjing]

Hi, I am looking at your hierarchical tokenizer. I have a question for how you calculate ['numeric_count'] per property.

The related codes are "weight = 1.0 / (num_subjects * total_events)" (line 100) and "res['numeric_count'] += len(v['numeric_samples']) / weight" (line 151). So res['numeric_count'] += len(v['numeric_samples'])*num_subjects * total_events. Should it be res['numeric_count'] += len(v['numeric_samples'])*weight instead? I think the idea here is to determine how many quantile bins each property gets depends on total number of numerical values, that's why it is divided by (num_subjects * total_events).

[EthanSteinberg]

Actually, the weight should be removed entirely, as it is unused for that calculation, because everything is normalized. I do agree that * weight would be the more correct formula.

femr/src/femr/models/tokenizer/hierarchical_tokenizer.py

Line 207 in 37a9974

frac_numeric = property_data["numeric_count"] / total_numeric_weight

I'll leave this issue here and remove the weighting from that calculation