Skip to content

fix(gene burden): normalise NaN effect sizes to null - #107

Open
vivienho wants to merge 1 commit into
mainfrom
vh-fix-gene-burden-nan
Open

vivienho wants to merge 1 commit into
mainfrom
vh-fix-gene-burden-nan

Conversation

@vivienho

Copy link
Copy Markdown
Contributor

Summary

  • Normal rows leave beta/oddsRatio as SQL NULL when no effect size is reported, e.g. most gene_burden rows correctly get directionOnTrait: NULL when both fields are null.
  • SKAT/SKAT-O rows from BRaVa and Genes & Health instead got directionOnTrait: 'risk' even though no effect size was ever computed, e.g. beta: NaN, oddsRatio: null.
  • The source files encode "no effect size" as the string "NaN" ('BETA Burden' in BRaVa's S14/S15, 'effect'/BETA in Genes & Health's ST9/ST13/ST15), which Spark's double cast turns into a float NaN rather than NULL.
  • Spark SQL orders NaN as larger than any other value, so direction_on_trait_expression's beta > 0.0 (or oddsRatio > 1.0) evaluates TRUE for a NaN input, misclassifying 5,617 rows as 'risk'.

Fix

  • In _process_brava_granular, normalise 'BETA Burden' from NaN to NULL right after casting it to double, so SKAT/SKAT-O rows (which carry no effect size) behave as missing.
  • In process_genes_and_health_gene_burden, normalise 'effect' from NaN to NULL right after unioning the three source tables, before it's used to derive beta/oddsRatio.
  • Corrected output: beta: null, oddsRatio: nulldirectionOnTrait: null instead of 'risk'.

@ireneisdoomed ireneisdoomed left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well spotted! I have double checked that the columns with NaN in the numeric columns are: beta, oddsRatio + the correspondent 4 fields about the upper and lower bounds of the confidence intervals.

Because you are standardising the effect and beta columns directly, the derived columns will be fixed too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants