[GEN-1704] Filter out germline variants from sv files

[rxu17]

Purpose: This PR will be a hotfix to filter out germline variants from the sv file at the consortium release step (and subsequently the public release step since it just copies over the consortium release sv file) before we can release the germline variant validation rule.

Changes: Changes are isolated to the store_sv_files function in database_to_staging.py. Had to make it case-insensitive because I've found that SV_STATUS is in the testing pipeline's consortium release sv file while SV_Status is found in the production pipeline 's consortium release sv file.

Testing: Followed the standard validation of new features guide. The germline variants are filtered out.

[sonarqubecloud]

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[thomasyu888]

🔥 LGTM. will defer to @danlu1 for final review.

[danlu1]

LGTM!

[danlu1]

@rxu17 just a follow up question, since you found "SV_STATUS is in the testing pipeline's consortium release sv file while SV_Status is found in the production pipeline 's consortium release sv file.". Do we have plan to update testing pipeline to make it align with prod?

[rxu17]

@rxu17 just a follow up question, since you found "SV_STATUS is in the testing pipeline's consortium release sv file while SV_Status is found in the production pipeline 's consortium release sv file.". Do we have plan to update testing pipeline to make it align with prod?

I think this is another ticket to figure out why it's happening and then resolve from there (because I'm not sure why it's happening). I think that's a low priority ticket given we have no known current issues. Overarching goal to resolve this would be having a data validation framework set up since we're not enforcing any specific data standards and checks in the release files.