Skip to content

feat: parallelize PGEN reading #273

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 52 commits into
base: main
Choose a base branch
from
Draft

feat: parallelize PGEN reading #273

wants to merge 52 commits into from

Conversation

aryarm
Copy link
Member

@aryarm aryarm commented Mar 2, 2025 •
edited
Loading

resolves #253

I've been working on this in my off-time while traveling. (Unlike my other work, this doesn't require access to the internet!) But there's still much to do and this isn't a priority, so this will probably remain unmerged for a while. See #253 for the full TODO list

aryarm added 30 commits September 7, 2024 03:01
@aryarm
refactor: move pgen chunk reading into a separate helper function
8fada1d
@aryarm
recover args from pickling
36482fc
@aryarm
declare shared vars as global instead
3ef62da
@aryarm
enable multiprocessing in GenotypesPLINK.read
a03d9f2
@aryarm
improve explanation for _init_mp
5d6336f
@aryarm
try to optimize cpu usage and use more accurate measure of num CPUs
bc4bb6f
@aryarm
refrain from doing unnecessary benchmarks and remove chunk_size param
c83c905
@aryarm
fix polyfit bug when vals for file type are not generated
777e96a
@aryarm
recover broken plotting of variants
a06cb6b
@aryarm
use ceil instead of floor in chunking
985e6e1
@aryarm
fix chunking bug in bench gts
c880c96
@aryarm
improve splitting of variants across CPUs using Navid's strategy
2a1471d
@aryarm
try to use mp.starmap's chunking
4b1f889
@aryarm
try bigger chunks in bench_gts
413c645
@aryarm
refactor: move pgen chunk reading into a separate helper function
02ea8f0
@aryarm
recover args from pickling
91297da
@aryarm
declare shared vars as global instead
523a5cb
@aryarm
enable multiprocessing in GenotypesPLINK.read
b224d21
@aryarm
improve explanation for _init_mp
0f9b7f4
@aryarm
try to optimize cpu usage and use more accurate measure of num CPUs
f8250d4
@aryarm
refrain from doing unnecessary benchmarks and remove chunk_size param
d0fe09d
@aryarm
fix polyfit bug when vals for file type are not generated
5cb9a05
@aryarm
recover broken plotting of variants
d656dd2
@aryarm
use ceil instead of floor in chunking
3087b08
@aryarm
fix chunking bug in bench gts
e6d8432
@aryarm
improve splitting of variants across CPUs using Navid's strategy
cbfd560
@aryarm
try to use mp.starmap's chunking
424786d
@aryarm
try bigger chunks in bench_gts
5422934
@aryarm
try to load num variants from cyvcf2
2fccf5e
@aryarm
try to add pgen file for example.vcf.gz
eb88bef
@aryarm aryarm marked this pull request as draft March 2, 2025 17:40
@aryarm aryarm mentioned this pull request Mar 20, 2025
Open
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

parallelize reading from a PGEN file
1 participant
@aryarm