Improve API for VCF read_variant_stats() and read_allele_count()#852
Merged
Conversation
Previously the method returned a pyarrow.Table but the return type hint was a pandas.DataFrame, so the new behavior is correct. A pyarrow.Table can still be retrieved via a new Dataset.read_variant_stats_arrow() method.
…stats_arrow() This allows multiple regions to be queried simultaneously. Consequently, a "contig" column was added to the returned DataFrame and Table, respectively, to clarify what contig positions occur on. The old "region" parameter is still present but has been deprecated in preference for the new "regions" parameter.
…nd consolidate regions This ensures that stats have a consistent ordering and that there isn't redundant rows when regions overlap. Note that a Region class was introduced to facilitate this functionality.
…ariant_stats_arrow() This allows variant stats to be computed relative to all samples in the dataset. To support this change, an AFComputer abstract class and two subclasses were introduced to encapsulate the AF stats code. Additionally, supporting types were introduced and variables that use these types were renamed to clarify how the code that uses these classes works.
…tats_arrow() This causes the "ref" allele to be omitted from the results.
Previously the method returned a pyarrow.Table but the return type hint was a pandas.DataFrame, so the new behavior is correct. A pyarrow.Table can still be retrieved via a new Dataset.read_allele_count_arrow() method.
…unt_arrow() This allows multiple regions to be queried simultaneously. Consequently, a "contig" column was added to the returned DataFrame and Table, respectively, to clarify what contig positions occur on. The old "region" parameter is still present but has been deprecated in preference for the new "regions" parameter. Also, the code for parsing and preparing input regions used by read_variant_stats() and read_variant_stats_arrow() was encapsulated in a _prepare_regions() method so that it may be used in this context as well.
This gives AlleleCountReader access to AlleleCount's private members, thus removing the need for redundant variables.
…RRAY member during construction
This means all regions passed to the API are now handled consistently, i.e. they are parsed, sorted, and consolidated the same way for every method.
…unt() This is to provide a more deterministic ordering than what is provided by the underlying C++ API.
Member
Author
|
@XanthosXanthopoulos This PR is ready for review. |
| "regions": regions, | ||
| "scan_all_samples": scan_all_samples, | ||
| } | ||
| return self.read_variant_stats_arrow(**kwargs).to_pandas() |
Collaborator
There was a problem hiding this comment.
Are any of the two flags described here helpful in this case?
Member
Author
There was a problem hiding this comment.
Thanks for the suggestion. I think this could benefit all of the .to_pandas() calls in this file. I'll do some testing and report back.
Member
Author
There was a problem hiding this comment.
@XanthosXanthopoulos I updated all of the Table.to_pandas() calls to use the technique described in the link you provided. I did some basic profiling of these calls with tracemalloc and didn't see a significant improvement. However, I see no harm in making this change and it may help when dealing with large datasets in production so I went ahead and pushed it.
| if self.mode != "r": | ||
| raise Exception("Dataset not open in read mode") | ||
|
|
||
| return self.read_allele_count_arrow(regions=regions).to_pandas() |
This was done using the split_blocks and self_destruct parameters in tandem.
XanthosXanthopoulos
approved these changes
Nov 5, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
read_variant_stats():read()and deprecated singleregionparameter.scan_all_sampleslikeread()."ref"alleles.DataFrameis now returned by default; arrowTablecan still be returned via newread_variant_stats_arrow()method .contigcolumn for joining.contig, thenpos.read_allele_count():readand deprecated singleregionparameter.DataFrameis now returned by default; arrowTablecan still be returned via newread_allele_count_arrow()method .contigcolumn for joining.contig, thenpos.Miscellaneous:
Regionclass in the Python API that is used to parse, sort, and consolidate regions; this is now applied to all regions input to the API.TODOin the related allele count code path.