Skip to content

libviewer for all#3264

Merged
lhoestq merged 5 commits intomainfrom
libviewer-for-all
Nov 24, 2025
Merged

libviewer for all#3264
lhoestq merged 5 commits intomainfrom
libviewer-for-all

Conversation

@lhoestq
Copy link
Copy Markdown
Member

@lhoestq lhoestq commented Nov 24, 2025

enable libviewer (page pruning features based on page index) for all datasets:

  • export metadata using libviewer, which enables the use of the page index (pyarrow misplaces it somehow)
  • use page pruning in /rows when a page index is present

this will make /rows faster and more efficient, and will enable cool use cases later like stopping re-converting datasets that are already in Parquet no matter their row group sizes

@lhoestq lhoestq merged commit 246420b into main Nov 24, 2025
29 checks passed
@lhoestq lhoestq deleted the libviewer-for-all branch November 24, 2025 23:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant