Skip to content

allow block index loading to use CSafeLoader#1920

Merged
braingram merged 2 commits into
asdf-format:mainfrom
braingram:block_index_parsing
May 8, 2025
Merged

allow block index loading to use CSafeLoader#1920
braingram merged 2 commits into
asdf-format:mainfrom
braingram:block_index_parsing

Conversation

@braingram
Copy link
Copy Markdown
Contributor

@braingram braingram commented Apr 24, 2025

On main the block index is always loaded with SafeLoader. For files with a very large number of blocks this can take a considerable length of time.

This PR replaces the SafeLoader use with the "base loader" from asdf.versioning:

_yaml_base_loader = yaml.CSafeLoader if getattr(yaml, "__with_libyaml__", None) else yaml.SafeLoader

which can be CSafeLoader.

For a file with 10000 blocks main takes ~0.5 second (more with profiling) to load the block index:

Screenshot 2025-04-24 at 1 41 23 PM

With this PR the time is about a factor of 10 less:
Screenshot 2025-04-24 at 1 39 08 PM

Tasks

  • run pre-commit on your machine
  • run pytest on your machine
  • Does this PR add new features and / or change user-facing code / API? (if not, label with no-changelog-entry-needed)
    • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types)
    • update relevant docstrings and / or docs/ page
    • for any new features, add unit tests
news fragment change types...
  • changes/<PR#>.feature.rst: new feature
  • changes/<PR#>.bugfix.rst: bug fix
  • changes/<PR#>.doc.rst: documentation change
  • changes/<PR#>.removal.rst: deprecation or removal of public API
  • changes/<PR#>.general.rst: infrastructure or miscellaneous change

@braingram braingram force-pushed the block_index_parsing branch from 09a890d to 04e7185 Compare April 24, 2025 18:00
@braingram braingram marked this pull request as ready for review April 24, 2025 18:00
@braingram braingram requested a review from a team as a code owner April 24, 2025 18:00
@braingram braingram force-pushed the block_index_parsing branch from 04e7185 to 654319d Compare May 8, 2025 14:09
@braingram braingram enabled auto-merge May 8, 2025 14:09
@braingram braingram merged commit 7356470 into asdf-format:main May 8, 2025
35 checks passed
@braingram braingram deleted the block_index_parsing branch May 8, 2025 14:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants