Skip to content

Make saving processor and gradient store optional#46

Merged
luciaquirke merged 4 commits intomainfrom
query
Oct 13, 2025
Merged

Make saving processor and gradient store optional#46
luciaquirke merged 4 commits intomainfrom
query

Conversation

@luciaquirke
Copy link
Collaborator

@luciaquirke luciaquirke commented Oct 12, 2025

This is the first of two planned PRs to add support for on-the-fly queries.

Other fixes in this PR:

  • For a sharded index build, accumulate one preconditioner over the shards rather than one preconditioner per shard. Because of our disk IO patterns we may still end up with one preconditioner shard for each file but the final one will have the accumulated information from every shard. Can clean this up more later.
  • Fix the default value for head_cfg - was a dict (!)
  • Add QueryConfig for planned on-the-fly query implementation
  • Add test_build file, as we support other saving options (e.g. in-memory return) or on-the-fly queries we can add tests here

Looking for review:

  • Made the GradientProcessor's preconditioner dict mutable
  • Not sure whether to pipe the QueryConfig into the current script as an optional second variable or duplicate the build.py code into a query.py script etc. to prevent build.py from growing in complexity too much.
    • or secret third option, look for some existing logic to extract out into another file to make build.py simpler

@luciaquirke luciaquirke changed the title [WIP] Support on-the-fly queries Make saving processor and gradient store optional Oct 12, 2025
@luciaquirke luciaquirke merged commit 56605ee into main Oct 13, 2025
3 checks passed
@luciaquirke luciaquirke deleted the query branch November 17, 2025 07:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant