Lazify get_candidates - #915
Draft
AdrianSosic wants to merge 11 commits into
Draft
Conversation
Hash computation and helper calls remain pandas-based for now
The register_hooks mechanism currently requires the types to be defined at runtime
AdrianSosic
force-pushed
the
refactor/narwhalify_candidates
branch
from
September 10, 2026 12:49
a278c74 to
f5de101
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Another step closer to lazification. Finally avoids candidate materialization by offering an appropriate lazy interface.
Why the two-method interface?
get_candidates()is part of the public API and must return a native type – returning a narwhals wrapper would leak an implementation detail to users. The natural return type for a potentially large candidate set is a lazy frame, so callers can defer materialization until they actually need the data.The tension is: internal library callers need an eager narwhals
DataFrameto perform operations like.filter(), positional indexing, and.sample(). Ifget_candidates()returned the native lazy frame, every such caller would need to re-wrap it in narwhals and normalize before collecting:The
.lazy()call in the first form is unavoidable: for a pandas-backed result,nw.from_native(pd.DataFrame)gives anw.DataFrame(not anw.LazyFrame), so.collect()does not exist on it without normalizing through.lazy()first. This boilerplate would appear at every internal call site.The two-method design resolves this cleanly:
get_candidates() -> IntoLazyFrame: public API, returns the native lazy frame. Callers own the decision of when to materialize._get_candidates() -> nw.LazyFrame: private, stays in narwhals. Internal callers collect exactly when and how they need to (eager frame, specific native type, or a lazy aggregation like a row count), without any re-wrapping overhead.