feat: Extend plugin support for IO and from_* methods/functions

[FBruzzesi]

Description

The core changes are in the eager_namespace for the backend resolution logic, which allows for backend= arguments to resolve for Narwhals plugins: users can pass a plugin's entry point name, module name, or the module itself.

• IO functions {read,scan}_{csv,parquet} dispatch to same-named top-level plugin hooks.
• Eager constructors (from_*, new_series, DataFrame.from_*, Series.from_*) work with zero extra plugin code if the plugin's compliant namespace implements the EagerNamespace protocol.

---

Most of the lines changes are in packages/test-plugin and tests/plugins_test.py so that we have coverage for the functionalities.

What type of PR is this? (check all applicable)

• 💾 Refactor
• ✨ Feature
• 🐛 Bug Fix
• 🔧 Optimization
• 📝 Documentation
• ✅ Test
• 🐳 Other

Related issues

• Closes Find way to support scan_csv for plugins #3713
• Related enh: Implementation for plugins #3042: the narwhals.plugins entry-point group is the registration mechanism we have. Not sure we can do much more than this dispatching

[dangotbanned]

Hey @FBruzzesi, just thought I'd share this here as I hinted at something I did in narwhals._plan a while back.
But I didn't really explain that it solved this issue 😅

Important

I'm not saying you need to do this, just inspiration if you want it

Examples

So here's a breakdown ...

read_csv

Most things look like some variation of this:

narwhals/src/narwhals/_plan/io/__init__.py

Lines 33 to 38 in d73c381

	def read_csv(
	source: FileSource, *, backend: IntoBackend[EagerAllowed], **kwds: Any
	) -> DataFrame[Any, Any]:
	source = normalize_path(source)
	manager = plugins.manager()
	return manager.dataframe(backend, _MAIN).read_csv(source, **kwds).to_narwhals()

Step 1

Get or initialize the plugin manager:

manager = plugins.manager()

Step 2

Import the compliant-level type that we want to construct with ¹:

tp_frame: type[CompliantDataFrame] = manager.dataframe(backend, Version.MAIN)

Step 3

Call the @classmethod constructor :

compliant_frame: CompliantDataFrame = tp_frame.read_csv(normalize_path(source), **kwds)

Yes, all the constructors are now @classmethod(s).
This makes a bunch of things simpler (including typing 😄):

narwhals/src/narwhals/_plan/compliant/dataframe.py

Lines 238 to 245 in d73c381

	@classmethod
	def from_dict(
	cls, data: Mapping[str, Any], /, *, schema: IntoSchema | None = None
	) -> Self: ...
	@classmethod
	def read_csv(cls, source: str, /, **kwds: Any) -> Self: ...
	@classmethod
	def read_parquet(cls, source: str, /, **kwds: Any) -> Self: ...

narwhals/src/narwhals/_plan/compliant/dataframe.py

Line 227 in d73c381

def concat(cls, dfs: Iterable[Self], /, how: ConcatMethod) -> Self: ...

Step 4

Wrap as the the narwhals-level object we'd like to return ²:

result: DataFrame = compliant_frame.to_narwhals()

Others

All follow the roughly same pattern, which is basically a successor to #2324 (minus two levels of Namespace):

• Series.from_iterable
• DataFrame.from_dict
• concat(Iterable[DataFrame]) ³
• LazyFrame.collect

Related

• https://github.com/narwhals-dev/narwhals/blob/d73c381def7aa8ca17f3f4d6687d63d1214e36bc/docs/api-reference-plan/plugins.md
• https://github.com/narwhals-dev/narwhals/blob/d73c381def7aa8ca17f3f4d6687d63d1214e36bc/src/narwhals/_plan/plugins/_manager.py
• https://github.com/narwhals-dev/narwhals/blob/d73c381def7aa8ca17f3f4d6687d63d1214e36bc/src/narwhals/_plan/io/__init__.py

Footnotes

1. You can ignore the version part for now ↩

2. The Version.MAIN has already propagated from the first call ↩

3. Skips the plugin manager part, but benefits from the @classmethod being exactly where we need it 🙂 ↩

[MarcoGorelli]

ok i love this 😍

[MarcoGorelli]

nice, thanks for testing this

[FBruzzesi]

Thanks both! @dangotbanned I think the implementation is not too far from what you have implemented, yet the machinery we have here is different. eager_namespace is in spirit the plugin manager you have. I will wait for merging to make sure you are also happy with the changes here