Skip to content

v0.11.3

Choose a tag to compare

@github-actions github-actions released this 27 Oct 00:13
Allow for passing pipelines to xbeam.Dataset constructors.

Associating a beam.Pipeline with an xbeam.Dataset means that a pipeline doesn't need to be applied later (e.g., to the result of `to_zarr`). This is both a little cleaner, and also potentially a significant optimization, because it means that Beam understands that it can reuse a ptransform rather than recomputing it.

This includes a new `_LazyPCollection` class to ensure that our optimizations for Transforms applied directly after xbeam.DatasetToChunks still works.

PiperOrigin-RevId: 824281688