Document the GeoArrow stream interface

Where the GeoArrow work is visible in the public API: snail.core.intersections
gained split_linestrings, split_polygons and SplitStream, and
snail.intersection gained to_geoarrow and read_split_stream. The C++
docstrings already carried through Sphinx into the API reference; the Python
side did not:

- SPLIT_BATCH_SIZE had a substantial rationale comment that never reached
  the docs, since a bare assignment renders nothing - moved it to a `#:`
  comment, which autodoc picks up as the constant's docstring.
- to_geoarrow and read_split_stream had one-line docstrings with no
  Parameters/Returns, unlike the rest of the module. read_split_stream in
  particular reads a stream to completion, which is easy to reach for
  without noticing it gives up the memory benefit the streaming interface
  exists for - the docstring now says so, and points at iterating the
  stream directly instead.
- snail.intersection.split_linestrings, the main entry point, never
  documented its bounded parameter, which changes what a split returns
  quite a bit. Added Parameters, since it sits right next to the streaming
  changes and a reader working out the new behaviour needs it.

None of this changes behaviour; verified by building the Sphinx docs and
checking the new cross-references resolve (they do) and no new warnings
appear (151 before and after, all pre-existing tutorial notebook issues).

Also added docs/source/splitting-arrow.rst, linked from the main toctree.
snail.intersection.split_linestrings/split_polygons already stream
internally, so most users need nothing new; this page is for the two
audiences who do: splitting a source that isn't already a GeoDataFrame
(a GeoParquet file bigger than memory, say), and code working directly
against snail.core.intersections. It explains why geometries cross the
Python/C++ boundary as an Arrow stream rather than an array, what sources
are accepted, and walks through splitting a GeoParquet file batch by
batch without loading it whole - verified runnable against the built
extension. It links out to the API reference rather than repeating it.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SqdWshmD4AUHhrqeMh86GS

Author: claude

docs/source/index.rst

@@ -114,6 +114,11 @@ Contents
 
     Tutorials <tutorials>
 
+.. toctree::
+    :maxdepth: 1
+
+    Splitting large datasets <splitting-arrow>
+
 .. toctree::
     :maxdepth: 3
 

docs/source/splitting-arrow.rst

@@ -0,0 +1,89 @@
+Splitting large datasets
+=========================
+
+:func:`snail.intersection.split_linestrings` and
+:func:`snail.intersection.split_polygons` (and the ``snail split``/``snail
+process`` commands built on them) already do everything on this page for
+you: they take a :class:`~geopandas.GeoDataFrame` and give one back. Read
+on only if you want to split a source that is not already loaded as a
+GeoDataFrame - a GeoParquet file bigger than memory, say - or you are
+working directly against the compiled extension.
+
+How a split runs
+-----------------
+
+Geometries cross into the C++ extension as `GeoArrow
+<https://geoarrow.org/>`_, over the `Arrow C stream interface
+<https://arrow.apache.org/docs/format/CStreamInterface.html>`_: the
+extension pulls one batch of geometries at a time from its source, splits
+it, and hands the pieces back the same way, as a stream of record batches.
+Nothing is split until that stream is read, and only one batch of the
+source is ever held at once - so a file far larger than memory can be
+split by a consumer that takes the results as they come, without loading
+the whole thing first.
+
+:func:`snail.core.intersections.split_linestrings` and
+:func:`~snail.core.intersections.split_polygons` accept *any* object
+implementing the Arrow PyCapsule stream interface (``__arrow_c_stream__``)
+or the single-array interface (``__arrow_c_array__``, read as a stream of
+one batch) - a :class:`pyarrow.ChunkedArray`, :class:`~pyarrow.Table` or
+:class:`~pyarrow.RecordBatchReader`, a GeoParquet or
+:class:`pyarrow.dataset.Dataset` reader, or
+:meth:`GeoSeries.to_arrow() <geopandas.GeoSeries.to_arrow>`. Coordinates
+may be interleaved (as geopandas exports them) or separated into x and y
+arrays (as GeoParquet stores them) - both are read directly, with no
+per-feature Python object built on either side of the interface.
+
+Splitting a file directly
+--------------------------
+
+This reads a GeoParquet file of linestrings in batches and splits each
+batch as it arrives, never holding more than one batch of geometries and
+one batch of pieces at a time::
+
+    import pyarrow
+    import pyarrow.parquet
+    from snail.core.intersections import split_linestrings
+    from snail.intersection import GridDefinition
+
+    grid = GridDefinition.from_raster("hazard.tif")
+
+    parquet_file = pyarrow.parquet.ParquetFile("edges.geoparquet")
+    geometry_batches = pyarrow.RecordBatchReader.from_batches(
+        parquet_file.schema_arrow, parquet_file.iter_batches(batch_size=10_000)
+    )
+
+    stream = split_linestrings(
+        geometry_batches, nrows=grid.height, ncols=grid.width, transform=grid.transform
+    )
+
+    reader = pyarrow.RecordBatchReader._import_from_c_capsule(stream.__arrow_c_stream__())
+    for batch in reader:
+        # batch has a "geometry" column of the pieces (GeoArrow-encoded)
+        # and a "parent" column: the index, in the source, of the
+        # geometry each piece was split from
+        ...
+
+``split_linestrings`` also takes a ``bounded`` argument, working the same
+way as the ``bounded`` parameter of
+:func:`snail.intersection.split_linestrings`: pass ``bounded=True`` to
+leave geometries (or parts of geometries) outside the grid whole, rather
+than split at every gridline they would otherwise cross.
+
+To get the pieces as shapely geometries instead - trading the streaming
+memory benefit for convenience - read the whole stream at once with
+:func:`snail.intersection.read_split_stream`, or convert a batch with
+:meth:`geopandas.GeoDataFrame.from_arrow`.
+
+Reference
+---------
+
+* :class:`snail.core.intersections.SplitStream` - what a split returns
+* :func:`snail.core.intersections.split_linestrings`,
+  :func:`~snail.core.intersections.split_polygons` - the extension
+  functions described above
+* :func:`snail.intersection.to_geoarrow`,
+  :func:`snail.intersection.read_split_stream` - the conversions
+  :func:`snail.intersection.split_linestrings` and
+  :func:`~snail.intersection.split_polygons_experimental` use to work with
+  an in-memory :class:`~geopandas.GeoDataFrame`

src/snail/intersection.py

@@ -38,16 +38,19 @@
     from snail.tqdm_standin import tqdm_standin as tqdm
 
 
-# The extension splits one Arrow batch at a time, so an in-memory geometry
-# column is presented to it in batches of this many features: splitting a
-# whole table at once would give no sign of progress on a long job, and
-# would hold every piece of every feature in memory at once. A source that
-# brings its own batching - a GeoParquet reader, a Dataset scan - can be
-# split directly, and keeps whatever batch size it was read with.
-#
-# Each batch costs something fixed to set up and read back, so batches much
-# smaller than this measurably slow a large split down; much larger ones buy
-# no more speed and only raise the peak memory.
+#: Batch size for splitting an in-memory geometry column.
+#:
+#: The extension splits one Arrow batch at a time, so an in-memory geometry
+#: column is presented to it in batches of this many features: splitting a
+#: whole table at once would give no sign of progress on a long job, and
+#: would hold every piece of every feature in memory at once. A source that
+#: brings its own batching - a GeoParquet reader, a Dataset scan - can be
+#: split directly, and keeps whatever batch size it was read with; this only
+#: governs the batches :func:`to_geoarrow` slices an in-memory column into.
+#:
+#: Each batch costs something fixed to set up and read back, so batches much
+#: smaller than this measurably slow a large split down; much larger ones buy
+#: no more speed and only raise the peak memory.
 SPLIT_BATCH_SIZE = 5000
 
 
@@ -58,6 +61,26 @@ def to_geoarrow(geometries: geopandas.GeoSeries, batch_size: int = SPLIT_BATCH_S
     which the extension reads directly - no geometry object is built per
     feature on either side of the interface. Batches are zero-copy slices
     of the one Arrow array, and the geometry type travels with them.
+
+    This is what :func:`split_linestrings` and :func:`split_polygons_experimental`
+    use to feed a geometry column to :func:`snail.core.intersections.split_linestrings`
+    or :func:`snail.core.intersections.split_polygons`; call it directly
+    only if you are working with those lower-level, Arrow-native functions
+    yourself.
+
+    Parameters
+    ----------
+    geometries: geopandas.GeoSeries
+        Column of LineString or Polygon geometries to split.
+    batch_size: int
+        Number of features per batch. See :data:`SPLIT_BATCH_SIZE`.
+
+    Returns
+    -------
+    pyarrow.Table
+        A single-column ``"geometry"`` table, chunked into batches of
+        ``batch_size`` features, implementing the Arrow PyCapsule stream
+        interface (``__arrow_c_stream__``) that the extension consumes.
     """
     # Import the geometry column through its capsules rather than with
     # pyarrow.array, which would keep the values but drop the GeoArrow
@@ -81,10 +104,34 @@ def to_geoarrow(geometries: geopandas.GeoSeries, batch_size: int = SPLIT_BATCH_S
 
 
 def read_split_stream(stream) -> tuple[numpy.ndarray, numpy.ndarray]:
-    """Read a stream of split pieces from the extension
+    """Read a stream of split pieces from the extension into memory
+
+    The split runs as the stream is read, a batch at a time, but this
+    drains the stream fully and concatenates every batch: use it when you
+    want the pieces as shapely geometries and are content to hold them all
+    at once, which is what :func:`split_linestrings` and
+    :func:`split_polygons_experimental` do. To keep the streaming memory
+    benefit - splitting a source larger than memory, for example - iterate
+    the stream yourself instead, e.g. with
+    ``pyarrow.RecordBatchReader._import_from_c_capsule(stream.__arrow_c_stream__())``,
+    and consume each record batch as it arrives.
 
-    The split runs as the stream is read, a batch at a time. Returns the
-    pieces, and for each piece the index of the geometry it came from.
+    Parameters
+    ----------
+    stream
+        A :class:`snail.core.intersections.SplitStream`, or any object
+        implementing the Arrow PyCapsule stream interface
+        (``__arrow_c_stream__``) with a ``"geometry"`` and a ``"parent"``
+        column, such as the result of
+        :func:`snail.core.intersections.split_linestrings` or
+        :func:`snail.core.intersections.split_polygons`.
+
+    Returns
+    -------
+    geometry: numpy.ndarray
+        The split pieces, as shapely geometries.
+    parent: numpy.ndarray
+        For each piece, the index of the geometry it was split from.
     """
     reader = pyarrow.RecordBatchReader._import_from_c_capsule(
         stream.__arrow_c_stream__()
@@ -296,6 +343,18 @@ def split_linestrings(
 
     Each piece lies within a single grid cell; together the pieces of a
     feature are that feature, cut up, so the split conserves its length.
+
+    Parameters
+    ----------
+    linestring_features: geopandas.GeoDataFrame
+        Features to split; other columns are carried over onto each piece.
+    grid: GridDefinition
+        Grid to split along.
+    bounded: bool
+        If False (the default), a feature is split for its whole length,
+        including any part that falls outside the grid. If True, splitting
+        stops at the grid's edge: pieces outside the grid are left whole
+        rather than cut at every gridline they would otherwise cross.
     """
     # TODO check for MultiLineString
     # throw error or coerce (df.explode)