ENH: spatial partitioning of the GeoDataFrame

[jorisvandenbossche]

For making spatial joins or overlays, spatial predicates, reading from spatially partitioned datasets, etc more efficient, we can have spatially partitioned dataframes: the bounds of each partition is known, and thus it can be checked based on those bounds whether on operation needs to involve that partition or not.
And then geodataframes can also be re-partitioned to optimize the bounds (minimize the overlap) as much as possible (initial costly shuffle operation, but can pay-off later).

This complicates the implementation (we need to keep track of the spatial partitioning, the partitions can change during spatial operations, ..), but I think it will also be critical for improving performance on large datasets.

---

How can we add this?

In the previous iteration at https://github.com/mrocklin/dask-geopandas, the dataframes had an additional _regions attribute, which was a geopandas.GeoSeries with the "regions" of each partition (so len(regions) == npartitions).

See https://github.com/mrocklin/dask-geopandas/blob/8133969bf03d158f51faf85d020641e86c9a7e28/dask_geopandas/core.py#L50

I think one advantage of using a GeoSeries is that this makes it easy to work with (eg it is easy to check which partitions would intersect with a given geometry).

In spatialpandas (https://github.com/holoviz/spatialpandas), there is a combo of partition_bounds and partition_sindex.
The partition_bounds is basically the total_bounds of each partition (so you could see it as the _regions but limited to a rectangular box and stores as the 4 (minx, miny, maxx, maxy) numbers). And then partition_sindex is a spatial index built on the partition_bounds.

See https://github.com/holoviz/spatialpandas/blob/master/spatialpandas/dask.py

---

I suppose starting with a basic "partition bounds" should be fine, and allows to later expand it with a spatial index or with more fine-grained shapes.

[borao]

What do you think about potentially partitioning by attribute?

For example, if I have a set of points (maybe cities) in the US with the state as a feature, and I want to spatially join it with a set of counties (polygons) in the US that also has the state as a feature, the computation could be significantly optimized by not having to worry about overlap/other geometric nuances in the borders, and most computation would be within the partition.

[mrocklin]

FWIW I found that spatial partitioning with a GeoPandas series felt pretty clean. Many of the operations like spatial joins were relativly straightforward to write. Ideally some the implementation in the previous version could be reused.

Things do become odd when you start considering geometries that can cross partition boundaries though. (This will be a problem regardless of which partitioning scheme you use). I think that there are ways to handle it, but it requires some thinking. For point-wise geometries though everything is pretty straightforward.

[jorisvandenbossche]

What do you think about potentially partitioning by attribute?

I think that's something that is already supported by dask (but @mrocklin can correct me if I am wrong). See eg the doc page on shuffling at https://docs.dask.org/en/latest/dataframe-groupby.html
You can for example do a set_index(..) for a certain attribute column, which will repartition the dataframe based on that attribute, making joins on this new index more efficient.

The spatial partitioning will then mostly be useful for cases when you don't have such attribute that can be used as index (or where the regional extent of the attribute is not exactly the same accross datasets).

Of course, if you know you have an attribute that describes certain regions, we could make it possible to specify this as input to the "spatial re-partitioning" function as a basis on how to repartition (instead of using some other logic to divide the space into regions).
(this would be similar as doing a set_index() and then calculating the bounds for the partitions as they are)

[jsignell]

I think copying Matt's original implementation of using a geoseries to represent the divisions makes sense. Is there a cost associated with that that makes the spatialpandas path more compelling?

[jorisvandenbossche]

Only boxes are in principle a bit simpler, but on the other hand, storing it as a geoseries might make working with it easier. And is also more general (and actually gives a spatial index on those partition bounds directly as well, as the geoseries has it).
We can also always simplify the regions on the partitions in the geoseries to boxes / envelopes to avoid the partition bounds being too detailed (a plain union of all geometries inside the partition might give a very detailed polygon).

So certainly on board with starting with such an approach.

Regarding naming, I actually like the "partition_bounds", only "bounds" is typically pointing to a bounding box.

[jsignell]

Can we just call them divisions like dask does?

[jorisvandenbossche]

I am not sure that will play nicely with dask? Eg does dask expect it to be a tuple?

Also, this are not divisions for the index, but a different concept, which might be confusing?

[caspervdw]

Only boxes are in principle a bit simpler, but on the other hand, storing it as a geoseries might make working with it easier. And is also more general (and actually gives a spatial index on those partition bounds directly as well, as the geoseries has it).
We can also always simplify the regions on the partitions in the geoseries to boxes / envelopes to avoid the partition bounds being too detailed (a plain union of all geometries inside the partition might give a very detailed polygon).

So certainly on board with starting with such an approach.

Regarding naming, I actually like the "partition_bounds", only "bounds" is typically pointing to a bounding box.

I might be looking at this issue too much from one side, but I am really drawn towards the bounding box-only approach. Such a partition looks more like an array chunk. For vector-raster analyses this match will potentially allow more optimalizations.

Maybe this would also allow borrowing some (battle-tested) logic from dask.array?

[jorisvandenbossche]

In the analogy with dask.array, I think some problems are that 1) we don't have a regular grid with rectangles (in dask, the chunksize can vary in each dimension, but it's still are all rectangles in a grid, I think), and 2) we will typically have overlapping bounding boxes / spatial partitions (with polygons of all shapes, and each polygon having to fit completely in one of the bounding boxes, I think it is unavoidable that the boxes will typically overlap slightly).

(that's not to say that a simpler bounding-box-only scheme couldn't be interesting, just pointing out it's quite a bit more complicated as dask.array's chunking scheme)

[caspervdw]

1. we don't have a regular grid with rectangles (in dask, the chunksize can vary in each dimension, but it's still are all rectangles in a grid, I think)

You are correct that dask.array works with rectangular grids. I'd propose to make that the default option also for the spatial partitioning of geometries. There are roughly two options: 1) a partition consists of all geometries that are intersecting with it or 2) a partition consists of all geometries that have some deterministically assigned point (e.g. centroid) in it. The first will give duplicates, but the second doesn't allow pruning partitions in some spatial operations like rasterizing or overlay. It depends on your typical geometry size if the duplicates are acceptable; in many cases they are and you still get a big performance increase because of the parallelization.

So I'd propose the first one (intersections, with duplicates) but keep the door open for developing more advanced schemes (carrying the convex hull of the parition along, quadtree, space filling curves, ...). Advantages of this approach (in my personal experience) are:

• The partitioning does not depend on the actual data. So you don't have to go through and pay the cost of IO / cpu before even starting to build the dask collection.
• The partitions can be defined with only 6 numbers (x0, y0, dx, dy, nx, ny) as opposed to N geometries. This will allow non-materialized layers (as in high level graphs) and efficient high level optimizations. This is especially useful if you are working with lots of partitions.
• With the correct tiling, the output could go into a tile grid, which are expected for many services (like vector tiles).

[jorisvandenbossche]

Interesting! That seems to point to a fundamental difference in thinking about "what is a spatial partition (bounds)". Because in my mind, it's still something else as the two options you list.

In your two options, you seem to start from the spatial extent, and then determine which geometries belong to that spatial partition (either all geometries intersecting with it, or having a representative point lying within it).

In my mind, and thinking about how a dask.dataframe works, another option (and what is currently done in the dummy implementation) is to start from the actual physical data: a dask dataframe is splitted into different partitions (sub-dataframes), and so the spatial partitioning is determined by the spatial extent (total bounding box for simplicity, or a more complex polygon) of the geometries that are stored in a certain partition. So the spatial bounds of a partition is a bounding box (or polygon) that at least fully covers all geometries of its partition.
(and then it of course becomes a matter of ensuring that the geometries are sorted over the different partitions in such a way that those spatial extents are efficient and not overlapping too much -> that's a whole separate topic of how to smartly "re"partition)

In that mindset, for non-point geometries and when working with simple bounding boxes, you will have by definition overlapping bounding boxes of the spatial partitions (unless all the polygon geometries are rectangular as wel, but let's not consider such special case for the general design ;))
For point geometries, it's of course possible to have a more efficient non-overlapping bounding boxes (like the resulting rectangles from a quadtree spatial index).

---

I want to get back to the two options you listed above as well, and discuss some potential problems (or to explain "properties" that I think are important for a spatial partitioning system):

1) a partition consists of all geometries that are intersecting with it.

As you mention, this will give duplicates. However, I am not sure this is actually practical with dask.dataframe to have duplicates. Either you need to accept that things like len(df) or df["attribute"].mean().compute() will give wrong answers (which I don't think is acceptable), or either such simple calculations like the length or the mean of a column will each time have to "deduplicate" the rows first (which I think is not that easy, will impact performance for non-spatial operations, and will also loose a lot of the benefits of using dask.dataframe).

2) a partition consists of all geometries that have some deterministically assigned point (e.g. centroid) in it.

For this one, you mention that it doesn't allow pruning partitions in some spatial operations. And I think that's indeed an essential property of the spatial partitioning is that the bounding box (or polygon) representing a given partition should fully cover the actual geometries (and not only contain a representative point, or only intersect with the polygon).
For example, if I want to check if a certain polygon intersects with any of the polygons in my dataframe (gdf.intersects(polygon) type of operation), I think we need to be able to rely on the spatial partitioning information to know which partitions to check based on whether the given polygon intersects with any of the bounding boxes of the partitions. If the bounding boxes are not guaranteed to fully cover its geometries, we don't have reliable information about potential intersection with the given polygon, and that means we would basically always need to check the interesects operation with all partitions, not being able to skip any partition, and thus loosing the benefit of having spatial partitions in the first place.