Add Geomermaids OSM daily exports to distribution examples - #296
Conversation
b3426d6 to
5a30f46
Compare
|
Sorry for the slow response! Thanks for the contribution. I'm slammed right now, but will try to find some time in the next couple weeks to do at least a minimal bit of review on the OSM exports. |
|
sure, no worries |
|
Managed to find some time to check. It looks great! Though it does make me want to update the spec and best practices to articulate the benefits of adding bbox column to geoparquet 2.0. In gpio I give warnings now. So once we get it so that's in the spec I think it makes sense to merge in. I think there's rough consensus to add bbox covering back into the spec as optional, so I'll try to make that PR soon (or feel free to go for it with any of this). Thanks! |
|
I don't know if it really needs to go down into the specs, but for remote usage on files > 500MB it definitely helps to hard prune on the bbox column. So it's not all use cases and probably better implementation of 2.0 on the reader side will help too. With my GeoPQ-Workbench I have no issue working with 2.0 remote files up to 3-4GB ! |
|
Yeah, I think it's worth having in the spec just so that if people want to do it then it's clearly an option for 2.0 and not some undocumented thing. I think we can have the best practices doc discuss what are the use cases where you'd actually want to use it. Your input here and anything more you want to put in is great - we can just say where it helps, and also the tradeoffs with it (ie increasing the overall size of your data). |
|
Wrote up the bbox covering case in #297, with size and timing numbers from these files. Happy to keep this PR open until the spec side lands. |
2 participants
Adds the Geomermaids OSM daily exports to the list of GeoParquet data distribution examples in
distributing-geoparquet.md.These are daily OpenStreetMap exports published as GeoParquet v2.0, with Hilbert sorting and spatial partitioning, hosted at https://parquetry.geomermaids.com/. They serve as a real-world example of the distribution best practices described in this document.