Shapefile driver lies about geometry type

[dbaston]

A common [case] I see that gives error code 1 but as a Warning is Shapefile -> GPKG when the .shp has MultiPolygon. I recently saw that warning also with ESRI File Geodatabase -> GPKG and MultiSurface. With CLI, it gives ( 3.14dev)

gdal convert poly_multipoly.shp out.gpkg
0...10...20..Warning 1: A geometry of type MULTIPOLYGON is inserted into layer
poly_multipoly of geometry type POLYGON, which is not normally allowed by the
GeoPackage specification, but the driver will however do it. To create a
conformant GeoPackage, if using ogr2ogr, the -nlt option can be used to
override the layer geometry type. This warning will no longer be emitted for
this combination of layer and feature geometry type.
.30...40...50...60...70...80...90...100 - done.

Many users will be familiar with that case and accustomed to specify -nlt. The message may be confusing in CLI since it refers to ogr2ogr, and gdal vector convert doesn't have an nlt option. Instead the solution is to use gdal vector set-geom-type. Maybe this is a code change in vector convert? The Migration Guide has "Converting a shapefile to a GeoPackage" giving ogr2ogr -> gdal vector convert but doesn't mention set-geom-type.

As a user I also wonder about the implications of creating a non-conformant GPKG. I would assume it's fine as long as the file will only be read by GDAL, but not if used in ArcGIS?

Originally posted by @ctoney in #14583 (comment)

[dbaston]

This is confusing as a user but also as a developer. When writing GDAL algorithms I've had to implement workarounds to handle the case where a layer claims to have Polygons but may in fact have MultiPolygons. Some thoughts:

• Are there other drivers that misrepresent the geometry type?
• What would break if the Shapefile driver were updated to always provide MultiPolygons?
• For formats such as GeoPackage/PostGIS, would it be feasible for gdal vector write to create layers with unknown geometry type, and then update the column with a more specific type after creation?

Related:

• Shapefile to FlatGeobuf conversion fails with mixed Polygons and MultiPolygons #2828
• [pgeo] Always promote polygon or line layers to multi-type geometries #4255
• shapefile to postgresql : ERROR: Geometry type (MultiPolygon) does not match column type (Polygon)
• inconsistent geometry when importing shapefiles containing multipolygons into PostGIS
• ogr2ogr shp import to postgres failed

[rouault]

Long known issue (> 20 years :-))

• Are there other drivers that misrepresent the geometry type?

None that I can think of

• What would break if the Shapefile driver were updated to always provide MultiPolygons?

See #2828 (comment)

We could potentially try doing the single->multi auto-promotion by default in the shapefile driver, with an open option to ask for the historic behavior.

• For formats such as GeoPackage/PostGIS, would it be feasible for gdal vector write to create layers with unknown geometry type, and then update the column with a more specific type after creation?

Possibly. For GeoPackage, the current AlterGeomFieldDefn() implementation is:

        // could be potentially done. Requires rewriting the CREATE TABLE
        // statement
        if (poGeomFieldDefn->GetType() != poNewGeomFieldDefn->GetType())
        {
            CPLError(CE_Failure, CPLE_NotSupported,
                     "Altering the geometry field type is not currently "
                     "supported for "
                     "GeoPackage");
            return OGRERR_FAILURE;
        }

For PostGIS, AlterGeomFieldDefn() is implemented for changing geometry type.

[jratike80]

Are there other drivers that misrepresent the geometry type?

Didn't check but I guess that there are lots of formats that can store both, and to be sure the whole dataset should be pre-scanned. GeoJSON, CSV, OpenJUMP JML, GML...
At least for some of them the geometry type is reported as Unknown/Any, but maybe not for all of them.

[jratike80]

would it be feasible for gdal vector write to create layers with unknown geometry type, and then update the column with a more specific type after creation

Not tested, but I think that with GPKG, if there was not a mixture of polygons/multipolygons it would be possible to update just the metadata tables. But if there is a mixture, then the polygons must be re-written as single-part multipolygons, so probably create a new table, insert data, delete the old table, vacuum. Not very effective.

[rouault]

But if there is a mixture, then the polygons must be re-written as single-part multipolygons, so probably create a new table, insert data, delete the old table, vacuum. Not very effective.

I was indeed forgetting this when I mentionned AlterGeomFieldDefn(). Its doc specifies:
"No modification to existing geometries will ever be performed, so this method cannot be used to e.g. promote single part geometries to their multipart equivalents."

[ctoney]

Are there other drivers that misrepresent the geometry type?

This is the MultiSurface case I came across recently with OpenFileGDB -> GPKG. The .gdb is publicly available but ~8 GB. I don't know why it has some MultiSurface mixed in. I believe it should be only MultiPolygon, but it's a curated dataset compiled from many sources. I'm reviewing the dataset but haven't gotten further than converting to GPKG so far.

(ogr2ogr() here is the VectorTranslate() binding in gdalraster, and this was GDAL 3.12.1)

ogr2ogr("LF_Public_Events_1999_2024.gdb", "LF_Public_Events_1999_2024.gpkg")

! GDAL WARNING 1: organizePolygons() received a polygon with more than 100 parts.  The processing may be really slow.  You can skip the processing by setting METHOD=SKIP.
! GDAL WARNING 1: A geometry of type MULTISURFACE is inserted into layer LF2024_Public_ModelReady_Events_CONUS of geometry type MULTIPOLYGON, which is not normally allowed by the GeoPackage specification, but the driver will however do it. To create a conformant GeoPackage, if using ogr2ogr, the -nlt option can be used to override the layer geometry type. This warning will no longer be emitted for this combination of layer and feature geometry type.
! GDAL WARNING 1: A geometry of type MULTISURFACE is inserted into layer LF2024_Public_Raw_Events_CONUS of geometry type MULTIPOLYGON, which is not normally allowed by the GeoPackage specification, but the driver will however do it. To create a conformant GeoPackage, if using ogr2ogr, the -nlt option can be used to override the layer geometry type. This warning will no longer be emitted for this combination of layer and feature geometry type.
! GDAL WARNING 1: A geometry of type MULTISURFACE is inserted into layer LF2024_Public_ModelReady_Events_AK of geometry type MULTIPOLYGON, which is not normally allowed by the GeoPackage specification, but the driver will however do it. To create a conformant GeoPackage, if using ogr2ogr, the -nlt option can be used to override the layer geometry type. This warning will no longer be emitted for this combination of layer and feature geometry type.
! GDAL WARNING 1: A geometry of type MULTISURFACE is inserted into layer LF2024_Public_Raw_Events_AK of geometry type MULTIPOLYGON, which is not normally allowed by the GeoPackage specification, but the driver will however do it. To create a conformant GeoPackage, if using ogr2ogr, the -nlt option can be used to override the layer geometry type. This warning will no longer be emitted for this combination of layer and feature geometry type.

[rouault]

This is the MultiSurface case I came across recently with OpenFileGDB -> GPKG. The .gdb is publicly available but ~8 GB.

Hasn't this been fixed ? or are there really curve parts in the result (FileGDB can indeed have curves, so this is a bit a case similar to shapefiles where we advertize non-curve types to maximize compatibility) ? If there are not, then this is a bug of organizePolygons(), and a ticket + link to the dataset would be appropriate

[ctoney]

I looked at one of the layers (LF2024_Public_ModelReady_Events_AK). It's advertised as a MultiPolygon layer in the .gdb, and converts to MultiPolygon in the GPKG (emitting the warning above). The output layer contains one feature with a MultiSurface geometry (out of 194929 total features, the rest all MultiPolygon):

WKT

"MULTISURFACE (CURVEPOLYGON (COMPOUNDCURVE ((449412.227 1367536.6736,449411.2269 1367537.3482,449411.1752 1367537.3847,449410.0009 1367538.2511,449409.927200001 1367538.309,449409.831800001 1367538.3964,449409.084899999 1367539.1418,449408.1862 1367539.6752,449408.114399999 1367539.7206,449407.1823 1367540.3478,449407.1383 1367540.3786,449405.965500001 1367541.2302,449404.9365 1367541.9263,449404.8082 1367542.0236,449403.872400001 1367542.8216,449403.696599999 1367542.9987,449403.638499999 1367543.0726,449403.1525 1367543.7317,449400.5019 1367545.4925,449400.367699999 1367545.5928,449400.1842 1367545.7761,449398.631899999 1367547.6209,449397.341 1367548.7549,449396.810900001 1367549.1003,449396.657299999 1367549.2151,449395.7498 1367549.9991,449395.5814 1367550.1699,449395.5101 1367550.262,449395.165100001 1367550.7448,449395.0865 1367550.8669,449395.0274 1367550.9801,449394.988600001 1367551.0628,449393.650800001 1367553.5331,449391.5931 1367556.7989,449391.563200001 1367556.8485,449391.4866 1367557.0004,449390.9253 1367558.2914,449390.8914 1367558.3765,449390.843699999 1367558.5337,449390.5834 1367559.5998,449390.141899999 1367560.3294,449390.113700001 1367560.379,449389.464199999 1367561.5794,449389.4191 1367561.6699,449388.958000001 1367562.6792,449388.464299999 1367563.5836,449388.3784 1367563.7703,449388.2346 1367564.1494,449385.780400001 1367564.4928),CIRCULARSTRING (449385.780400001 1367564.4928,449385.595902454 1367564.53051775,449385.4176 1367564.5911),(449385.4176 1367564.5911,449379.5973 1367566.9859,449378.604499999 1367567.4081),CIRCULARSTRING (449378.604499999 1367567.4081,449378.231805004 1367567.63561889,449377.9405 1367567.9609),(449377.9405 1367567.9609,449376.376399999 1367570.3255,449374.9057 1367572.1417,449374.7742 1367572.3324,449373.4967 1367574.5323,449371.9362 1367576.2313),CIRCULARSTRING (449371.9362 1367576.2313,449371.701427949 1367576.57093758,449371.5681 1367576.9617),(449371.5681 1367576.9617,449371.3214 1367578.2404),CIRCULARSTRING (449371.3214 1367578.2404,449372.51806751 1367579.98673265,449374.269200001 1367578.7971),(449374.269200001 1367578.7971,449374.4332 1367577.9476,449375.8168 1367576.4412,449375.861199999 1367576.3907,449376.009299999 1367576.1798,449377.3105 1367573.9391,449378.7533 1367572.1573,449378.8387 1367572.0408,449380.1938 1367569.9921,449380.7523 1367569.7546,449386.3839 1367567.4375,449389.529999999 1367566.9973),CIRCULARSTRING (449389.529999999 1367566.9973,449390.257844356 1367566.68410353,449390.7246 1367566.0438),(449390.7246 1367566.0438,449391.1469 1367564.9305,449391.6171 1367564.0692,449391.664899999 1367563.9738,449392.1269 1367562.9624,449392.735099999 1367561.8383,449393.277000001 1367560.9429,449393.395400001 1367560.6925,449393.4431 1367560.5353,449393.727700001 1367559.3699,449394.192299999 1367558.3014,449396.215399999 1367555.0904,449396.245300001 1367555.0408,449396.2653 1367555.0051,449397.6121 1367552.4923,449397.638800001 1367552.4476,449397.6829 1367552.3739,449397.8467 1367552.152,449398.533600001 1367551.5584,449399.0689 1367551.2097,449399.214299999 1367551.102,449399.2401 1367551.0798,449400.696799999 1367549.8001,449400.854499999 1367549.639,449402.3412 1367547.8722,449405.0328 1367546.0841),CIRCULARSTRING (449405.0328 1367546.0841,449405.237154982 1367545.92109909,449405.41 1367545.725),(449405.41 1367545.725,449405.9505 1367544.9921,449406.686100001 1367544.3647,449407.665899999 1367543.702,449407.706900001 1367543.6732,449408.878699999 1367542.8223,449409.7541 1367542.2332,449410.7773 1367541.6259,449410.9759 1367541.4851,449411.0713 1367541.3977,449411.872099999 1367540.5986,449412.930500001 1367539.8177,449413.9046 1367539.1606),CIRCULARSTRING (449413.9046 1367539.1606,449414.533026235 1367537.60521404,449413.326300001 1367536.4399),CIRCULARSTRING (449413.326300001 1367536.4399,449412.753931972 1367536.44988682,449412.227 1367536.6736))))"

[rouault]

The output layer contains one feature with a MultiSurface geometry:

ok so there's indeed a curve part in it and MultiSurface is legitimate.

[ctoney]

Right it behaves as expected and the warning message is clear about what the driver is doing.

It's an example in the wild besides shapefile where a layer can potentially contain a geometry with different type than advertised.

In terms of the error documentation in PR #14583, it's a case where a user of gdal vector convert would get the warning message that refers to ogr2ogr and nlt, but there's not a nlt option in vector convert. I wonder if gdal vector set-geom-type should be mentioned somewhere in the Migration Guide. The example there for vector convert is shapefile -> GPKG. In practice, gdal vector set-geom-type will often be the conversion command for shapefile polygons. That's such a common conversion and a lot of users coming from the traditional utilities use nlt automatically out of habit.