type = waterway is missing

[bomilkar]

I see this on the map:
https://www.openstreetmap.org/relation/2574854#map=11/48.7795/12.2539

But not this:
https://www.openstreetmap.org/relation/410545#map=11/48.7397/12.1735

Both are:
type=waterway
waterway=river

Why is the first creek on the map and not the 2nd ??

[bomilkar]

I'm trying to figure out what's happening. I guess I could use some help, please.

Following the instructions in the README.md I imported niederbayern-260224.osm.pbf . My machine is just about big enough for this. And the creeks in question are in Niederbayern.

I ran ./reduce_water.py
After that I see entries in the table:

osm=# select osm_id, name from water_lines where name like '%Laber';
   osm_id   |     name     
------------+--------------
   29281250 | Kleine Laber
   30587553 | Kleine Laber
  113792995 | Kleine Laber
  536562088 | Kleine Laber
  536562089 | Kleine Laber
   39473405 | Große Laber
 1233703582 | Kleine Laber
   30736605 | Große Laber
  817836052 | Kleine Laber
   39704154 | Kleine Laber
   29288773 | Kleine Laber
   32316454 | Kleine Laber
  319326292 | Kleine Laber
   45261095 | Kleine Laber
   32764539 | Kleine Laber
   32959294 | Kleine Laber
 1290705704 | Kleine Laber
  339314138 | Kleine Laber
   30587421 | Kleine Laber
   31004551 | Kleine Laber
  339314139 | Kleine Laber
  839379276 | Kleine Laber
   28950313 | Alte Laber
 1363504896 | Große Laber
  839379277 | Kleine Laber
  817836053 | Kleine Laber
   89124633 | Kleine Laber
 1447840111 | Große Laber
(28 Zeilen)

osm=#

But running ./export_water.py fails:

df = pd.read_sql(
/usr/lib/python3/dist-packages/geopandas/io/sql.py:473: UserWarning: pandas only supports SQLAlchemy connectable (engine/connection) or database string URI or sqlite3 DBAPI2 connection. Other DBAPI2 objects are not tested. Please consider using SQLAlchemy.
return pd.read_sql(spatial_ref_sys_sql, con)
Traceback (most recent call last):
File "/media/data2/xcsoar/xcsoar-mapgen-topology/scripts/./export_water.py", line 49, in <module>
  gdf_small.to_file(
  ~~~~~~~~~~~~~~~~~^
      output_shapefile_small, driver="ESRI Shapefile", encoding="ISO-8859-1", crs=crs
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  )
  ^
File "/usr/lib/python3/dist-packages/geopandas/geodataframe.py", line 1536, in to_file
  _to_file(self, filename, driver, schema, index, **kwargs)
  ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/geopandas/io/file.py", line 686, in _to_file
  _to_file_pyogrio(df, filename, driver, schema, crs, mode, metadata, **kwargs)
  ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/geopandas/io/file.py", line 742, in _to_file_pyogrio
  raise ValueError("Passing 'crs' is not supported with the 'pyogrio' engine.")
ValueError: Passing 'crs' is not supported with the 'pyogrio' engine.

It seems as if some python packages are wrong or missing.
Unfortunately pip install -r requirements.txt is not available because there is no requirements.txt

[bomilkar]

I got 1 step further:
Instead of this:

gdf_small.to_file(
      output_shapefile_small, driver="ESRI Shapefile", encoding="ISO-8859-1", crs=crs
  )

Use this:

gdf_small.to_file(
      output_shapefile_lines, driver="ESRI Shapefile", encoding="ISO-8859-1", crs=crs, engine="fiona"
  )

And there is a typo. It's "water_line.shp" not "water_lines.shp"
output_shapefile_lines = output_dir + "water_line.shp"

It generates the expected files. BUT while running it complains:
RuntimeWarning: One or several characters couldn't be converted correctly from UTF-8 to ISO-8859-1.
It might stumble on the 'ß' in "Große Laber".

To get "Große Laber" into the map, build table water_lines like this:

DROP TABLE IF EXISTS water_lines;
CREATE TABLE water_lines AS
SELECT osm_id, name, ST_Simplify(way, 15) AS way
FROM planet_osm_line
WHERE waterway IN ('river','stream','canal');

However, I wonder where the bounding box comes into play.
I have not yet seen it in the any of the scripts.
Please help!

[lordfolken]

I find some time tonight. I fear the libaries moved on from this code.

[bomilkar]

I find some time tonight. I fear the libaries moved on from this code.

Take your time. This isn't urgent. We can look at it after the release is out.
I'm just collecting the issues as I dig into it.

[bomilkar]

I find some time tonight. I fear the libaries moved on from this code.

@lordfolken : can you point me to the actual reduce* and export* scripts, please. That would allow me to continue working on it.

I successfully imported from here: https://download.geofabrik.de/europe/germany/bayern.html
This is all of Bavaria, which should be good enough for preliminary tests. If that works, the rest-of-world should be close.

[bomilkar]

@lordfolken : just a friendly reminder: can you point me to the actual reduce* and export* scripts, please. That would allow me to continue working on it.

[lordfolken]

Heya sorry for the late reply:

So the script that selects river here:

xcsoar-mapgen-topology/scripts/reduce_water.py

Lines 22 to 34 in 657462b

	cur.execute(
	"""
	DROP TABLE IF EXISTS water_polygons_small;
	DROP TABLE IF EXISTS water_polygons_large;
	CREATE TABLE water_polygons_small AS
	SELECT osm_id, name, ST_SimplifyPreserveTopology(way, 15) AS way
	FROM planet_osm_polygon
	WHERE ("natural" = 'water' OR "landuse" = 'reservoir' OR "landuse" = 'basin')
	AND ST_Area(way) >= 1000 AND ST_Area(way) < 300000;
	CREATE TABLE water_polygons_large AS
	SELECT osm_id, name, ST_SimplifyPreserveTopology(way, 15) AS way

There isn't really a bounding box, there is just a surface area selector so we don't select every pond on the planet..

AND ST_Area(way) >= 1000 AND ST_Area(way) < 300000;

[lordfolken]

but i see these are just lakes not rivers...

[lordfolken]

https://mapgen-data.sigkill.ch/manifest defines the layers and their datasets for the mapgenerator

[bomilkar]

There isn't really a bounding box, there is just a surface area selector so we don't select every pond on the planet..

AND ST_Area(way) >= 1000 AND ST_Area(way) < 300000;

1. that means the *.shp files for GER include every object on the planet (or if the manifest has any effect, from Europe: { "name": "vmap0/eur", "bounds": [ -35, 180, 90, 30] }. This would explain their sizes.)

2. can you confirm the units of ST_Area() being ft² ? and ST_Simplify() to be ft ?

[lordfolken]

1. I never had a need for a bounding box, as i was importing planet.osm into a postgres server with postgis extensions. (Warning: 64GB Ram needed at the minimum..)

2. https://postgis.net/docs/ST_Area.html yes, its freedom units unfortunately.

[bomilkar]

After a lot of analysis I come to the conclusion the scripts I'm looking at here are not the ones being used by the map generator available at https://mapgen.xcsoar.org/

1. there MUST be a bounding box which I can't discover in the scripts here.
2. the bounding box implemented in https://mapgen.xcsoar.org/ is not working as expected.

For demonstration purposes I'm looking at the *.shp files in various xcm files.
I'm comparing water_line.shp from 2 different xcm files:

The current xcm file was generated today. The info.txt file:

map name: CentralEurope_v2026_04
generator: XCSoar Map Generator
creation time: 01.04.2026 10:37:35 (1775032655)
latitude range: 47.0 to 55.2
longitude range: 5.5 to 24.0
author: ronald_niederhagen@freenet.de

The reference xcm file was generated in 2022. The info.txt file:

map name: central_europe_2022_2
generator: XCSoar Map Generator
creation time: 24.06.2022 11:45:16 (1656063916)
latitude range: 47.0 to 55.2
longitude range: 5.5 to 24.0
author: ronald_niederhagen@freenet.de

To get an overview of which area is covered by the shp files I'm using https://www.filestash.app/tools/shp-viewer.html to visualize water_line.shp an compare the 2 map files.

For the reference map file it looks like this:

For the map file created today it looks like this:

It is obvious:

1. there is an active bounding box for both cases, which is reasonable and expected but nowhere to find in any of the scripts here. (Expected in the reduce_*.py scripts.)
2. the bounding box for the new map generator is broken. (The reference map has the expected area covered.)

So I don't see how I can continue debugging (or improving) the current map generator.

[bomilkar]

@lordfolken just a friendly reminder: I'd like to continue working on the issue.
Where can I find the scripts, etc. which are used by the map generator. Obviously it's not this here.