osrm-partition - Failed mapping file: Invalid argument: iostream errorinclude/util/mmap_file.hpp:33

[tsworman]

OS is Windows 7.
Running Docker-Toolkit with boot2docker vm image in Oracle VM VirtualBox 5.2.4.
VM is allocated 28GB of memory, 200gb storage, 4 processors.

Downloaded multiple osm.pbf from both Geofabrik and Mapzen. Successfully re-produced on the Mapzen Detroit file and Geofabrik's North America, USA-Michigan, and Berlin files.

Reproducible with both osrm-backend latest docker image and also 5.14.2. Did not try other version.

Files are located in c:\users\public\mapdata so they are visible within Docker quickstart terminal.
Run the following commands from within that directory. I tried with all the files listed above and both osrm-backend (current) and the tagged v5.14.2. I'm only showing the current commands below.

$ docker run -t -v $(pwd):/data osrm/osrm-backend osrm-extract -p /opt/car.lua /data/detroit_michigan.osm.pbf

Generated this:

[info] Parsed 0 location-dependent features with 0 GeoJSON polygons
[info] Using script /opt/car.lua
[info] Input file: detroit_michigan.osm.pbf
[info] Profile: car.lua
[info] Threads: 4
[info] Parsing in progress..
[info] input file generated by osmconvert 0.8.5
[info] timestamp: 2017-12-25T15:04:02Z
[info] Using profile api version 4
[info] Found 3 turn restriction tags:
[info] motorcar
[info] motor_vehicle
[info] vehicle
[info] Parse relations ...
[info] Parse ways and nodes ...
[info] Using profile api version 4
[info] Using profile api version 4
[info] Using profile api version 4
Can't parse value: no_indications
Can't parse value: no_indications
Can't parse value: no_indications
Can't parse value: no_indications
Can't parse value: default
....
[info] Found 860 SCC (1 large, 859 small)
[info] SCC run took: 0.0610813s
[info] Building r-tree ...
[info] Constructing r-tree of 1344925 segments build on-top of 1218287 coordinat
es
[info] finished r-tree construction in 3.23277 seconds
[info] Writing edge-based-graph edges ...
[info] ok, after 2.06304s
[info] Processed 1711329 edges
[info] Expansion: 24935 nodes/sec and 17336 edges/sec
[info] To prepare the data for routing, run: ./osrm-contract "/data/detroit_mich
igan.osrm"
[info] RAM: peak bytes used: 695767040

So that looked successful. Following the guide here (https://hub.docker.com/r/osrm/osrm-backend/) I next run:
$ docker run -t -v $(pwd):/data osrm/osrm-backend osrm-partition /data/detroit_michigan.osrm

Which generates the error:

[info] Computing recursive bisection
[info] Loaded compressed node based graph: 908644 edges, 1218287 nodes
[info] running partition: 128 1.2 0.25 10 1000 # max_cell_size balance boundary cuts small_component_size
[info] Found 890958 SCC (1 large, 890957 small)
[info] SCC run took: 0.0660343s
[info] Full bisection done in 22.2809s
[info] Loaded node based graph to edge based graph mapping
[info] Loaded edge based graph for mapping partition ids: 3422078 edges, 847033 nodes
[info] Fixed 153237 unconnected nodes
[info] Edge-based-graph annotation:
[info] level 1 #cells 3884 bit size 12
[info] level 2 #cells 307 bit size 9
[info] level 3 #cells 21 bit size 5
[info] level 4 #cells 1 bit size 1
terminate called after throwing an instance of 'osrm::util::exception'
what(): File "/data/detroit_michigan.osrm.fileIndex" mapping failed: failed mapping file: Invalid argument: iostream errorinclude/util/mmap_file.hpp:33

[daniel-j-h]

As discussed in IRC this looks like a Windows + Docker-Toolkit problem.

In the meantime try using a Linux box or use the native Windows binaries.

As for next actions here: someone with a Windows box needs to debug this (cc @oxidase ?).

[oxidase]

@tsworman @daniel-j-h the issue is caused by a mmap fail in osrm-partition with PROT_WRITE and MAP_SHARED flags.

open("/data/Downloads/monaco-latest.osrm.fileIndex", O_RDWR) = 3
fstat(3, {st_mode=S_IFREG|0777, st_size=97520, ...}) = 0
mmap(NULL, 97520, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = -1 EINVAL (Invalid argument)

At the same time osrm-routed without PROT_WRITE flag works fine

open("/data/Downloads/monaco-latest.osrm.fileIndex", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0777, st_size=97520, ...}) = 0
mmap(NULL, 97520, PROT_READ, MAP_SHARED, 3, 0) = 0x7f65a2d27000

Similar issue https://stackoverflow.com/a/18421071 and potentially related issue in VirtualBox https://www.virtualbox.org/ticket/819

A workaround fix can be a read-modify-write renumbering instead of the current inplace one
/cc @TheMarex

[ibrt]

Interesting, I ran into this issue using a Ubuntu guest in VirtualBox on macOS host. I was running the Docker OSRM container in the guest OS. I figured it happens when the fileIndex is inside a VirtualBox shared directory. If I put the fileIndex somewhere else, it works fine. Likely some issue with vboxfs.

[grib0]

Probably linked to #5086 a chmod 644 fileIndex should work

[alitarraf]

EDIT: I uninstalled docker toolbox, and virtual box. Then upgraded to WIN 10 PRO. Then installed Docker for Windows. Everything works like a charm now with Hyper-V. I think the error is related to virtual box/windows. Dont have any errors now.

I am having a similar issue. Running Win 10 Home, with Docker Toolbox. However, I get an mmap_file.hpp:60 failure after the R-tree construction in the osrm-extract step.
I have tried latest docker image and v5.21.0 and v5.20.0 with similar result.

Here is my console output.
PS D:\Documents\Projects\OSRM-backend\try-docker\data> docker run -t -v //d/Documents/Projects/OSRM-backen d/try-docker/data/:/data osrm/osrm-backend:v5.20.0 osrm-extract -p /opt/car.lua /data/monaco-latest.osm.pb f [info] Parsed 0 location-dependent features with 0 GeoJSON polygons [info] Using script /opt/car.lua [info] Input file: monaco-latest.osm.pbf [info] Profile: car.lua [info] Threads: 1 [info] Parsing in progress.. [info] input file generated by osmium/1.8.0 [info] timestamp: 2019-03-01T21:15:02Z [info] Using profile api version 4 [info] Found 3 turn restriction tags: [info] motorcar [info] motor_vehicle [info] vehicle [info] Parse relations ... [info] Parse ways and nodes ... [info] Parsing finished after 0.328125 seconds [info] Raw input contains 43726 nodes, 3175 ways, and 26 relations, 44 restrictions [info] Sorting used nodes ... ok, after 0.000231s [info] Erasing duplicate nodes ... ok, after 1.5e-05s [info] Sorting all nodes ... ok, after 5.9e-05s [info] Building node id map ... ok, after 8.3e-05s [info] Confirming/Writing used nodes ... ok, after 0.00833s [info] Writing barrier nodes ... ok, after 0s [info] Writing traffic light nodes ... ok, after 0s [info] Processed 4836 nodes [info] Sorting edges by start ... ok, after 0.000303s [info] Setting start coords ... ok, after 0.000297s [info] Sorting edges by target ... ok, after 0.000257s [info] Computing edge weights ... ok, after 0.001121s [info] Sorting edges by renumbered start ... ok, after 0.000344s [info] Writing used edges ... ok, after 0.000918s -- Processed 5056 edges [info] Writing way meta-data ... ok, after 0.000297s -- Metadata contains << 837 entries. [info] Sorting used ways ... ok, after 7.1e-05s [info] Collecting start/end information on 0 maneuver overrides...ok, after 8e-06s [info] Collecting start/end information on 0 maneuver overrides...ok, after 2e-06s [info] Collecting start/end information on 44 restrictions...ok, after 3.5e-05s [info] Collecting start/end information on 44 restrictions...ok, after 3e-05s [info] writing street name index ... ok, after 0.001513s [info] extraction finished after 0.366118s [info] Generating edge-expanded graph representation [info] . 10% . 20% . 30% . 40% . 50% . 60% . 70% . 80% . 90% . 100% [info] Node compression ratio: 0.167494 [info] Edge compression ratio: 0.20214 [info] graph compression removed 629 annotations of 837 in 0.000332 seconds [info] Find segregated edges in node-based graph ... [info] ok, after 0.003219s [info] Segregated edges count = 27 [info] Writing nodes for nodes-based and edges-based graphs ... [info] Geometry successfully removed: compressed edges: 2100 compressed geometries: 10102 longest chain length: 46 cmpr ratio: 0.20788 avg chain length: 4.81048 [info] Generating edge expanded nodes ... [info] . 10% . 20% . 30% . 40% . 50% . 60% . 70% . 80% . 90% . 100% [info] Expanding via-way turn restrictions ... [info] [info] Generated 1445 nodes (0 of which are duplicates) and 5046 segments in edge-expanded graph [info] Generating edge-expanded edges [info] . 10% . 20% . 30% . 40% . 50% . 60% . 70% . 80% . 90% . 100% [info] Sorting and writing 0 maneuver overrides... [info] done. [info] Renumbering turns [info] Writing 0 conditional turn penalties... [info] Generated 5046 edge based node segments [info] Node-based graph contains 1445 edges [info] Edge-expanded graph ... [info] contains 2131 edges [info] Timing statistics for edge-expanded graph: [info] Renumbering edges: 4.9e-05s [info] Generating nodes: 0.007678s [info] Generating edges: 0.058718s [info] Generating guidance turns [info] . 10% . 20% . 30% . 40% . 50% . 60% . 70% . 80% . 90% . 100% [info] done. [info] Created 15 entry classes and 476 Bearing Classes [info] Handled: 7 of 31 lanes: 22.5806 %. [info] Assigned 2631 turn instruction types: [info] new name: 179 (6.80%) [info] continue: 201 (7.64%) [info] turn: 766 (29.11%) [info] fork: 48 (1.82%) [info] end of road: 223 (8.48%) [info] enter roundabout: 62 (2.36%) [info] enter and exit roundabout: 6 (0.23%) [info] enter rotary: 3 (0.11%) [info] enter and exit rotary: 2 (0.08%) [info] enter roundabout turn: 15 (0.57%) [info] (noturn): 301 (11.44%) [info] (suppressed): 631 (23.98%) [info] roundabout: 6 (0.23%) [info] exit roundabout: 76 (2.89%) [info] rotary: 2 (0.08%) [info] exit rotary: 7 (0.27%) [info] exit roundabout turn: 14 (0.53%) [info] (stay on roundabout): 89 (3.38%) [info] Assigned 2631 turn instruction modifiers: [info] uturn: 154 (5.85%) [info] sharp right: 54 (2.05%) [info] right: 541 (20.56%) [info] slight right: 179 (6.80%) [info] straight: 984 (37.40%) [info] slight left: 190 (7.22%) [info] left: 441 (16.76%) [info] sharp left: 88 (3.34%) [info] Guidance turn annotations took 0.084691s [info] Writing Intersection Classification Data [info] ok, after 0.002845s [info] Writing Turns and Lane Data... [info] ok, after 0.004326s [info] Saving edge-based node weights to file. [info] Done writing. (0.001548) [info] Computing strictly connected components ... [info] Found 67 SCC (1 large, 66 small) [info] SCC run took: 6.4058e-05s [info] Building r-tree ... [info] Constructing r-tree of 5046 segments build on-top of 4836 coordinates terminate called after throwing an instance of 'osrm::util::exception' what(): File "/data/monaco-latest.osrm.fileIndex" mapping failed: failed mapping file: Invalid argument: iostream errorinclude/util/mmap_file.hpp:60

[prusswan]

Another workaround is to mount the share dir, but copy the contents to a location within the container, i.e. don't perform I/O within the mounted location. This is useful if the shared contents are not too big in filesize.

[Kruszylo]

Hi, I'm experiencing similar error when I try to run docker container:

docker run --rm -ti -v /home/docker/osm:/osm -p 5000:5000 -p 3000:3000 -e CPUS=4 -e DATA_PATH=/osm -e OSM_PBF_URL=http://download.geofabrik.de/europe/poland/malopolskie-latest.osm.pbf -e SHAREDMEMORY=0 kruszylo/vroom-server

Error is:

.                                                                                                                                                                                                     
 100%                                                                                                                                                                                                 
[info] Contracted graph has 1817014 edges.                                                                                                                                                            
[info] Contraction took 114.926 sec                                                                                                                                                                   
[info] Preprocessing : 115.117 seconds                                                                                                                                                                
[info] finished preprocessing                                                                                                                                                                         
[info] RAM: peak bytes used: 193404928                                                                                                                                                                
[info] starting up engines, v5.22.0                                                                                                                                                                   
[info] Threads: 4                                                                                                                                                                                     
[info] IP address: 0.0.0.0                                                                                                                                                                            
[info] IP port: 5000                                                                                                                                                                                  
terminate called after throwing an instance of 'osrm::util::exception'                                                                                                                                
  what():  File "/osm/osrm/processed/car/malopolskie-latest.osrm.fileIndex" mapping failed: failed opening file: Permission denied: iostream errorinclude/util/mmap_file.hpp:33                       
2019-07-12 20:27:42,743 INFO exited: osrm (terminated by SIGABRT; not expected)                                                                                                                       
2019-07-12 20:27:43,746 INFO spawned: 'osrm' with pid 53                                                                                                                                              

[Kruszylo]

Guys, I solved my problem, I needed to make chmod -R 775 /osm* to allow read from compiled files. Docker image should works now (if anybody interested).

[denplis]

workflow with Docker Toolbox
Moving berlin-latest.osm.pbf inside image will help.

Dockerfile

FROM osrm/osrm-backend
COPY berlin-latest.osm.pbf /berlin-latest.osm.pbf
RUN osrm-extract -p /opt/car.lua /berlin-latest.osm.pbf
RUN osrm-partition /berlin-latest.osrm
RUN osrm-customize /berlin-latest.osrm

CMD osrm-routed --algorithm mld /berlin-latest.osrm

docker build -t my_osrm .
docker run -t -i -p 5000:5000 my_osrm