PBF support?

[jmarca]

This is a resepctful request from a new user to add pbf format support. The osm files take up so much more space than pbf files.

Please close this issue if this has already been discussed and rejected on the mailing list.

[cvvergara]

No promise of when, but we are contemplating pbf support.
Tagged as discussion, to discuss how to do it

[smellman]

How about using libosmium to support pbf.
https://github.com/osmcode/libosmium
I try porting osm-history-importer to use current libosmium.
After my work, I try to rewrite osm2pgrouting with libosmium.

[cvvergara]

Would be nice to have the rewrite by when pgRouting is 3.0 :)
Since pgRouting 2.1, the requirement for c++ compiler is c++0x but osmium needs c++11
So pgRouting we can make minimum requirement to be c++11 also in version 3.0.

[ManuelB]

Hi,
this would be a cool feature. I just downloaded the data for germany. To give you some numbers:

wget https://download.geofabrik.de/europe/germany-latest.osm.pbf (2.7GB)
osmconvert germany-latest.osm.pbf > germany-latest.osm (~50GB)

Only the conversion took about 20 minutes.

osm2pgrouting -f germany-latest.osm -d routing -U postgres -W ****

After 20 minutes I killed the process because I read on the wiki that osm2pgrouting does not support large countries. I will try it with a smaller sample.

/Manuel

[ManuelB]

The following postgresql might help to read data into PostgreSQL from a PBF file:
https://github.com/vpikulik/postgres_osm_pbf_fdw

I didn't used it but want to mention it here. I ended up just reading Sachsen-Anhalt from germany.

[dkastl]

While I don't disagree, that PBF support is a good idea, it doesn't necessarily solve the issue of running out of memory. It definitely helps to work with smaller files though.

Many converters write intermediate processing steps into "temporary", which osm2pgrouting doesn't do. The only thing you can do right now is splitting your single file into smaller areas and process them one by one with --addnodes, which adds new data to an existing table.

There are many possible improvements, some might be not too difficult to implement. Anyone, who wants to contribute new functionality or fund such a feature, is very welcome to do so!

[ManuelB]

@dkastl I fully agree. I already worked a lot with files with multiply hundred mb to multiple gb.

A parallel streaming solution using a SAX or STAX API using multiple transactions and database connection would speed this up a lot.

So if somebody is going to sponsor this feature I would be able to implement it :-)

[cayetanobv]

Hi @ManuelB ,

I think is not true that osm2pgrouting does not support large countries (maybe we need to review wiki). If you have enough memory you can import what you want (even if it is not done in the most efficient way). Here you can see how the same OSM file (germany-latest) is loaded by other person incrementing swap memory: #222

On the other hand If you use osmconvert with drop-author and drop-version flags you can reduce a lot your osm file:

osmconvert output_data.osm.pbf --drop-author --drop-version --out-osm -o=output_data_reduc.osm

Read more about it here: https://github.com/pgRouting/osm2pgrouting#tips

[ManuelB]

@cayetanobv sorry for beeing impatient with osm2pgrouting. The whole routing project is awesome and I was able to produce results within a few hours.

My comment about the support of large files was based on a statement that I read in the wiki:

Can't process "large" files, continents, big countries, or very large states.

https://github.com/pgRouting/osm2pgrouting/wiki/Documentation-for-osm2pgrouting-v2.1

[cayetanobv]

@ManuelB thanks for your interest and feedback. That statement is from an earlier version (sorry for the confusion).