Missing relations members in daily diffs

[flacombe]

Hello

I was investigating an issue in which relations got from daily diffs were lacking of geometry.
Geometry is missing because unedited members are missing in the daily diff, which is major issue I didn't thought I could face to.

As we certainly don't want to fall back to daily init from updated OSH, I have ton find a solution for such problem, including getting individual information from overpass or something.

One question remains to me: how can imposm/osm2pgsql handle such situation on a topic focused database that will lack many members to maintain consistency?
Would it exist full and partial diffs, the first including members?

Here is what it means practically:
On the 2025-11-27 diff:

• This relations has been edited: https://www.openstreetmap.org/relation/6221839
• This member is not present in the diff: https://www.openstreetmap.org/way/85224545

So the geometry associated to the relation can't be rebuild and therefore the relation can't be linked to boundaries.

[PanierAvide]

Don't we have also a PBF which should be quite recent in cache ? We could give a check at it to find missing objects and avoid relying on Overpass.

[flacombe]

Yes but in this direction we should maintain the PBF up to date everyday and that's not scalable.
But imposm and osm2pgsl don't need this, so there is another solution

Typical use case:
50 nodes already exist in the database since 2 years. One day, someone shows up and creates a way that matches the imposm filter, involves the 50 nodes but doesn't change their tags nor position. The diff will contain the way, how renders will get the nodes?

[flacombe]

After weekend investigations here are the facts:

• Osm2pgsql and imposm maintain local indices that hold the whole pbf file processed during import.
• Then, daily diffs doesn't contain referenced features as local importer will dig into their indices to find them
• Any import with any tags will weight the same on disk as the indices will be the same (~7GB for France, 1 TB for world)
• Currently, bogus podoma world scale projects for MapYourGrid weight 67 GB

As it's out of question to switch from 67 GB to 1 TB for a very few features, the possible solution for fix are:

• Look for unresolved features in each project and call for Overpass (up to each podoma instance to choose the right Overpass server for their needs)
• As some projects may overlap in the same instance (for mapyourgrid most of the lines supports are found in another project) we could resolve locally some missing features with another project. That should be optional but will reduce the amount of features asked to Overpass
• Create another kind of daily diffs to retrieve referenced features
• Switch to imposm or osm2pgsql

The last one is directly discarded for disk usage reasons