Fix json rendering of large osm ids

[imannamdari]

Issue

Fix #7016

Tasklist

• CHANGELOG.md entry (How to write a changelog entry)
• update relevant Wiki pages
• add tests (see testing documentation)
• review
• adjust for comments
• cherry pick to release branch

Requirements / Relations

Link any requirements here. Other pull requests this PR is based on?

[afarber]

I have tried with a locally built docker/Dockerfile-alpine and the fix works for me, for example for the query below:

wget -O- 'http://127.0.0.1:5000/match/v1/driving/13.416270,52.526382;13.416849,52.526115;13.416849,52.525750;13.416538,52.525430;13.416184,52.525123;13.415755,52.525306;13.415154,52.525567;13.414596,52.525802?overview=false&generate_hints=false&gaps=ignore&annotations=nodes'

[afarber]

Is the if/else block really needed? Can't a single line below handle the big integer number up to 18446744073709551615 (which is 2^64-1)?

fmt::format_to(std::back_inserter(buffer), FMT_COMPILE("{}"), static_cast<std::uint64_t>(number.value));

[afarber]

Ah, nevermind, I see that it would break the fractional numbers

[imannamdari]

Exactly. For floating-points there gonna be problems.
The previous approach implemented before this commit can be found here (and it works fine):

osrm-backend/include/util/json_renderer.hpp

Line 48 in 376282d

void operator()(const Number &number)

A more robust solution could be separating floats from integers. Instead of Number struct, we can have Float and Integer structs for example. Each with their own json rendering.
However, that's a significant change! :D
If you agree with this approach (separating floats from integers), I can work on it in another pull request.

[afarber]

Dear @DennisOSRM and @SiarheiFedartsou now that CI build is working again, could you please consider merging this fix by @imannamdari ?

In the past week I have tested his fix with map matching and it really seems to fix the problem with large OSM node/way ids.

And please consider adding a new git tag v5.28.1 after the merge?

Because the current tag v5.27.1 is already three years old and it would be nice to set a newer stable source code "checkpoint" for all users.

[DennisOSRM]

new release coming soon

[MarcelloPerathoner]

I have proposed a solution with separate treatment of OSM ids as uint64_t in #7142.

[imannamdari]

@DennisOSRM whats your opinion? should we temporarily change precision from 10 to 11 to solve large osm ids rendering, and then working on separating json number struct to two integer and double structs for more scalable solution?

That's a workaround that would buy some time to do the other changes I think. Any more code changes necessary to do it?

It seems that, for now, the maximum osm id is around 13,000,000,000. I increased the precision from 10 to 12, which should buy us a few more years.

After merging this, the bug should no longer appear in our upcoming release. In the meantime, we can work on @MarcelloPerathoner 's PR to implement a better solution for this issue.

[MarcelloPerathoner]

The problem with this patch is that it changes the format for all numbers. It adds even more digits where 9 digits should be amply sufficient. And this still doesn't fix the problem for people that use high IDs for private extensions.

[imannamdari]

f 9 digits are enough to represent a number, it will produce 9 digits (it won’t expand it to 12 digits). This is the difference between .g and .f (see the tests).

Hmm, I didn’t know about private extensions. Previously, you mentioned that as a short-term solution, instead of casting, we should simply increase the precision from 10 to 11. If this increase doesn’t solve the issue, and casting is an expensive operation (which I personally don’t believe it is, since I deployed this code in production with over 10k RPS, and the CPU usage is no different from the without-cast solution), then we don’t have a viable short-term solution.

[MarcelloPerathoner]

Private extensions are mentioned here: #7069