Snapshot routing tests

[flaktack]

Snapshot test for the routing could be added to increase the test coverage, verify that modifications to various parts of the code base produce the expected changes in routing results.

As an example removing STL_TRAVERSE_COST from incrementTimeInSeconds() in #3164 caused an expected 1 second difference, which snapshot test would have made explicit in the changes to the saved trip plans.

#3190 contains an example of this based on json-snapshot, this is to continue the discussion.

[flaktack]

json-snapshot adds an external dependency and has a few drawbacks:

• files must manually be deleted
• a single file contains the json for all tests in a class
• jackson versions differ slightly which may cause problems

Instead of using an external library an alternative could be to have a simple class to create and verify the snapshots, which would give us greater control over the process:

• there could be a maven profile to recreate the snapshots
• each test could be in its own json file instead of one large file for each test class

Would having the snapshot testing code in-tree, with easier regeneration of the tests be a better approach?

[t2gran]

Thank you for raising this issue - I do agree that OTP lack automatic tests. For the past 2 years one of my goal have been to push for improving the design and architecture of OTP. We have very limited resources, so we need to prioritize what we focus on.

The thing with API tests is that they are expensive TestPyramid. They tend to be less expensive compared to manual testing, but is much more expensive than unit-testing. I would like to se what I call "acceptance-tests" on the API, verifying that we get results - and that there is no braking changes in the API. But, I do not want to verify business logic, coverage or data. If a test fails it should pinpoint where the problem is right down to the exact line of code - this is impossible with API tests. By reducing the scope tests become more precise. Tests on business logic should be on function, classes and modules - unit tests. To me it is very important that we do not have false positives. We have had a few tests that have cased a lot of noice in the past. These tests steel time from everyone, not just the developer who made a valid change to the code. And sometimes the test fails because the runtime environment is different.

The future of the REST API is uncertain, we would like to replace it with a new Schema GraphQL API. One of the reasons is get better control of API changes - using a schema eliminate part of the need for testing the API. We can add smoke tests and be fairly sure everything works, using unit tests for testing business logic.

On the positive side I have to admit that we have had problems where the API is changed/broken without knowing it. But, it comes down to what is least expensive.

I am not familiar with the library you use for these API tests, so I can not say if my concerns are valid.