Check frameworks for HTTP compliance

[Kaliumhexacyanoferrat]

Currently there are a lot of frameworks marked as realistic that do not provide a proper HTTP webserver implementation but just respond to the test tools in the expected manner. As they skip a lot of scenarios a real-world web server has to handle, they are of course faster.

After the discussion in #9391 I've created a project intended to check a HTTP web server for compliance with the HTTP specification. This is a very rough draft because this is a functionality not very much projects would need - so I am willing to spend more time only, if you are interested in integrating such tool into your test framework. The executable will take an HTTP endpoint and return a status code whether the server is compliant or not.

You can find the project here: https://github.com/Kaliumhexacyanoferrat/http-conformity

The information provided by the tool could be used in various ways:

• Add another filter to the results dashboard
• Do not allow frameworks that are not compliant to be marked as realistic
• Fail the build if a non-compliant framework is added
• Exclude non-compliant frameworks

Please let me know what you think :)

[joanhey]

We created our own functional tests for Adapterman and Workerman.
https://github.com/joanhey/AdapterMan/actions/runs/11104804508

All the tests:
https://github.com/joanhey/AdapterMan/actions/runs/11104804508/job/30849613841#step:5:1

As only 1 test is specific to PHP, recently we have move it to it's own project to use with any server or app server.
And we are working in selecting which tests to run, it's easy (WIP).
https://github.com/joanhey/TestMan

Actually we have 103 tests and 511 assertions.

But for run the tests we need a file in the server to return the values.
That's not possible to add in all the frameworks from this bench.

So with the code that we have from this bench, the number of tests that we can run it's very small and never will be realistic. Only will detect the more radical implementations.

[Kaliumhexacyanoferrat]

Nice, thanks - yes, without the server fulfilling some contract we can test only a few assertions. Nevertheless detecting and disabling radical implementations on official rounds already would be a great improvement as they tend to be on top - so I would encourage such an usage of your suite with just a few of the tests.

[joanhey]

We can start a list of the tests, that we can use in this bench:

• 404 Not Found
• 400 Bad Request
• POST method and return the same that with the GET
• HEAD method and return the body
• Accept-Encoding: gzip Request, server need to return a gzip response
• ...

[Kaliumhexacyanoferrat]

I think it's fine for a server to return 200 OK for a non-existent URL so the tests need to be very defensive - but yes, based on the test spec we should be able to find some things to be tested.

Some frameworks have very simplified socket handling (e.g. they expect the full request to be already in the buffer), but this would require something different than a HTTP client - the approach above would probably be fine.

[joanhey]

A non-existent URL is always a 404 Not Found !!

[joanhey]

Another problem that we had some years ago, and still now, it's that some frameworks return a compressed response always.
Technically a server only need to return a compressed response if the client send the Accept-Encoding: header.
And this bench never send this header in the Request (I think so). If the bench send it, we need to change this behavior.

Added to the list, only with gzip to be defensive.

[joanhey]

@NateBrady23 what do you think about this test list ?

[NateBrady23]

@joanhey This is a good list! I also like the idea of failing the build if the test is marked as realistic but fails this list.