[WIP] Extendability #41
Conversation
The runner.go file is the new benchmark entry point. The implementation is currently neither feature complete nor completely correct, which will be fixed and refactored in a following commit.
The new benchmark and test drivers allow for better extendability. Benchmarks are split into different types of benchmarks, e.g. ParameterReadWrite which benchmarks the reading and writing of a named path segment. These types are the only shared information between benchmarks and Router implementations. A Router implementor specifies the types of benchmarks a Router can run as well as other metadata, such as Router name and homepage URL, during registration. Further allows the new driver implementation to run tests for each benchmark, which uncovered more errors than the current test implementation.
Refactored benchmark fixtures to use the new driver and implemented test cases for each benchmark.
The Possum implementation was previously broken. Possum’s router.Simple is not sufficient for dynamic path segment routing. Further makes Possum’s current implementation of router.RegEx it impossible to benchmark ParameterReadWrite. ParameterReadWrite should be re-enabled once Possum implements named RegEx groups.
Revels implementation contains now more internals directly copied from the source code because without it the tests fail constantly with „nil pointer dereference“ errors.
Zeus is currently disabled. Enable again once daryl/zeus#2 is fixed.
|
Here is a preview of the generated markdown and CLI output. Note that the highlighting algorithm is broken. |
|
Awesome! I haven't looked over all your changes yet, but the idea is great. You can already run certain benchmarks only with e.g. |
Yes, I know. I switched PCs quite a bit at that time and I tried to avoid downloading all dependencies on every machine. ~~While I ran the benchmarks I noticed that most of the execution time (~50%-60%) seems to come from invoking the GC to calculate the memory consumption of a Router object. Changing Note that even with the new concurrent GC in 1.5, invoking I profiled the benchmark runner and found that the overhead in relation to the ideal execution time is not from the GC, which runs in 2ms to 5ms. The estimator in the |
|
? |
|
Whoops, delete the wrong branch and thanks to wrong aliasing on the current machine I pushed it upstream. |
Earlier this year I wanted to benchmark my internal Router implementation with your benchmarking suite. I looked for a way to disable other Routers, so I don't have to
go getthe packages. But as everything was basicly implemented in one big file, there was no way to avoid these dependencies. So I split everything into seperate files with build tags. During this step I discovered that not all implementations work correctly (you fixed most of them as you included a simple test case), which led me to implement a new benchmark runner.The new benchmark and test drivers allow for better extendability. Benchmarks are split into different types of benchmarks, e.g.
ParameterReadWritewhich benchmarks the reading and writing of a named path segment. These types are the only shared information between benchmarks and Router implementations. A Router implementor specifies the types of benchmarks a Router can run as well as other metadata, such as Router name and homepage URL, during registration. Further allows the new driver implementation to run tests for each benchmark, which uncovered more errors than the current test implementation.Errors found by the new test driver:
router.Simpleis not sufficient for dynamic path segment routing. Further makes Possum’s current implementation ofrouter.RegExit impossible to benchmarkParameterReadWrite.Also, I removed the RegEx call inside Beego's constructor - it does not interfere with the benchmarks but is still unnecessary.
Todo:
runner.goas it is quit messy!to invert) for Router