Stateful Vs. Stateless Automatic Tests
While we were working on updating ermestjs and chaise test cases to work with the stateful branch of ermrest, we noticed an increase in the test runtime.
To investigate it, we decided to run some experiments in ermrestjs first, to see how different the running time is in different scenarios. The following summarizes the test runtimes for ermestjs.
To just test the setup up without any test cases, I replaced all the specs that we had with just one dummy spec which wouldn't take longer than a few milliseconds. The +5Cols were supposed to test the same schema/table/column size since in stateful we're always injecting the five system columns. This shows increase both in the test case runtime and setup. Setup increase makes sense because:
-
We're creating each table individually which is slow and ermest has to do some bookkeeping for each of them. We could improve this by using the new bulk table creation API in ermrest.
-
We're creating each foreign key individually which again is expensive and can be improved with the new ermrset API.
The increase in test case is more than what we assumed it would be. This much of an increase would have made sense if our test cases were mainly doing writes, which is not the case.
I also tested the runtime by omitting RMT (disabling history logic), but that didn't make a difference.
Running multiple experiments in chaise is not easy because of the inconsistency of build time. But this is an estimate of build time in different scenarios:
The result is as expected. The test case are the majority of runtime here and the stateful branch is taking longer than the stateless.