Conversation
…ements; updated tests to account for parallel_id in container names
…orker session; more cleanup functions editing so parallel tests can run; made test_jobs sequential (only way to ensure they are run on the same worker)
…d for ensuring test_jobs.py are in the same group; updated documentation and pytest invocation command
|
I do understand it is not ready yet. Two questions still:
|
- base test/tmp directory remains, only subfolders are cleaned - updated run-tests script with new pytest run commands - added dummy metric providers for tests (same as cpuprocfs provider, but skips base.py check_system as that causes collisions during paralleizations) - temporarily turn off test_volume_loading::test_volume_loading_subdirectories_subdir2 test as that is failing for unknown reasons - added serial pytest mark for tests that cannot be parallelized (test_jobs) - removed now unnecessary jobs table cleanup in test_jobs - removed duplicate test in test_jobs
Aye, I can make drafts for the future
I think there was a miscommunication here, I thought we had agreed that the runner.py implementation was to be changed? Patching the usage_scenario will not be enough, as we also need to change the /tmp/green-metrics-tool folder that gets used to be unique. The discussion regarding patching the usage_scenario was in response to a checking I was doing against the container names from the flow section to match against the parallel_id. You pointed out that patching the object at runtime would make more sense than regex matching, but I thought this was still in context of within the runner.py - which is what I've implemented. We also had a discussion about patching the usage_scenario directly regarding this ticket: #653, but that was entirely unrelated |
…s and examples directory tests
…ly in test_functions
…ctions to helper files; test_function's setup_runner now parallizes loaded yml files;
…h parallelization correctly; gmt tmp folders now have parallel id pre-pended, not appended; fix issue with yml dumping during parallelization where key order was not preserved; better default parameters for setup_runner; added flags for setup_runner to fine-grain how it is used
…ctory; - isntead of _ for parallel id; depends_on now properly writes yaml in test parallelization; setup_runner doesn't override uri if its passed in; yml_parsing tests serialized;
…ity officially destroyed); update reference to green-coding-solutions in test_jobs
…uld have been; setup_runner now properly runs with no_build flag by default; cleaned up calls to setup_runner; build gcb_stress_gmt_run_tmp image before test runs; revert _tmp_image_name parallelization (unneeded);
…directories in /tmp/;
Old Energy EstimationEco-CI Output:
📈 Energy graph:
7.74 ┤ ╭╮
7.14 ┤ ╭╮ ││
6.55 ┤ ╭╯│ ╭╮ ││
5.95 ┤ │ │ ││ ││
5.35 ┤ │ │ ╭╯╰─╮ │╰──╮ ╭─╮ ╭╮ ╭╮ ╭╮
4.75 ┤ ╭╮ │ ╰╮ ╭─╮╭╮ │ │ │ │ │ ╰╮ ╭─╯│╭─╮╭─╮╭╮ ╭─╮ │╰╮╭─────╮ ╭╮ ╭──╮╭──╮││╭╮ ╭╮
4.16 ┤ ╭─╮ ╭─╮ ╭─╮ ╭╯╰╮ ╭╮ ╭╮ ╭╮ ╭╮╭╮ ╭─╮ ╭╮ ╭╮ │ ╰─╯ ╰╯│ ╭╯ ╰─╯ │ ╭╯ ╰─╮╭╯ ╰╯ ╰╯ ╰╯╰─╯ ╰─╯ ╰╯ ╰─────╯╰─╮ ╭╮ ╭╮ ╭─╮ ╭─╯ ╰╯ ╰╯╰╯╰─╯╰╮╭───╮ ╭╮ ╭╮ ╭╮ ╭╮╭╮ ╭╮ ╭
3.56 ┤ │ ╰────╯ ╰─╯ ╰───╯ ╰───╯╰────╯╰──╯╰──────────╯╰╯╰╮╭╯ ╰─────────────────────╯╰────────╯╰──╯ ╰╮│ │ ╭╮ ╭╯ ╰╯ ╰──────╯╰──╯╰─╯ │ ╭─╯ ╰╯ ╰──╯╰──╯╰──────╯╰──────╯╰╯╰─╯╰─╮ ╭╮ ╭╮ ╭╯
2.96 ┤ │ ╰╯ ││ │ ││ │ │ │ │ ╭╯│ ││ │
2.37 ┤ ╭╯ ╰╯ │ ││ │ │ │ │ │ ╰╮ ││ │
1.77 ┼────╯ ╰───────╯╰─╯ ╰─╯ ╰────────╯ ╰────────╯╰─╯
Watts over time |
Old Energy EstimationEco-CI Output:
📈 Energy graph:
7.69 ┤ ╭╮ ╭╮
7.10 ┤ ││ ││
6.51 ┤ ││ ╭╮ ││
5.91 ┤ │╰╮ ││ ││╭╮
5.32 ┤ ╭╮ │ │ ╭╯╰─╮ │╰╯│ ╭╮ ╭╮ ╭╮ ╭╮╭─╮
4.73 ┤ │╰╮ ╭╮ ╭╮ │ │ ╭─╮╭─╮ ╭╯ │ │ │ ╭╯╰─╮ ╭╯╰──╯│╭╮╭─╮╭─╮ ╭╮╭─╯││ ╰╮ ╭╮ ╭────╮╭──╮╭╮╭╮
4.14 ┤ ╭─╯ ╰╮ ╭╯╰───╮ ╭─╮ ╭─╮ ╭╮╭╮ ╭─╮ ╭╮ ││ ╭─╮ ╭╯ ╰─╯ ╰╯ │ │ ╰─╯ ╰╮ ╭╯ │╭╮╭─╯ ╰╯╰╯ ╰╯ ╰─╮╭╯╰╯ ╰╯ ╰────╯╰╮╭╮ ╭╮╭─╮╭╮╭╮ ╭╮ ╭╯ ╰╯ ╰╯╰╯╰╮╭─╮╭─╮ ╭╮ ╭╮ ╭─╮ ╭─╮ ╭─╮ ╭──╮╭╮ ╭╮ ╭╮ ╭
3.55 ┤ ╭─╯ ╰─╯ ╰─╯ ╰──╮│ ╰───╯╰╯│ │ ╰──────────────╯╰─────╯╰───────╯ ╰─╯ ╰╮ │ │ ╭╮│ ╰╯╰╯ ╰╯ ╰╯╰──╯╰╯ ╰╯╰╯╰──╯╰╮ ╭╯ ╰╯ ╰╯ ╰─╯╰╮╭╯╰╮│ ╰─╯ ╰─╯ ╰─╯ ╰╯╰╮╭╯╰─╯╰╮ ╭╮ ╭╮ ╭╯
2.95 ┤ ╭─╮│ ││ ╰╮╭╮│ │ │ │ │││ │ │ ╰╯ ╰╯ ││ ╰╮ ╭╯│ ││ │
2.36 ┤ │ ││ ╰╯ ││││ │ │ │ │╰╯ │ │ ╰╯ │ ╭╯ ╰╮ ││ │
1.77 ┼─────╯ ╰╯ ╰╯╰╯ ╰─╯ ╰───────╯ ╰─╯ ╰───────╯ ╰────────╯╰─╯
Watts over time |
|
@ArneTR - ready for review! |
|
|
Summary:
As for the reduction, it is what I expect. These are number and times of the kinds of tests: Parallel: 57 tests; time: 0:01:23 so of that 5.30, 2.24 minutes test time, 3,06 workflow overhead From the last main branch workflow, 72 tests; 0:03:41 test time and 3,32 workflow overhead. Which is almost 50% reduction in the test time itself. It is what I expect as I see a similar percentage reduction in local: Serialized: 0:07:16 total test time However.... I have to take back that this PR is ready. When I was running the tests locally to benchmark these times some more I'm seeing some sporadic failures still. It didn't show up when I tested earlier today during pushing, or in the PR tests, but they're there. As best as I can tell right now I think some cleanups are happening early, but I'm not sure yet. At least one more commit is incoming. |
|
Eco-CI Output:
📈 Energy graph:
7.78 ┤ ╭╮ ╭╮
7.18 ┤ ││ ││
6.58 ┤ ││ ╭╮ ││
5.98 ┤ ││ ││ ││╭╮
5.38 ┤ │╰╮ ╭╯╰─╮ │╰╯│ ╭─╮ ╭╮ ╭─╮
4.77 ┤ ╭╮ ╭╯ │ ╭─╮╭─╮ ╭╯ │╭╯ ╰╮ ╭╯ ╰╮╭╮╭─╯╰───╮╭╮╭─╮╭─╮╭╮╭─────╮│ ╰╮ ╭╮ ╭────────╮╭╮╭╮ ╭╮
4.17 ┤ ╭╯│ ╭─╮ ╭─╮ ╭─╮╭─╮ ╭╮ ╭─╮ ╭─╮ │ ╰─╯ ╰╯ │ │ ╰╯ │ ╭╯ ││││ ╰╯╰╯ ╰╯ ╰╯││ ╰╯ ╰────╯╰─╮ ╭╮ ╭╯ ╰╯╰╯╰─╯╰──╮ ╭╮ ╭╮ ╭╮ ╭╮ ╭╮ ╭╮ ╭
3.57 ┤ │ ╰─────╯ ╰─╯ ╰──╯ ╰╯ ╰────────────────────────────────╯╰─╮╭─╮│ ╰────────╮ ╭─────────────╯ ╰───────────╯ ╰╮│ │ ╭╮│ ╰╯╰╯ ╰╯ ╰──────────────╯╰╮ ╭╯ ╰─╯╰──────╯╰──────╯╰──╯╰───╮│╰─╯╰─╮ ╭─╮ ╭╮ ╭╯
2.97 ┤ │ ╰╯ ││ │ │ ││ │ │││ │ │ ╰╯ │ │ │ ││ │
2.37 ┤ ╭╯ ╰╯ │ │ ││ │ │╰╯ │ │ │ │ │╭╮ │╰╮│
1.77 ┼────╯ ╰───────────╯ ╰╯ ╰───────╯ ╰─╯ ╰────────╯ ╰╯╰───────╯ ╰╯
Watts over time |
|
Ok, my local failures had to do with my system, I had some test-containers still living in the background. I would still want to test more before merging, I'll run some more local sanity checks tomorrow. |
|
@ribalba Please give it a review and validate if the approach is feasible and satisfies the complexity for the expected gains. Then I would like to plan further steps on this |
|
I wil make this PR as a draft to have it as concept in the backlog. The idea and general implemementation of the parallelization is nice but the PR has some issues:
|
2 participants
Not ready yet, making PR to test in VM