Low latency checks to submission checker and report low latency #2017
Conversation
pgmpablo157321
commented
Jan 6, 2025
pgmpablo157321
commented
- Add llama2 low latencies to configuration file
- Add low latency submission checks
- Report low latency result in a different column
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
pgmpablo157321
force-pushed
the
low_latency_submission_checker
branch
2 times, most recently
from
43f9059 to
7fa675c
Compare
|
@pgmpablo157321 : In the README, you show an example of running Offline in the Interactive mode. That can be confusing, because Offline is the same and only Server changes. I would suggest that the example would be for server and say that for the Offline run nothing changes with the new flags. |
| # settings.FromConfig(args.mlperf_conf, "llama2-70b", args.scenario) | ||
| settings.FromConfig(args.user_conf, "llama2-70b", args.scenario) | ||
| # settings.FromConfig(args.mlperf_conf, args.lg_model_name, args.scenario) | ||
| settings.FromConfig(args.user_conf, args.lg_model_name, args.scenario) |
There was a problem hiding this comment.
Should we still comment out the llama2-70b-interactive if we are loading from the user config here?
There was a problem hiding this comment.
Actually, do we still need the user.conf change if the mlperf.conf already has the new latency?
There was a problem hiding this comment.
Mmmm there might be other settings that submitters could modify. So, leaving the user.conf there might be helpful
pgmpablo157321
force-pushed
the
low_latency_submission_checker
branch
from
ea45000 to
9d322cb
Compare
|
@pgmpablo157321 the github action for submission checker is testing the submissions from v4.1 with the v5.0 submission checker code. That's why they are failing. Please ignore it. I'll add a test submission results repo soon with v5.0 results. |
|
We have tested the PR and we can merge it. We will raise issues if we find any |
