run benchmark on multi-thread

[weixlu]

Motivation

Benchmarks take a long time to run. Since it's an important way to evaluate model and find optimization opportunities, it is worthwhile to speed up this process.

I profiled the process and found that most of the time is spent on CPU-side. GPU-side (i.e. segment and transformer inference) accounts for < 50%. Modern CPUs typically have 4+ cores, so obviously it can speed up using multiple threads (cores).

Implementation

The existing code already divides the dataset into batches (i.e. chunks in the code), with 10 samples per batch (i.e. batch_size=10). So we just use these batches in the multi-thread execution flow:

1. Place all N batches into the pending queue.
2. Start multiple worker threads. Each worker process a batch, and then places the result into the completed queue, and get a new batch.
3. Main thread find a newly completed batch (from completed queue), and prints the progress information and writes the result to the database.

Results

I measured the runtime on the Polish Scores dataset:

• 1 thread: 769.2 seconds
• 2 threads: 454.1 seconds — 1.7× speedup
• 3 threads: 341.1 seconds — 2.25× speedup
• 4 threads: 289.1 seconds — 2.66× speedup

We do get some speedup! But please note that optimization is hardware-dependent. On a machine with a strong CPU but a weak GPU, GPU inference is more likely to become the bottleneck, in which case multi-thread may provide poor improvement.

I would welcome benchmark results from other hardware configurations and look forward to receiving feedback! Simply add --workers 3 will make it work, like poetry run python -m validation.polish-scores --tool homr --workers 3

Other Changes

Again I also made several code refactor. The existing code in run_benchmark() makes extensive use of tuples, which makes the meaning of variables hard to understand. I replaced these with classes whose named fields clearly express their intent. Also added set_success and set_failure methods, making the control flow easier to follow.

[liebharc]

Thanks so much for introducing these! I'm actually a big fan of classes over tuples myself. I should really watch more carefully what the AI agents spits out.

[weixlu]

I (and my colleagues) are heavily using AI as well (like spend 100USD per month). Yes, AI can produce huge code in a short time, making it impossible to review every line. We havn't think about a good way to review it yet. My colleague propose to use a different AI model to review, honestly I don't think so, and this sounds ironic

[liebharc]

Reviewing code and ensuring that it is actually matching the requirements really seems to be the difficult part now. Orchestrating multiple agents (e.g. planning, execution, review, ...) seems to be a trend right now. And I guess everything which improves the results before you even look at it is useful. But at least as of now the cost also seems to be significant.

[liebharc]

Thanks, I haven't run it yet. Looks good, feel free to hit the merge button as soon as you are ready and I will likely give it a go soon then.