Commit 60a7088
authored
Fix cuvs_bench pytest pareto assert (#2027)
It is possible that the `throughput.csv` and `latency.csv` files can have fewer rows than `raw.csv`. That is because when we create the pareto csv(s) we drop certain rows if the latency and recall values are "dominated" by another point. For example if the latency and recall is lower than another point, we do not include that in the pareto. So we should relax the assert.
```
for rel_path, expectations in expected_files.items():
file_path = temp_datasets_dir / rel_path
assert file_path.exists(), f"Expected file {file_path} does not exist."
assert file_path.stat().st_size > 0, (
f"Expected file {file_path} is empty."
)
df = pd.read_csv(file_path)
actual_header = list(df.columns)
actual_rows = len(df)
# breakpoint()
assert actual_header == expectations["header"], (
f"Wrong header produced in file f{rel_path}"
)
> assert actual_rows == expectations["rows"]
E assert 1 == 2
tests/test_cli.py:442: AssertionError
```
This scenario is possible for certain hardware (gpu, cuda version) and configs. First datapoint is strictly worse than the second, so it is dropped in the pareto csv. This leads to the assertion error.
```
nprobe=1: Recall=0.1788, items_per_second=1.29669M/s, Latency=77.1218u
nprobe=5: Recall=0.3722, items_per_second=1.3073M/s, Latency=76.4963u
```
Authors:
- Anupam (https://github.com/aamijar)
Approvers:
- Dante Gama Dessavre (https://github.com/dantegd)
URL: #20271 parent 7a08cc7 commit 60a7088
1 file changed
Lines changed: 20 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | | - | |
| 2 | + | |
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| |||
439 | 439 | | |
440 | 440 | | |
441 | 441 | | |
442 | | - | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
443 | 461 | | |
444 | 462 | | |
445 | 463 | | |
| |||
0 commit comments