levante-bench/api_retry_metadata_rerun.log at main · langcog/levante-bench · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
Running evaluation: version=v1, device=cpu, output_base=results_api_retry_metadata (per-model: results_api_retry_metadata/v1/<model>-<size>[-<lang>]/)
  Data root: /home/david/levante/levante-bench/data
  Models: gpt53, gemini_pro
  egma-math: loaded human proportions for 327 items
  egma-math:   0%|          | 0/179 [00:00<?, ?batch/s]  egma-math:   1%|          | 1/179 [00:02<05:59,  2.02s/batch]  egma-math:   1%|          | 2/179 [00:04<06:29,  2.20s/batch]  egma-math:   2%|▏         | 3/179 [00:06<05:48,  1.98s/batch]  egma-math:   2%|▏         | 4/179 [00:08<05:48,  1.99s/batch]  egma-math:   3%|▎         | 5/179 [00:09<05:41,  1.96s/batch]  egma-math:   3%|▎         | 6/179 [00:11<05:16,  1.83s/batch]  egma-math:   4%|▍         | 7/179 [00:13<05:12,  1.82s/batch]  egma-math:   4%|▍         | 8/179 [00:14<04:42,  1.65s/batch]  egma-math:   5%|▌         | 9/179 [00:16<04:35,  1.62s/batch]  egma-math:   6%|▌         | 10/179 [00:17<04:41,  1.66s/batch]  egma-math:   6%|▌         | 11/179 [00:19<04:47,  1.71s/batch]  egma-math:   7%|▋         | 12/179 [00:22<05:22,  1.93s/batch]  egma-math:   7%|▋         | 13/179 [00:23<04:55,  1.78s/batch]  egma-math:   8%|▊         | 14/179 [00:25<04:41,  1.71s/batch]  egma-math:   8%|▊         | 15/179 [00:26<04:37,  1.69s/batch]  egma-math:   9%|▉         | 16/179 [00:28<04:34,  1.68s/batch]  egma-math:   9%|▉         | 17/179 [00:29<04:16,  1.58s/batch]  egma-math:  10%|█         | 18/179 [00:32<05:15,  1.96s/batch]  egma-math:  11%|█         | 19/179 [00:33<04:40,  1.75s/batch]  egma-math:  11%|█         | 20/179 [00:37<06:24,  2.42s/batch]  egma-math:  12%|█▏        | 21/179 [00:39<05:54,  2.25s/batch]  egma-math:  12%|█▏        | 22/179 [00:41<05:16,  2.02s/batch]  egma-math:  13%|█▎        | 23/179 [00:42<04:49,  1.86s/batch]  egma-math:  13%|█▎        | 24/179 [00:44<04:29,  1.74s/batch]  egma-math:  14%|█▍        | 25/179 [00:45<04:15,  1.66s/batch]  egma-math:  15%|█▍        | 26/179 [00:47<04:06,  1.61s/batch]  egma-math:  15%|█▌        | 27/179 [00:48<04:02,  1.60s/batch]  egma-math:  16%|█▌        | 28/179 [00:50<03:50,  1.52s/batch]  egma-math:  16%|█▌        | 29/179 [00:51<03:39,  1.46s/batch]  egma-math:  17%|█▋        | 30/179 [00:52<03:30,  1.41s/batch]  egma-math:  17%|█▋        | 31/179 [00:54<03:40,  1.49s/batch]  egma-math:  18%|█▊        | 32/179 [00:57<04:38,  1.89s/batch]  egma-math:  18%|█▊        | 33/179 [00:58<04:09,  1.71s/batch]  egma-math:  19%|█▉        | 34/179 [01:00<04:00,  1.66s/batch]  egma-math:  20%|█▉        | 35/179 [01:05<06:32,  2.72s/batch]  egma-math:  20%|██        | 36/179 [01:06<05:44,  2.41s/batch]  egma-math:  21%|██        | 37/179 [01:10<06:19,  2.67s/batch]  egma-math:  21%|██        | 38/179 [01:11<05:35,  2.38s/batch]  egma-math:  22%|██▏       | 39/179 [01:13<05:11,  2.22s/batch]  egma-math:  22%|██▏       | 40/179 [01:15<04:35,  1.98s/batch]  egma-math:  23%|██▎       | 41/179 [01:16<04:12,  1.83s/batch]  egma-math:  23%|██▎       | 42/179 [01:19<05:02,  2.21s/batch]  egma-math:  24%|██▍       | 43/179 [01:21<04:35,  2.03s/batch]  egma-math:  25%|██▍       | 44/179 [01:22<04:08,  1.84s/batch]  egma-math:  25%|██▌       | 45/179 [01:28<06:29,  2.90s/batch]  egma-math:  26%|██▌       | 46/179 [01:29<05:26,  2.46s/batch]  egma-math:  26%|██▋       | 47/179 [01:30<04:34,  2.08s/batch]  egma-math:  27%|██▋       | 48/179 [01:32<03:59,  1.83s/batch]  egma-math:  27%|██▋       | 49/179 [01:33<03:34,  1.65s/batch]  egma-math:  28%|██▊       | 50/179 [01:34<03:17,  1.53s/batch]  egma-math:  28%|██▊       | 51/179 [01:35<03:02,  1.42s/batch]  egma-math:  29%|██▉       | 52/179 [01:36<02:51,  1.35s/batch]  egma-math:  30%|██▉       | 53/179 [01:38<02:43,  1.30s/batch]  egma-math:  30%|███       | 54/179 [01:39<02:37,  1.26s/batch]  egma-math:  31%|███       | 55/179 [01:40<02:35,  1.26s/batch]  egma-math:  31%|███▏      | 56/179 [01:41<02:31,  1.23s/batch]  egma-math:  32%|███▏      | 57/179 [01:43<03:02,  1.50s/batch]  egma-math:  32%|███▏      | 58/179 [01:45<03:13,  1.60s/batch]  egma-math:  33%|███▎      | 59/179 [01:46<02:58,  1.49s/batch]  egma-math:  34%|███▎      | 60/179 [01:48<02:50,  1.43s/batch]  egma-math:  34%|███▍      | 61/179 [01:49<02:48,  1.43s/batch]  egma-math:  35%|███▍      | 62/179 [01:50<02:41,  1.38s/batch]  egma-math:  35%|███▌      | 63/179 [01:52<02:36,  1.35s/batch]  egma-math:  36%|███▌      | 64/179 [01:53<02:49,  1.47s/batch]  egma-math:  36%|███▋      | 65/179 [01:55<02:43,  1.44s/batch]  egma-math:  37%|███▋      | 66/179 [01:56<02:41,  1.43s/batch]  egma-math:  37%|███▋      | 67/179 [01:57<02:38,  1.42s/batch]  egma-math:  38%|███▊      | 68/179 [01:59<02:39,  1.44s/batch]  egma-math:  39%|███▊      | 69/179 [02:00<02:34,  1.40s/batch]  egma-math:  39%|███▉      | 70/179 [02:01<02:22,  1.31s/batch]  egma-math:  40%|███▉      | 71/179 [02:03<02:20,  1.30s/batch]  egma-math:  40%|████      | 72/179 [02:04<02:20,  1.31s/batch]  egma-math:  41%|████      | 73/179 [02:06<02:27,  1.39s/batch]  egma-math:  41%|████▏     | 74/179 [02:07<02:18,  1.32s/batch]  egma-math:  42%|████▏     | 75/179 [02:08<02:20,  1.35s/batch]  egma-math:  42%|████▏     | 76/179 [02:10<02:22,  1.38s/batch]  egma-math:  43%|████▎     | 77/179 [02:11<02:20,  1.38s/batch]  egma-math:  44%|████▎     | 78/179 [02:12<02:18,  1.37s/batch]  egma-math:  44%|████▍     | 79/179 [02:13<02:11,  1.31s/batch]  egma-math:  45%|████▍     | 80/179 [02:16<02:30,  1.52s/batch]  egma-math:  45%|████▌     | 81/179 [02:17<02:30,  1.53s/batch]  egma-math:  46%|████▌     | 82/179 [02:19<02:35,  1.61s/batch]  egma-math:  46%|████▋     | 83/179 [02:20<02:29,  1.55s/batch]  egma-math:  47%|████▋     | 84/179 [02:22<02:23,  1.51s/batch]  egma-math:  47%|████▋     | 85/179 [02:23<02:24,  1.54s/batch]  egma-math:  48%|████▊     | 86/179 [02:25<02:21,  1.52s/batch]  egma-math:  49%|████▊     | 87/179 [02:26<02:10,  1.42s/batch]  egma-math:  49%|████▉     | 88/179 [02:27<02:12,  1.45s/batch]  egma-math:  50%|████▉     | 89/179 [02:29<02:10,  1.45s/batch]  egma-math:  50%|█████     | 90/179 [02:30<02:09,  1.46s/batch]  egma-math:  51%|█████     | 91/179 [02:33<02:43,  1.86s/batch]  egma-math:  51%|█████▏    | 92/179 [02:35<02:35,  1.78s/batch]  egma-math:  52%|█████▏    | 93/179 [02:36<02:20,  1.63s/batch]  egma-math:  53%|█████▎    | 94/179 [02:37<02:12,  1.56s/batch]  egma-math:  53%|█████▎    | 95/179 [02:39<02:01,  1.45s/batch]  egma-math:  54%|█████▎    | 96/179 [02:40<01:56,  1.41s/batch]  egma-math:  54%|█████▍    | 97/179 [02:43<02:25,  1.77s/batch]  egma-math:  55%|█████▍    | 98/179 [02:44<02:17,  1.70s/batch]  egma-math:  55%|█████▌    | 99/179 [02:45<02:02,  1.53s/batch]  egma-math:  56%|█████▌    | 100/179 [02:48<02:32,  1.93s/batch]  egma-math:  56%|█████▋    | 101/179 [02:49<02:13,  1.71s/batch]  egma-math:  57%|█████▋    | 102/179 [02:51<02:01,  1.58s/batch]  egma-math:  58%|█████▊    | 103/179 [02:52<01:56,  1.53s/batch]  egma-math:  58%|█████▊    | 104/179 [02:53<01:49,  1.45s/batch]  egma-math:  59%|█████▊    | 105/179 [02:55<01:43,  1.40s/batch]  egma-math:  59%|█████▉    | 106/179 [02:56<01:37,  1.33s/batch]  egma-math:  60%|█████▉    | 107/179 [02:57<01:42,  1.42s/batch]  egma-math:  60%|██████    | 108/179 [02:59<01:40,  1.42s/batch]  egma-math:  61%|██████    | 109/179 [03:00<01:38,  1.40s/batch]  egma-math:  61%|██████▏   | 110/179 [03:02<01:35,  1.39s/batch]  egma-math:  62%|██████▏   | 111/179 [03:03<01:30,  1.33s/batch]  egma-math:  63%|██████▎   | 112/179 [03:05<01:45,  1.57s/batch]  egma-math:  63%|██████▎   | 113/179 [03:11<03:18,  3.00s/batch]  egma-math:  64%|██████▎   | 114/179 [03:13<02:45,  2.54s/batch]  egma-math:  64%|██████▍   | 115/179 [03:16<02:54,  2.73s/batch]  egma-math:  65%|██████▍   | 116/179 [03:19<02:55,  2.79s/batch]  egma-math:  65%|██████▌   | 117/179 [03:21<02:43,  2.64s/batch]  egma-math:  66%|██████▌   | 118/179 [03:22<02:16,  2.24s/batch]  egma-math:  66%|██████▋   | 119/179 [03:24<02:00,  2.01s/batch]  egma-math:  67%|██████▋   | 120/179 [03:25<01:45,  1.78s/batch]  egma-math:  68%|██████▊   | 121/179 [03:26<01:35,  1.65s/batch]  egma-math:  68%|██████▊   | 122/179 [03:28<01:31,  1.60s/batch]  egma-math:  69%|██████▊   | 123/179 [03:29<01:25,  1.53s/batch]  egma-math:  69%|██████▉   | 124/179 [03:31<01:23,  1.53s/batch]  egma-math:  70%|██████▉   | 125/179 [03:32<01:19,  1.47s/batch]  egma-math:  70%|███████   | 126/179 [03:33<01:14,  1.41s/batch]  egma-math:  71%|███████   | 127/179 [03:35<01:11,  1.38s/batch]  egma-math:  72%|███████▏  | 128/179 [03:36<01:11,  1.40s/batch]  egma-math:  72%|███████▏  | 129/179 [03:37<01:08,  1.37s/batch]  egma-math:  73%|███████▎  | 130/179 [03:39<01:10,  1.43s/batch]  egma-math:  73%|███████▎  | 131/179 [03:41<01:10,  1.46s/batch]  egma-math:  74%|███████▎  | 132/179 [03:42<01:11,  1.53s/batch]  egma-math:  74%|███████▍  | 133/179 [03:44<01:12,  1.57s/batch]  egma-math:  75%|███████▍  | 134/179 [03:47<01:31,  2.04s/batch]  egma-math:  75%|███████▌  | 135/179 [03:49<01:25,  1.95s/batch]  egma-math:  76%|███████▌  | 136/179 [03:51<01:22,  1.91s/batch]  egma-math:  77%|███████▋  | 137/179 [03:52<01:12,  1.73s/batch]  egma-math:  77%|███████▋  | 138/179 [03:55<01:24,  2.06s/batch]  egma-math:  78%|███████▊  | 139/179 [03:57<01:20,  2.01s/batch]  egma-math:  78%|███████▊  | 140/179 [03:59<01:18,  2.01s/batch]  egma-math:  79%|███████▉  | 141/179 [04:00<01:11,  1.87s/batch]  egma-math:  79%|███████▉  | 142/179 [04:03<01:16,  2.07s/batch]  egma-math:  80%|███████▉  | 143/179 [04:04<01:09,  1.94s/batch]  egma-math:  80%|████████  | 144/179 [04:06<01:04,  1.84s/batch]  egma-math:  81%|████████  | 145/179 [04:08<01:03,  1.88s/batch]  egma-math:  82%|████████▏ | 146/179 [04:10<01:03,  1.93s/batch]  egma-math:  82%|████████▏ | 147/179 [04:12<01:06,  2.07s/batch]  egma-math:  83%|████████▎ | 148/179 [04:14<01:04,  2.07s/batch]  egma-math:  83%|████████▎ | 149/179 [04:16<00:59,  1.98s/batch]  egma-math:  84%|████████▍ | 150/179 [04:21<01:19,  2.73s/batch]  egma-math:  84%|████████▍ | 151/179 [04:23<01:17,  2.75s/batch]  egma-math:  85%|████████▍ | 152/179 [04:28<01:28,  3.29s/batch]  egma-math:  85%|████████▌ | 153/179 [04:33<01:42,  3.93s/batch]  egma-math:  86%|████████▌ | 154/179 [04:38<01:41,  4.05s/batch]  egma-math:  87%|████████▋ | 155/179 [04:40<01:20,  3.35s/batch]  egma-math:  87%|████████▋ | 156/179 [04:41<01:05,  2.85s/batch]  egma-math:  88%|████████▊ | 157/179 [04:46<01:17,  3.50s/batch]  egma-math:  88%|████████▊ | 158/179 [04:49<01:08,  3.24s/batch]  egma-math:  89%|████████▉ | 159/179 [04:56<01:28,  4.40s/batch]  egma-math:  89%|████████▉ | 160/179 [04:57<01:07,  3.53s/batch]  egma-math:  90%|████████▉ | 161/179 [05:00<00:57,  3.21s/batch]  egma-math:  91%|█████████ | 162/179 [05:02<00:46,  2.75s/batch]  egma-math:  91%|█████████ | 163/179 [05:03<00:37,  2.37s/batch]  egma-math:  92%|█████████▏| 164/179 [05:05<00:31,  2.09s/batch]  egma-math:  92%|█████████▏| 165/179 [05:06<00:28,  2.03s/batch]  egma-math:  93%|█████████▎| 166/179 [05:08<00:26,  2.02s/batch]  egma-math:  93%|█████████▎| 167/179 [05:10<00:23,  1.99s/batch]  egma-math:  95%|█████████▍| 170/179 [05:12<00:10,  1.14s/batch]  egma-math:  96%|█████████▌| 171/179 [05:13<00:09,  1.23s/batch]  egma-math:  96%|█████████▌| 172/179 [05:15<00:09,  1.31s/batch]  egma-math: 100%|██████████| 179/179 [05:15<00:00,  1.76s/batch]
  gpt53/egma-math: wrote /home/david/levante/levante-bench/results_api_retry_metadata/v1/gpt53/egma-math-by-type.csv
  gpt53/egma-math: 0.9609 (172/179)
  matrix-reasoning: loaded human proportions for 157 items
  matrix-reasoning:   0%|          | 0/80 [00:00<?, ?batch/s]  matrix-reasoning:   1%|▏         | 1/80 [00:03<04:51,  3.69s/batch]  matrix-reasoning:   2%|▎         | 2/80 [00:07<05:14,  4.03s/batch]  matrix-reasoning:   4%|▍         | 3/80 [00:10<04:19,  3.37s/batch]  matrix-reasoning:   5%|▌         | 4/80 [00:15<05:11,  4.10s/batch]  matrix-reasoning:   6%|▋         | 5/80 [00:20<05:14,  4.19s/batch]  matrix-reasoning:   8%|▊         | 6/80 [00:28<07:01,  5.69s/batch]  matrix-reasoning:   9%|▉         | 7/80 [00:32<06:15,  5.14s/batch]  matrix-reasoning:  10%|█         | 8/80 [00:37<06:05,  5.08s/batch]  matrix-reasoning:  11%|█▏        | 9/80 [00:41<05:26,  4.59s/batch]  matrix-reasoning:  12%|█▎        | 10/80 [00:48<06:10,  5.30s/batch]  matrix-reasoning:  14%|█▍        | 11/80 [00:52<05:37,  4.89s/batch]  matrix-reasoning:  15%|█▌        | 12/80 [00:57<05:46,  5.10s/batch]  matrix-reasoning:  16%|█▋        | 13/80 [01:04<06:24,  5.73s/batch]  matrix-reasoning:  18%|█▊        | 14/80 [01:08<05:47,  5.26s/batch]  matrix-reasoning:  19%|█▉        | 15/80 [01:19<07:23,  6.82s/batch]  matrix-reasoning:  20%|██        | 16/80 [01:27<07:31,  7.06s/batch]  matrix-reasoning:  21%|██▏       | 17/80 [01:34<07:37,  7.26s/batch]  matrix-reasoning:  22%|██▎       | 18/80 [01:41<07:29,  7.25s/batch]  matrix-reasoning:  24%|██▍       | 19/80 [01:48<07:17,  7.17s/batch]  matrix-reasoning:  25%|██▌       | 20/80 [01:59<08:13,  8.22s/batch]  matrix-reasoning:  26%|██▋       | 21/80 [02:05<07:15,  7.38s/batch]  matrix-reasoning:  28%|██▊       | 22/80 [02:14<07:36,  7.88s/batch]  matrix-reasoning:  29%|██▉       | 23/80 [02:19<06:54,  7.28s/batch]  matrix-reasoning:  30%|███       | 24/80 [02:31<07:58,  8.55s/batch]  matrix-reasoning:  31%|███▏      | 25/80 [02:45<09:17, 10.13s/batch]  matrix-reasoning:  32%|███▎      | 26/80 [02:50<07:42,  8.56s/batch]  matrix-reasoning:  34%|███▍      | 27/80 [03:00<07:59,  9.05s/batch]  matrix-reasoning:  35%|███▌      | 28/80 [03:13<08:53, 10.27s/batch]  matrix-reasoning:  36%|███▋      | 29/80 [03:22<08:25,  9.91s/batch]  matrix-reasoning:  38%|███▊      | 30/80 [03:30<07:46,  9.32s/batch]  matrix-reasoning:  39%|███▉      | 31/80 [03:42<08:16, 10.13s/batch]  matrix-reasoning:  40%|████      | 32/80 [03:48<07:11,  8.98s/batch]  matrix-reasoning:  41%|████▏     | 33/80 [03:58<07:14,  9.25s/batch]  matrix-reasoning:  42%|████▎     | 34/80 [04:08<07:16,  9.49s/batch]  matrix-reasoning:  44%|████▍     | 35/80 [04:18<07:05,  9.44s/batch]  matrix-reasoning:  45%|████▌     | 36/80 [04:25<06:31,  8.90s/batch]  matrix-reasoning:  46%|████▋     | 37/80 [04:29<05:20,  7.45s/batch]  matrix-reasoning:  48%|████▊     | 38/80 [04:38<05:29,  7.84s/batch]  matrix-reasoning:  49%|████▉     | 39/80 [04:48<05:41,  8.33s/batch]  matrix-reasoning:  50%|█████     | 40/80 [04:52<04:51,  7.29s/batch]  matrix-reasoning:  51%|█████▏    | 41/80 [04:58<04:22,  6.74s/batch]  matrix-reasoning:  52%|█████▎    | 42/80 [05:05<04:21,  6.88s/batch]  matrix-reasoning:  54%|█████▍    | 43/80 [05:14<04:36,  7.47s/batch]  matrix-reasoning:  55%|█████▌    | 44/80 [05:19<04:02,  6.73s/batch]  matrix-reasoning:  56%|█████▋    | 45/80 [05:23<03:31,  6.04s/batch]  matrix-reasoning:  57%|█████▊    | 46/80 [05:28<03:10,  5.60s/batch]  matrix-reasoning:  59%|█████▉    | 47/80 [05:34<03:08,  5.73s/batch]  matrix-reasoning:  60%|██████    | 48/80 [05:39<02:56,  5.51s/batch]  matrix-reasoning:  61%|██████▏   | 49/80 [05:50<03:41,  7.15s/batch]  matrix-reasoning:  62%|██████▎   | 50/80 [05:55<03:13,  6.43s/batch]  matrix-reasoning:  64%|██████▍   | 51/80 [05:58<02:40,  5.52s/batch]  matrix-reasoning:  65%|██████▌   | 52/80 [06:11<03:36,  7.73s/batch]  matrix-reasoning:  66%|██████▋   | 53/80 [06:15<03:00,  6.70s/batch]  matrix-reasoning:  68%|██████▊   | 54/80 [06:24<03:09,  7.28s/batch]  matrix-reasoning:  69%|██████▉   | 55/80 [06:31<03:01,  7.27s/batch]  matrix-reasoning:  70%|███████   | 56/80 [06:43<03:30,  8.76s/batch]  matrix-reasoning:  71%|███████▏  | 57/80 [06:56<03:46,  9.83s/batch]  matrix-reasoning:  72%|███████▎  | 58/80 [07:04<03:27,  9.42s/batch]  matrix-reasoning:  74%|███████▍  | 59/80 [07:07<02:39,  7.58s/batch]  matrix-reasoning:  75%|███████▌  | 60/80 [07:10<02:03,  6.20s/batch]  matrix-reasoning:  76%|███████▋  | 61/80 [07:20<02:18,  7.26s/batch]  matrix-reasoning:  78%|███████▊  | 62/80 [07:32<02:33,  8.54s/batch]  matrix-reasoning:  79%|███████▉  | 63/80 [07:35<01:56,  6.88s/batch]  matrix-reasoning:  80%|████████  | 64/80 [07:45<02:08,  8.01s/batch]  matrix-reasoning:  81%|████████▏ | 65/80 [07:49<01:42,  6.83s/batch]  matrix-reasoning:  82%|████████▎ | 66/80 [07:53<01:22,  5.86s/batch]  matrix-reasoning:  84%|████████▍ | 67/80 [08:05<01:38,  7.60s/batch]  matrix-reasoning:  85%|████████▌ | 68/80 [08:15<01:42,  8.56s/batch]  matrix-reasoning:  86%|████████▋ | 69/80 [08:22<01:28,  8.07s/batch]  matrix-reasoning:  88%|████████▊ | 70/80 [08:29<01:15,  7.50s/batch]  matrix-reasoning:  89%|████████▉ | 71/80 [08:32<00:56,  6.29s/batch]  matrix-reasoning:  90%|█████████ | 72/80 [08:45<01:05,  8.19s/batch]  matrix-reasoning:  91%|█████████▏| 73/80 [08:53<00:56,  8.12s/batch]  matrix-reasoning:  92%|█████████▎| 74/80 [09:03<00:53,  8.90s/batch]  matrix-reasoning:  94%|█████████▍| 75/80 [09:07<00:36,  7.36s/batch]  matrix-reasoning:  95%|█████████▌| 76/80 [09:21<00:37,  9.31s/batch]  matrix-reasoning:  96%|█████████▋| 77/80 [09:31<00:28,  9.44s/batch]  matrix-reasoning:  98%|█████████▊| 78/80 [09:38<00:17,  8.83s/batch]  matrix-reasoning:  99%|█████████▉| 79/80 [09:53<00:10, 10.64s/batch]  matrix-reasoning: 100%|██████████| 80/80 [10:02<00:00, 10.25s/batch]  matrix-reasoning: 100%|██████████| 80/80 [10:02<00:00,  7.53s/batch]
  gpt53/matrix-reasoning: 0.5500 (44/80)
  mental-rotation: loaded human proportions for 12 items
  mental-rotation:   0%|          | 0/83 [00:00<?, ?batch/s]  mental-rotation:   1%|          | 1/83 [00:02<02:59,  2.18s/batch]  mental-rotation:   2%|▏         | 2/83 [00:04<02:45,  2.04s/batch]  mental-rotation:   4%|▎         | 3/83 [00:06<03:12,  2.41s/batch]  mental-rotation:   5%|▍         | 4/83 [00:10<03:40,  2.79s/batch]  mental-rotation:   6%|▌         | 5/83 [00:12<03:29,  2.69s/batch]  mental-rotation:   7%|▋         | 6/83 [00:16<03:40,  2.86s/batch]  mental-rotation:   8%|▊         | 7/83 [00:18<03:36,  2.85s/batch]  mental-rotation:  10%|▉         | 8/83 [00:23<04:08,  3.31s/batch]  mental-rotation:  11%|█         | 9/83 [00:29<05:08,  4.17s/batch]  mental-rotation:  12%|█▏        | 10/83 [00:32<04:51,  3.99s/batch]  mental-rotation:  13%|█▎        | 11/83 [00:37<05:13,  4.35s/batch]  mental-rotation:  14%|█▍        | 12/83 [00:41<04:48,  4.06s/batch]  mental-rotation:  16%|█▌        | 13/83 [00:44<04:20,  3.72s/batch]  mental-rotation:  22%|██▏       | 18/83 [00:47<01:40,  1.55s/batch]  mental-rotation:  24%|██▍       | 20/83 [00:51<01:51,  1.77s/batch]  mental-rotation:  27%|██▋       | 22/83 [00:54<01:37,  1.60s/batch]  mental-rotation:  28%|██▊       | 23/83 [00:57<01:51,  1.86s/batch]  mental-rotation:  29%|██▉       | 24/83 [00:59<01:53,  1.92s/batch]  mental-rotation:  30%|███       | 25/83 [01:03<02:24,  2.49s/batch]  mental-rotation:  31%|███▏      | 26/83 [01:07<02:41,  2.84s/batch]  mental-rotation:  33%|███▎      | 27/83 [01:11<02:50,  3.04s/batch]  mental-rotation:  34%|███▎      | 28/83 [01:16<03:10,  3.47s/batch]  mental-rotation:  35%|███▍      | 29/83 [01:18<02:55,  3.24s/batch]  mental-rotation:  36%|███▌      | 30/83 [01:26<04:01,  4.56s/batch]  mental-rotation:  37%|███▋      | 31/83 [01:33<04:32,  5.24s/batch]  mental-rotation:  39%|███▊      | 32/83 [01:36<03:52,  4.56s/batch]  mental-rotation:  40%|███▉      | 33/83 [01:40<03:45,  4.52s/batch]  mental-rotation:  41%|████      | 34/83 [01:45<03:47,  4.65s/batch]  mental-rotation:  43%|████▎     | 36/83 [01:49<02:34,  3.28s/batch]  mental-rotation:  45%|████▍     | 37/83 [01:52<02:31,  3.29s/batch]  mental-rotation:  53%|█████▎    | 44/83 [01:56<00:53,  1.37s/batch]  mental-rotation:  54%|█████▍    | 45/83 [01:59<00:59,  1.57s/batch]  mental-rotation:  55%|█████▌    | 46/83 [02:05<01:22,  2.23s/batch]  mental-rotation:  57%|█████▋    | 47/83 [02:09<01:29,  2.48s/batch]  mental-rotation:  58%|█████▊    | 48/83 [02:13<01:41,  2.90s/batch]  mental-rotation:  59%|█████▉    | 49/83 [02:16<01:41,  2.97s/batch]  mental-rotation:  60%|██████    | 50/83 [02:22<02:02,  3.72s/batch]  mental-rotation:  61%|██████▏   | 51/83 [02:29<02:24,  4.52s/batch]  mental-rotation:  63%|██████▎   | 52/83 [02:36<02:37,  5.07s/batch]  mental-rotation:  64%|██████▍   | 53/83 [02:45<03:08,  6.27s/batch]  mental-rotation:  65%|██████▌   | 54/83 [02:55<03:31,  7.31s/batch]  mental-rotation:  66%|██████▋   | 55/83 [03:02<03:21,  7.18s/batch]  mental-rotation:  67%|██████▋   | 56/83 [03:10<03:24,  7.56s/batch]  mental-rotation:  69%|██████▊   | 57/83 [03:16<02:59,  6.92s/batch]  mental-rotation:  70%|██████▉   | 58/83 [03:27<03:25,  8.22s/batch]  mental-rotation:  71%|███████   | 59/83 [03:31<02:45,  6.89s/batch]  mental-rotation:  72%|███████▏  | 60/83 [03:40<02:56,  7.65s/batch]  mental-rotation:  73%|███████▎  | 61/83 [03:47<02:39,  7.24s/batch]  mental-rotation:  75%|███████▍  | 62/83 [03:56<02:44,  7.84s/batch]  mental-rotation:  76%|███████▌  | 63/83 [04:07<02:58,  8.94s/batch]  mental-rotation:  77%|███████▋  | 64/83 [04:13<02:32,  8.05s/batch]  mental-rotation:  78%|███████▊  | 65/83 [04:19<02:10,  7.25s/batch]  mental-rotation:  80%|███████▉  | 66/83 [04:27<02:09,  7.64s/batch]  mental-rotation:  81%|████████  | 67/83 [04:32<01:50,  6.91s/batch]  mental-rotation:  82%|████████▏ | 68/83 [04:39<01:40,  6.71s/batch]  mental-rotation:  83%|████████▎ | 69/83 [04:47<01:38,  7.03s/batch]  mental-rotation:  84%|████████▍ | 70/83 [04:52<01:25,  6.57s/batch]  mental-rotation:  86%|████████▌ | 71/83 [04:59<01:20,  6.71s/batch]  mental-rotation:  87%|████████▋ | 72/83 [05:05<01:11,  6.46s/batch]  mental-rotation:  88%|████████▊ | 73/83 [05:10<01:01,  6.12s/batch]  mental-rotation:  89%|████████▉ | 74/83 [05:19<01:02,  6.98s/batch]  mental-rotation:  90%|█████████ | 75/83 [05:29<01:01,  7.69s/batch]  mental-rotation:  92%|█████████▏| 76/83 [05:35<00:50,  7.26s/batch]  mental-rotation:  93%|█████████▎| 77/83 [05:41<00:41,  6.93s/batch]  mental-rotation:  94%|█████████▍| 78/83 [05:52<00:40,  8.04s/batch]  mental-rotation:  95%|█████████▌| 79/83 [05:59<00:31,  7.82s/batch]  mental-rotation:  96%|█████████▋| 80/83 [06:03<00:19,  6.62s/batch]  mental-rotation:  98%|█████████▊| 81/83 [06:14<00:15,  7.92s/batch]  mental-rotation:  99%|█████████▉| 82/83 [06:22<00:07,  7.92s/batch]  mental-rotation: 100%|██████████| 83/83 [06:29<00:00,  7.70s/batch]  mental-rotation: 100%|██████████| 83/83 [06:29<00:00,  4.69s/batch]
  gpt53/mental-rotation: 0.6506 (54/83)
  synthetic-vocab:   0%|          | 0/100 [00:00<?, ?batch/s]  synthetic-vocab:   1%|          | 1/100 [00:08<13:13,  8.01s/batch]  synthetic-vocab:   2%|▏         | 2/100 [00:14<11:46,  7.21s/batch]  synthetic-vocab:   3%|▎         | 3/100 [00:22<12:18,  7.61s/batch]  synthetic-vocab:   4%|▍         | 4/100 [00:31<13:04,  8.17s/batch]  synthetic-vocab:   5%|▌         | 5/100 [00:38<11:49,  7.47s/batch]  synthetic-vocab:   6%|▌         | 6/100 [00:45<11:49,  7.55s/batch]  synthetic-vocab:   7%|▋         | 7/100 [00:52<11:33,  7.46s/batch]  synthetic-vocab:   8%|▊         | 8/100 [00:59<10:44,  7.01s/batch]  synthetic-vocab:   9%|▉         | 9/100 [01:06<10:44,  7.09s/batch]  synthetic-vocab:  10%|█         | 10/100 [01:13<10:40,  7.12s/batch]  synthetic-vocab:  11%|█         | 11/100 [01:22<11:12,  7.56s/batch]  synthetic-vocab:  12%|█▏        | 12/100 [01:28<10:23,  7.09s/batch]  synthetic-vocab:  13%|█▎        | 13/100 [01:33<09:45,  6.73s/batch]  synthetic-vocab:  14%|█▍        | 14/100 [01:39<09:16,  6.48s/batch]  synthetic-vocab:  15%|█▌        | 15/100 [01:45<08:56,  6.31s/batch]  synthetic-vocab:  16%|█▌        | 16/100 [01:53<09:36,  6.87s/batch]  synthetic-vocab:  17%|█▋        | 17/100 [02:00<09:31,  6.88s/batch]  synthetic-vocab:  18%|█▊        | 18/100 [02:06<09:03,  6.62s/batch]  synthetic-vocab:  19%|█▉        | 19/100 [02:14<09:18,  6.90s/batch]  synthetic-vocab:  20%|██        | 20/100 [02:20<08:53,  6.67s/batch]  synthetic-vocab:  21%|██        | 21/100 [02:27<09:01,  6.85s/batch]  synthetic-vocab:  22%|██▏       | 22/100 [02:38<10:13,  7.87s/batch]  synthetic-vocab:  23%|██▎       | 23/100 [02:43<08:59,  7.01s/batch]  synthetic-vocab:  24%|██▍       | 24/100 [02:50<09:04,  7.17s/batch]  synthetic-vocab:  25%|██▌       | 25/100 [03:00<09:55,  7.94s/batch]  synthetic-vocab:  26%|██▌       | 26/100 [03:06<09:13,  7.48s/batch]  synthetic-vocab:  27%|██▋       | 27/100 [03:15<09:25,  7.75s/batch]  synthetic-vocab:  28%|██▊       | 28/100 [03:22<09:05,  7.58s/batch]  synthetic-vocab:  29%|██▉       | 29/100 [03:30<09:12,  7.78s/batch]  synthetic-vocab:  30%|███       | 30/100 [03:40<09:43,  8.33s/batch]  synthetic-vocab:  31%|███       | 31/100 [03:46<08:55,  7.75s/batch]  synthetic-vocab:  32%|███▏      | 32/100 [03:55<09:13,  8.13s/batch]  synthetic-vocab:  33%|███▎      | 33/100 [04:03<09:09,  8.19s/batch]  synthetic-vocab:  34%|███▍      | 34/100 [04:10<08:25,  7.65s/batch]  synthetic-vocab:  35%|███▌      | 35/100 [04:19<08:56,  8.26s/batch]  synthetic-vocab:  36%|███▌      | 36/100 [04:27<08:31,  8.00s/batch]  synthetic-vocab:  37%|███▋      | 37/100 [04:33<07:44,  7.38s/batch]  synthetic-vocab:  38%|███▊      | 38/100 [04:40<07:32,  7.30s/batch]  synthetic-vocab:  39%|███▉      | 39/100 [04:48<07:39,  7.54s/batch]  synthetic-vocab:  40%|████      | 40/100 [04:54<07:03,  7.06s/batch]  synthetic-vocab:  41%|████      | 41/100 [05:00<06:39,  6.77s/batch]  synthetic-vocab:  42%|████▏     | 42/100 [05:06<06:09,  6.38s/batch]  synthetic-vocab:  43%|████▎     | 43/100 [05:12<06:09,  6.48s/batch]  synthetic-vocab:  44%|████▍     | 44/100 [05:20<06:27,  6.91s/batch]  synthetic-vocab:  45%|████▌     | 45/100 [05:27<06:12,  6.77s/batch]  synthetic-vocab:  46%|████▌     | 46/100 [05:35<06:27,  7.17s/batch]  synthetic-vocab:  47%|████▋     | 47/100 [05:41<06:08,  6.95s/batch]  synthetic-vocab:  48%|████▊     | 48/100 [05:50<06:23,  7.38s/batch]  synthetic-vocab:  49%|████▉     | 49/100 [05:59<06:44,  7.93s/batch]  synthetic-vocab:  50%|█████     | 50/100 [06:05<06:14,  7.48s/batch]  synthetic-vocab:  51%|█████     | 51/100 [06:11<05:47,  7.09s/batch]  synthetic-vocab:  52%|█████▏    | 52/100 [06:20<06:09,  7.70s/batch]  synthetic-vocab:  53%|█████▎    | 53/100 [06:26<05:37,  7.18s/batch]  synthetic-vocab:  54%|█████▍    | 54/100 [06:32<05:14,  6.83s/batch]  synthetic-vocab:  55%|█████▌    | 55/100 [06:38<04:56,  6.59s/batch]  synthetic-vocab:  56%|█████▌    | 56/100 [06:45<04:50,  6.60s/batch]  synthetic-vocab:  57%|█████▋    | 57/100 [06:52<04:49,  6.72s/batch]  synthetic-vocab:  58%|█████▊    | 58/100 [06:58<04:34,  6.53s/batch]  synthetic-vocab:  59%|█████▉    | 59/100 [07:03<04:11,  6.14s/batch]  synthetic-vocab:  60%|██████    | 60/100 [07:10<04:12,  6.31s/batch]  synthetic-vocab:  61%|██████    | 61/100 [07:17<04:06,  6.32s/batch]  synthetic-vocab:  62%|██████▏   | 62/100 [07:23<04:03,  6.42s/batch]  synthetic-vocab:  63%|██████▎   | 63/100 [07:29<03:50,  6.23s/batch]  synthetic-vocab:  64%|██████▍   | 64/100 [07:36<03:54,  6.52s/batch]  synthetic-vocab:  65%|██████▌   | 65/100 [07:45<04:11,  7.19s/batch]  synthetic-vocab:  66%|██████▌   | 66/100 [07:54<04:21,  7.69s/batch]  synthetic-vocab:  67%|██████▋   | 67/100 [08:02<04:23,  7.99s/batch]  synthetic-vocab:  68%|██████▊   | 68/100 [08:10<04:07,  7.74s/batch]  synthetic-vocab:  69%|██████▉   | 69/100 [08:16<03:47,  7.34s/batch]  synthetic-vocab:  70%|███████   | 70/100 [08:22<03:29,  7.00s/batch]  synthetic-vocab:  71%|███████   | 71/100 [08:30<03:28,  7.17s/batch]  synthetic-vocab:  72%|███████▏  | 72/100 [08:35<03:07,  6.68s/batch]  synthetic-vocab:  73%|███████▎  | 73/100 [08:45<03:21,  7.46s/batch]  synthetic-vocab:  74%|███████▍  | 74/100 [08:51<03:03,  7.05s/batch]  synthetic-vocab:  75%|███████▌  | 75/100 [08:59<03:06,  7.44s/batch]  synthetic-vocab:  76%|███████▌  | 76/100 [09:08<03:12,  8.00s/batch]  synthetic-vocab:  77%|███████▋  | 77/100 [09:14<02:50,  7.40s/batch]  synthetic-vocab:  78%|███████▊  | 78/100 [09:22<02:44,  7.47s/batch]  synthetic-vocab:  79%|███████▉  | 79/100 [09:29<02:32,  7.24s/batch]  synthetic-vocab:  80%|████████  | 80/100 [09:38<02:35,  7.78s/batch]  synthetic-vocab:  81%|████████  | 81/100 [09:46<02:27,  7.78s/batch]  synthetic-vocab:  82%|████████▏ | 82/100 [09:52<02:12,  7.38s/batch]  synthetic-vocab:  83%|████████▎ | 83/100 [09:58<01:58,  6.97s/batch]  synthetic-vocab:  84%|████████▍ | 84/100 [10:08<02:05,  7.83s/batch]  synthetic-vocab:  85%|████████▌ | 85/100 [10:17<02:02,  8.18s/batch]  synthetic-vocab:  86%|████████▌ | 86/100 [10:24<01:50,  7.90s/batch]  synthetic-vocab:  87%|████████▋ | 87/100 [10:34<01:49,  8.43s/batch]  synthetic-vocab:  88%|████████▊ | 88/100 [10:41<01:36,  8.03s/batch]  synthetic-vocab:  89%|████████▉ | 89/100 [10:50<01:30,  8.27s/batch]  synthetic-vocab:  90%|█████████ | 90/100 [10:57<01:19,  7.95s/batch]  synthetic-vocab:  91%|█████████ | 91/100 [11:04<01:08,  7.65s/batch]  synthetic-vocab:  92%|█████████▏| 92/100 [11:11<01:01,  7.63s/batch]  synthetic-vocab:  93%|█████████▎| 93/100 [11:18<00:51,  7.30s/batch]  synthetic-vocab:  94%|█████████▍| 94/100 [11:24<00:41,  6.97s/batch]  synthetic-vocab:  95%|█████████▌| 95/100 [11:30<00:33,  6.67s/batch]  synthetic-vocab:  96%|█████████▌| 96/100 [11:37<00:27,  6.78s/batch]  synthetic-vocab:  97%|█████████▋| 97/100 [11:44<00:20,  6.84s/batch]  synthetic-vocab:  98%|█████████▊| 98/100 [11:51<00:13,  6.98s/batch]  synthetic-vocab:  99%|█████████▉| 99/100 [11:59<00:07,  7.07s/batch]  synthetic-vocab: 100%|██████████| 100/100 [12:05<00:00,  6.94s/batch]  synthetic-vocab: 100%|██████████| 100/100 [12:05<00:00,  7.26s/batch]
  gpt53/synthetic-vocab: 0.9900 (99/100)
  theory-of-mind: loaded human proportions for 95 items
  theory-of-mind:   0%|          | 0/37 [00:00<?, ?batch/s]  theory-of-mind:   3%|▎         | 1/37 [00:02<01:15,  2.09s/batch]  theory-of-mind:   5%|▌         | 2/37 [00:04<01:14,  2.12s/batch]  theory-of-mind:   8%|▊         | 3/37 [00:06<01:16,  2.25s/batch]  theory-of-mind:  11%|█         | 4/37 [00:08<01:09,  2.12s/batch]  theory-of-mind:  14%|█▎        | 5/37 [00:11<01:20,  2.52s/batch]  theory-of-mind:  16%|█▌        | 6/37 [00:14<01:17,  2.52s/batch]  theory-of-mind:  19%|█▉        | 7/37 [00:16<01:11,  2.40s/batch]  theory-of-mind:  22%|██▏       | 8/37 [00:18<01:10,  2.44s/batch]  theory-of-mind:  24%|██▍       | 9/37 [00:23<01:26,  3.08s/batch]  theory-of-mind:  27%|██▋       | 10/37 [00:25<01:10,  2.62s/batch]  theory-of-mind:  30%|██▉       | 11/37 [00:26<01:02,  2.39s/batch]  theory-of-mind:  32%|███▏      | 12/37 [00:29<00:58,  2.35s/batch]  theory-of-mind:  35%|███▌      | 13/37 [00:31<00:57,  2.40s/batch]  theory-of-mind:  38%|███▊      | 14/37 [00:33<00:53,  2.33s/batch]  theory-of-mind:  41%|████      | 15/37 [00:35<00:47,  2.17s/batch]  theory-of-mind:  43%|████▎     | 16/37 [00:37<00:42,  2.02s/batch]  theory-of-mind:  46%|████▌     | 17/37 [00:39<00:39,  1.95s/batch]  theory-of-mind:  49%|████▊     | 18/37 [00:40<00:35,  1.87s/batch]  theory-of-mind:  51%|█████▏    | 19/37 [00:43<00:38,  2.16s/batch]  theory-of-mind:  54%|█████▍    | 20/37 [00:46<00:42,  2.49s/batch]  theory-of-mind:  57%|█████▋    | 21/37 [00:49<00:40,  2.53s/batch]  theory-of-mind:  59%|█████▉    | 22/37 [00:52<00:41,  2.75s/batch]  theory-of-mind:  62%|██████▏   | 23/37 [00:56<00:41,  2.99s/batch]  theory-of-mind:  65%|██████▍   | 24/37 [00:58<00:37,  2.85s/batch]  theory-of-mind:  68%|██████▊   | 25/37 [01:00<00:31,  2.62s/batch]  theory-of-mind:  70%|███████   | 26/37 [01:02<00:26,  2.39s/batch]  theory-of-mind:  73%|███████▎  | 27/37 [01:05<00:23,  2.39s/batch]  theory-of-mind:  76%|███████▌  | 28/37 [01:07<00:20,  2.31s/batch]  theory-of-mind:  78%|███████▊  | 29/37 [01:09<00:18,  2.29s/batch]  theory-of-mind:  81%|████████  | 30/37 [01:11<00:15,  2.15s/batch]  theory-of-mind:  84%|████████▍ | 31/37 [01:13<00:12,  2.09s/batch]  theory-of-mind:  86%|████████▋ | 32/37 [01:15<00:10,  2.18s/batch]  theory-of-mind:  89%|████████▉ | 33/37 [01:17<00:08,  2.18s/batch]  theory-of-mind:  92%|█████████▏| 34/37 [01:20<00:07,  2.43s/batch]  theory-of-mind:  95%|█████████▍| 35/37 [01:23<00:04,  2.33s/batch]  theory-of-mind:  97%|█████████▋| 36/37 [01:25<00:02,  2.26s/batch]  theory-of-mind: 100%|██████████| 37/37 [01:27<00:00,  2.23s/batch]  theory-of-mind: 100%|██████████| 37/37 [01:27<00:00,  2.36s/batch]
  gpt53/theory-of-mind: wrote /home/david/levante/levante-bench/results_api_retry_metadata/v1/gpt53/theory-of-mind-by-type.csv
  gpt53/theory-of-mind: 0.6757 (25/37)
  trog: loaded human proportions for 99 items
  trog:   0%|          | 0/99 [00:00<?, ?batch/s]  trog:   1%|          | 1/99 [00:01<02:50,  1.74s/batch]  trog:   2%|▏         | 2/99 [00:03<02:57,  1.83s/batch]  trog:   3%|▎         | 3/99 [00:05<03:12,  2.01s/batch]  trog:   4%|▍         | 4/99 [00:07<03:11,  2.01s/batch]  trog:   5%|▌         | 5/99 [00:09<02:59,  1.91s/batch]  trog:   6%|▌         | 6/99 [00:11<02:58,  1.92s/batch]  trog:   7%|▋         | 7/99 [00:13<02:54,  1.90s/batch]  trog:   8%|▊         | 8/99 [00:15<02:58,  1.96s/batch]  trog:   9%|▉         | 9/99 [00:17<03:02,  2.02s/batch]  trog:  10%|█         | 10/99 [00:20<03:12,  2.16s/batch]  trog:  11%|█         | 11/99 [00:21<02:57,  2.02s/batch]  trog:  12%|█▏        | 12/99 [00:26<04:01,  2.78s/batch]  trog:  13%|█▎        | 13/99 [00:29<04:22,  3.05s/batch]  trog:  14%|█▍        | 14/99 [00:31<03:52,  2.73s/batch]  trog:  15%|█▌        | 15/99 [00:34<03:36,  2.58s/batch]  trog:  16%|█▌        | 16/99 [00:37<03:39,  2.64s/batch]  trog:  17%|█▋        | 17/99 [00:39<03:22,  2.47s/batch]  trog:  18%|█▊        | 18/99 [00:42<03:36,  2.68s/batch]  trog:  19%|█▉        | 19/99 [00:44<03:15,  2.44s/batch]  trog:  20%|██        | 20/99 [00:46<03:12,  2.44s/batch]  trog:  21%|██        | 21/99 [00:48<03:04,  2.36s/batch]  trog:  22%|██▏       | 22/99 [00:50<02:56,  2.29s/batch]  trog:  23%|██▎       | 23/99 [00:53<03:10,  2.50s/batch]  trog:  24%|██▍       | 24/99 [00:55<02:54,  2.33s/batch]  trog:  25%|██▌       | 25/99 [00:58<03:05,  2.51s/batch]  trog:  26%|██▋       | 26/99 [01:01<03:02,  2.50s/batch]  trog:  27%|██▋       | 27/99 [01:03<03:04,  2.56s/batch]  trog:  28%|██▊       | 28/99 [01:06<02:54,  2.46s/batch]  trog:  29%|██▉       | 29/99 [01:08<02:51,  2.46s/batch]  trog:  30%|███       | 30/99 [01:10<02:46,  2.41s/batch]  trog:  31%|███▏      | 31/99 [01:15<03:19,  2.94s/batch]  trog:  32%|███▏      | 32/99 [01:17<03:00,  2.69s/batch]  trog:  33%|███▎      | 33/99 [01:19<02:42,  2.46s/batch]  trog:  34%|███▍      | 34/99 [01:21<02:31,  2.33s/batch]  trog:  35%|███▌      | 35/99 [01:23<02:23,  2.24s/batch]  trog:  36%|███▋      | 36/99 [01:25<02:18,  2.20s/batch]  trog:  37%|███▋      | 37/99 [01:27<02:24,  2.34s/batch]  trog:  38%|███▊      | 38/99 [01:30<02:30,  2.47s/batch]  trog:  39%|███▉      | 39/99 [01:33<02:25,  2.43s/batch]  trog:  40%|████      | 40/99 [01:35<02:31,  2.56s/batch]  trog:  41%|████▏     | 41/99 [01:38<02:23,  2.47s/batch]  trog:  42%|████▏     | 42/99 [01:40<02:21,  2.48s/batch]  trog:  43%|████▎     | 43/99 [01:43<02:19,  2.48s/batch]  trog:  44%|████▍     | 44/99 [01:45<02:15,  2.46s/batch]  trog:  45%|████▌     | 45/99 [01:47<02:03,  2.28s/batch]  trog:  46%|████▋     | 46/99 [01:49<01:57,  2.21s/batch]  trog:  47%|████▋     | 47/99 [01:51<01:53,  2.18s/batch]  trog:  48%|████▊     | 48/99 [01:53<01:48,  2.13s/batch]  trog:  49%|████▉     | 49/99 [01:57<02:06,  2.53s/batch]  trog:  51%|█████     | 50/99 [01:59<02:05,  2.57s/batch]  trog:  52%|█████▏    | 51/99 [02:02<02:10,  2.72s/batch]  trog:  53%|█████▎    | 52/99 [02:06<02:21,  3.00s/batch]  trog:  54%|█████▎    | 53/99 [02:09<02:19,  3.03s/batch]  trog:  55%|█████▍    | 54/99 [02:12<02:15,  3.00s/batch]  trog:  56%|█████▌    | 55/99 [02:14<02:03,  2.81s/batch]  trog:  57%|█████▋    | 56/99 [02:16<01:51,  2.59s/batch]  trog:  58%|█████▊    | 57/99 [02:19<01:46,  2.53s/batch]  trog:  59%|█████▊    | 58/99 [02:21<01:43,  2.53s/batch]  trog:  60%|█████▉    | 59/99 [02:23<01:34,  2.37s/batch]  trog:  61%|██████    | 60/99 [02:26<01:36,  2.48s/batch]  trog:  62%|██████▏   | 61/99 [02:28<01:32,  2.44s/batch]  trog:  63%|██████▎   | 62/99 [02:31<01:30,  2.44s/batch]  trog:  64%|██████▎   | 63/99 [02:34<01:36,  2.69s/batch]  trog:  65%|██████▍   | 64/99 [02:37<01:39,  2.85s/batch]  trog:  66%|██████▌   | 65/99 [02:40<01:31,  2.69s/batch]  trog:  67%|██████▋   | 66/99 [02:42<01:28,  2.67s/batch]  trog:  68%|██████▊   | 67/99 [02:46<01:30,  2.84s/batch]  trog:  69%|██████▊   | 68/99 [02:49<01:34,  3.06s/batch]  trog:  70%|██████▉   | 69/99 [02:52<01:30,  3.03s/batch]  trog:  71%|███████   | 70/99 [02:55<01:28,  3.04s/batch]  trog:  72%|███████▏  | 71/99 [02:57<01:16,  2.72s/batch]  trog:  73%|███████▎  | 72/99 [03:00<01:12,  2.68s/batch]  trog:  74%|███████▎  | 73/99 [03:03<01:12,  2.81s/batch]  trog:  75%|███████▍  | 74/99 [03:06<01:10,  2.82s/batch]  trog:  76%|███████▌  | 75/99 [03:08<01:04,  2.68s/batch]  trog:  77%|███████▋  | 76/99 [03:12<01:11,  3.13s/batch]  trog:  78%|███████▊  | 77/99 [03:16<01:10,  3.22s/batch]  trog:  79%|███████▉  | 78/99 [03:20<01:12,  3.44s/batch]  trog:  80%|███████▉  | 79/99 [03:22<01:01,  3.09s/batch]  trog:  81%|████████  | 80/99 [03:25<00:57,  3.02s/batch]  trog:  82%|████████▏ | 81/99 [03:28<00:55,  3.06s/batch]  trog:  83%|████████▎ | 82/99 [03:31<00:51,  3.05s/batch]  trog:  84%|████████▍ | 83/99 [03:34<00:47,  2.97s/batch]  trog:  85%|████████▍ | 84/99 [03:36<00:43,  2.87s/batch]  trog:  86%|████████▌ | 85/99 [03:39<00:38,  2.77s/batch]  trog:  87%|████████▋ | 86/99 [03:41<00:33,  2.60s/batch]  trog:  88%|████████▊ | 87/99 [03:45<00:35,  2.93s/batch]  trog:  89%|████████▉ | 88/99 [03:48<00:32,  2.92s/batch]  trog:  90%|████████▉ | 89/99 [03:52<00:32,  3.27s/batch]  trog:  91%|█████████ | 90/99 [03:54<00:27,  3.08s/batch]  trog:  92%|█████████▏| 91/99 [03:58<00:25,  3.14s/batch]  trog:  93%|█████████▎| 92/99 [04:02<00:24,  3.43s/batch]  trog:  94%|█████████▍| 93/99 [04:04<00:18,  3.12s/batch]  trog:  95%|█████████▍| 94/99 [04:07<00:15,  3.03s/batch]  trog:  96%|█████████▌| 95/99 [04:10<00:11,  2.95s/batch]  trog:  97%|█████████▋| 96/99 [04:12<00:08,  2.88s/batch]  trog:  98%|█████████▊| 97/99 [04:17<00:06,  3.30s/batch]  trog:  99%|█████████▉| 98/99 [04:21<00:03,  3.67s/batch]  trog: 100%|██████████| 99/99 [04:24<00:00,  3.45s/batch]  trog: 100%|██████████| 99/99 [04:24<00:00,  2.67s/batch]
  gpt53/trog: 0.9798 (97/99)
  vocab: loaded human proportions for 164 items
  vocab:   0%|          | 0/170 [00:00<?, ?batch/s]  vocab:   1%|          | 1/170 [00:01<04:23,  1.56s/batch]  vocab:   1%|          | 2/170 [00:03<04:27,  1.59s/batch]  vocab:   2%|▏         | 3/170 [00:05<04:45,  1.71s/batch]  vocab:   2%|▏         | 4/170 [00:07<05:34,  2.02s/batch]  vocab:   3%|▎         | 5/170 [00:08<04:58,  1.81s/batch]  vocab:   4%|▎         | 6/170 [00:10<04:39,  1.71s/batch]  vocab:   4%|▍         | 7/170 [00:12<04:49,  1.77s/batch]  vocab:   5%|▍         | 8/170 [00:14<04:59,  1.85s/batch]  vocab:   5%|▌         | 9/170 [00:16<04:57,  1.85s/batch]  vocab:   6%|▌         | 10/170 [00:18<05:21,  2.01s/batch]  vocab:   6%|▋         | 11/170 [00:20<05:09,  1.95s/batch]  vocab:   7%|▋         | 12/170 [00:22<04:58,  1.89s/batch]  vocab:   8%|▊         | 13/170 [00:24<05:39,  2.16s/batch]  vocab:   8%|▊         | 14/170 [00:27<05:54,  2.27s/batch]  vocab:   9%|▉         | 15/170 [00:29<05:27,  2.11s/batch]  vocab:   9%|▉         | 16/170 [00:30<04:55,  1.92s/batch]  vocab:  10%|█         | 17/170 [00:32<04:33,  1.79s/batch]  vocab:  11%|█         | 18/170 [00:34<04:45,  1.88s/batch]  vocab:  11%|█         | 19/170 [00:35<04:23,  1.74s/batch]  vocab:  12%|█▏        | 20/170 [00:37<04:41,  1.87s/batch]  vocab:  12%|█▏        | 21/170 [00:39<04:46,  1.92s/batch]  vocab:  13%|█▎        | 22/170 [00:41<04:48,  1.95s/batch]  vocab:  14%|█▎        | 23/170 [00:44<04:54,  2.01s/batch]  vocab:  14%|█▍        | 24/170 [00:46<04:53,  2.01s/batch]  vocab:  15%|█▍        | 25/170 [00:47<04:38,  1.92s/batch]  vocab:  15%|█▌        | 26/170 [00:49<04:35,  1.92s/batch]  vocab:  16%|█▌        | 27/170 [00:52<05:21,  2.25s/batch]  vocab:  16%|█▋        | 28/170 [00:55<05:24,  2.28s/batch]  vocab:  17%|█▋        | 29/170 [00:57<05:46,  2.46s/batch]  vocab:  18%|█▊        | 30/170 [00:59<05:13,  2.24s/batch]  vocab:  18%|█▊        | 31/170 [01:01<04:56,  2.14s/batch]  vocab:  19%|█▉        | 32/170 [01:03<04:50,  2.10s/batch]  vocab:  19%|█▉        | 33/170 [01:05<04:40,  2.05s/batch]  vocab:  20%|██        | 34/170 [01:07<04:24,  1.94s/batch]  vocab:  21%|██        | 35/170 [01:09<04:55,  2.19s/batch]  vocab:  21%|██        | 36/170 [01:12<04:56,  2.21s/batch]  vocab:  22%|██▏       | 37/170 [01:13<04:25,  2.00s/batch]  vocab:  22%|██▏       | 38/170 [01:15<04:30,  2.05s/batch]  vocab:  23%|██▎       | 39/170 [01:17<04:15,  1.95s/batch]  vocab:  24%|██▎       | 40/170 [01:19<04:22,  2.02s/batch]  vocab:  24%|██▍       | 41/170 [01:22<04:47,  2.22s/batch]  vocab:  25%|██▍       | 42/170 [01:24<04:43,  2.21s/batch]  vocab:  25%|██▌       | 43/170 [01:26<04:29,  2.12s/batch]  vocab:  26%|██▌       | 44/170 [01:28<04:18,  2.05s/batch]  vocab:  26%|██▋       | 45/170 [01:30<04:24,  2.12s/batch]  vocab:  27%|██▋       | 46/170 [01:32<04:16,  2.07s/batch]  vocab:  28%|██▊       | 47/170 [01:34<04:06,  2.00s/batch]  vocab:  28%|██▊       | 48/170 [01:36<04:07,  2.03s/batch]  vocab:  29%|██▉       | 49/170 [01:39<04:23,  2.18s/batch]  vocab:  29%|██▉       | 50/170 [01:41<04:39,  2.33s/batch]  vocab:  30%|███       | 51/170 [01:44<04:45,  2.40s/batch]  vocab:  31%|███       | 52/170 [01:46<04:15,  2.17s/batch]  vocab:  31%|███       | 53/170 [01:47<03:57,  2.03s/batch]  vocab:  32%|███▏      | 54/170 [01:50<04:08,  2.15s/batch]  vocab:  32%|███▏      | 55/170 [01:51<03:53,  2.03s/batch]  vocab:  33%|███▎      | 56/170 [01:53<03:47,  1.99s/batch]  vocab:  34%|███▎      | 57/170 [01:56<04:15,  2.26s/batch]  vocab:  34%|███▍      | 58/170 [01:58<04:10,  2.23s/batch]  vocab:  35%|███▍      | 59/170 [02:01<04:13,  2.28s/batch]  vocab:  35%|███▌      | 60/170 [02:04<04:38,  2.53s/batch]  vocab:  36%|███▌      | 61/170 [02:06<04:22,  2.41s/batch]  vocab:  36%|███▋      | 62/170 [02:09<04:28,  2.48s/batch]  vocab:  37%|███▋      | 63/170 [02:10<04:00,  2.25s/batch]  vocab:  38%|███▊      | 64/170 [02:13<03:55,  2.22s/batch]  vocab:  38%|███▊      | 65/170 [02:15<03:53,  2.23s/batch]  vocab:  39%|███▉      | 66/170 [02:17<03:35,  2.07s/batch]  vocab:  39%|███▉      | 67/170 [02:19<03:34,  2.08s/batch]  vocab:  40%|████      | 68/170 [02:20<03:16,  1.93s/batch]  vocab:  41%|████      | 69/170 [02:22<03:20,  1.98s/batch]  vocab:  41%|████      | 70/170 [02:24<03:18,  1.98s/batch]  vocab:  42%|████▏     | 71/170 [02:27<03:45,  2.28s/batch]  vocab:  42%|████▏     | 72/170 [02:29<03:25,  2.10s/batch]  vocab:  43%|████▎     | 73/170 [02:31<03:27,  2.14s/batch]  vocab:  44%|████▎     | 74/170 [02:33<03:28,  2.17s/batch]  vocab:  44%|████▍     | 75/170 [02:37<03:54,  2.47s/batch]  vocab:  45%|████▍     | 76/170 [02:39<03:48,  2.43s/batch]  vocab:  45%|████▌     | 77/170 [02:41<03:38,  2.35s/batch]  vocab:  46%|████▌     | 78/170 [02:43<03:25,  2.24s/batch]  vocab:  46%|████▋     | 79/170 [02:45<03:15,  2.15s/batch]  vocab:  47%|████▋     | 80/170 [02:47<03:05,  2.06s/batch]  vocab:  48%|████▊     | 81/170 [02:49<03:00,  2.02s/batch]  vocab:  48%|████▊     | 82/170 [02:50<02:48,  1.92s/batch]  vocab:  49%|████▉     | 83/170 [02:54<03:24,  2.35s/batch]  vocab:  49%|████▉     | 84/170 [02:55<03:04,  2.15s/batch]  vocab:  50%|█████     | 85/170 [02:57<02:50,  2.01s/batch]  vocab:  51%|█████     | 86/170 [02:59<02:33,  1.83s/batch]  vocab:  51%|█████     | 87/170 [03:01<02:40,  1.93s/batch]  vocab:  52%|█████▏    | 88/170 [03:03<02:36,  1.90s/batch]  vocab:  52%|█████▏    | 89/170 [03:04<02:28,  1.83s/batch]  vocab:  53%|█████▎    | 90/170 [03:06<02:17,  1.71s/batch]  vocab:  54%|█████▎    | 91/170 [03:08<02:39,  2.01s/batch]  vocab:  54%|█████▍    | 92/170 [03:10<02:36,  2.01s/batch]  vocab:  55%|█████▍    | 93/170 [03:12<02:34,  2.01s/batch]  vocab:  55%|█████▌    | 94/170 [03:14<02:25,  1.92s/batch]  vocab:  56%|█████▌    | 95/170 [03:16<02:34,  2.05s/batch]  vocab:  56%|█████▋    | 96/170 [03:18<02:27,  2.00s/batch]  vocab:  57%|█████▋    | 97/170 [03:20<02:22,  1.96s/batch]  vocab:  58%|█████▊    | 98/170 [03:22<02:24,  2.01s/batch]  vocab:  58%|█████▊    | 99/170 [03:24<02:21,  1.99s/batch]  vocab:  59%|█████▉    | 100/170 [03:27<02:36,  2.23s/batch]  vocab:  59%|█████▉    | 101/170 [03:29<02:26,  2.12s/batch]  vocab:  60%|██████    | 102/170 [03:31<02:28,  2.18s/batch]  vocab:  61%|██████    | 103/170 [03:33<02:22,  2.13s/batch]  vocab:  61%|██████    | 104/170 [03:35<02:19,  2.11s/batch]  vocab:  62%|██████▏   | 105/170 [03:38<02:25,  2.24s/batch]  vocab:  62%|██████▏   | 106/170 [03:40<02:19,  2.17s/batch]  vocab:  63%|██████▎   | 107/170 [03:43<02:26,  2.33s/batch]  vocab:  64%|██████▎   | 108/170 [03:44<02:15,  2.18s/batch]  vocab:  64%|██████▍   | 109/170 [03:47<02:28,  2.44s/batch]  vocab:  65%|██████▍   | 110/170 [03:50<02:21,  2.36s/batch]  vocab:  65%|██████▌   | 111/170 [03:51<02:10,  2.21s/batch]  vocab:  66%|██████▌   | 112/170 [03:54<02:05,  2.16s/batch]  vocab:  66%|██████▋   | 113/170 [03:56<02:11,  2.30s/batch]  vocab:  67%|██████▋   | 114/170 [03:58<02:08,  2.29s/batch]  vocab:  68%|██████▊   | 115/170 [04:00<01:59,  2.18s/batch]  vocab:  68%|██████▊   | 116/170 [04:03<01:59,  2.22s/batch]  vocab:  69%|██████▉   | 117/170 [04:05<01:52,  2.12s/batch]  vocab:  69%|██████▉   | 118/170 [04:09<02:24,  2.79s/batch]  vocab:  70%|███████   | 119/170 [04:12<02:21,  2.77s/batch]  vocab:  71%|███████   | 120/170 [04:14<02:17,  2.75s/batch]  vocab:  71%|███████   | 121/170 [04:17<02:07,  2.61s/batch]  vocab:  72%|███████▏  | 122/170 [04:19<02:03,  2.57s/batch]  vocab:  72%|███████▏  | 123/170 [04:21<01:53,  2.41s/batch]  vocab:  73%|███████▎  | 124/170 [04:24<01:50,  2.41s/batch]  vocab:  74%|███████▎  | 125/170 [04:26<01:47,  2.40s/batch]  vocab:  74%|███████▍  | 126/170 [04:28<01:40,  2.28s/batch]  vocab:  75%|███████▍  | 127/170 [04:30<01:35,  2.23s/batch]  vocab:  75%|███████▌  | 128/170 [04:32<01:29,  2.12s/batch]  vocab:  76%|███████▌  | 129/170 [04:34<01:25,  2.08s/batch]  vocab:  76%|███████▋  | 130/170 [04:36<01:25,  2.15s/batch]  vocab:  77%|███████▋  | 131/170 [04:38<01:23,  2.13s/batch]  vocab:  78%|███████▊  | 132/170 [04:40<01:18,  2.05s/batch]  vocab:  78%|███████▊  | 133/170 [04:43<01:21,  2.21s/batch]  vocab:  79%|███████▉  | 134/170 [04:44<01:14,  2.08s/batch]  vocab:  79%|███████▉  | 135/170 [04:49<01:33,  2.66s/batch]  vocab:  80%|████████  | 136/170 [04:50<01:20,  2.38s/batch]  vocab:  81%|████████  | 137/170 [04:52<01:16,  2.33s/batch]  vocab:  81%|████████  | 138/170 [04:56<01:28,  2.76s/batch]  vocab:  82%|████████▏ | 139/170 [04:58<01:18,  2.55s/batch]  vocab:  82%|████████▏ | 140/170 [05:00<01:10,  2.36s/batch]  vocab:  83%|████████▎ | 141/170 [05:02<01:05,  2.26s/batch]  vocab:  84%|████████▎ | 142/170 [05:04<01:00,  2.15s/batch]  vocab:  84%|████████▍ | 143/170 [05:06<00:54,  2.03s/batch]  vocab:  85%|████████▍ | 144/170 [05:08<00:51,  1.97s/batch]  vocab:  85%|████████▌ | 145/170 [05:10<00:48,  1.95s/batch]  vocab:  86%|████████▌ | 146/170 [05:12<00:46,  1.96s/batch]  vocab:  86%|████████▋ | 147/170 [05:14<00:48,  2.10s/batch]  vocab:  87%|████████▋ | 148/170 [05:16<00:46,  2.11s/batch]  vocab:  88%|████████▊ | 149/170 [05:18<00:43,  2.07s/batch]  vocab:  88%|████████▊ | 150/170 [05:21<00:48,  2.42s/batch]  vocab:  89%|████████▉ | 151/170 [05:23<00:44,  2.34s/batch]  vocab:  89%|████████▉ | 152/170 [05:26<00:40,  2.27s/batch]  vocab:  90%|█████████ | 153/170 [05:28<00:38,  2.26s/batch]  vocab:  91%|█████████ | 154/170 [05:30<00:34,  2.18s/batch]  vocab:  91%|█████████ | 155/170 [05:32<00:32,  2.18s/batch]  vocab:  92%|█████████▏| 156/170 [05:34<00:31,  2.24s/batch]  vocab:  92%|█████████▏| 157/170 [05:36<00:28,  2.16s/batch]  vocab:  93%|█████████▎| 158/170 [05:39<00:27,  2.30s/batch]  vocab:  94%|█████████▎| 159/170 [05:41<00:24,  2.26s/batch]  vocab:  94%|█████████▍| 160/170 [05:43<00:22,  2.25s/batch]  vocab:  95%|█████████▍| 161/170 [05:46<00:20,  2.28s/batch]  vocab:  95%|█████████▌| 162/170 [05:48<00:17,  2.24s/batch]  vocab:  96%|█████████▌| 163/170 [05:50<00:15,  2.18s/batch]  vocab:  96%|█████████▋| 164/170 [05:53<00:14,  2.42s/batch]  vocab:  97%|█████████▋| 165/170 [05:56<00:13,  2.72s/batch]  vocab:  98%|█████████▊| 166/170 [06:00<00:12,  3.15s/batch]  vocab:  98%|█████████▊| 167/170 [06:02<00:08,  2.79s/batch]  vocab:  99%|█████████▉| 168/170 [06:04<00:04,  2.46s/batch]  vocab:  99%|█████████▉| 169/170 [06:06<00:02,  2.31s/batch]  vocab: 100%|██████████| 170/170 [06:08<00:00,  2.29s/batch]  vocab: 100%|██████████| 170/170 [06:08<00:00,  2.17s/batch]
  gpt53/vocab: 0.9941 (169/170)
  Summary: /home/david/levante/levante-bench/results_api_retry_metadata/v1/gpt53/summary.csv
  egma-math: loaded human proportions for 327 items
  egma-math:   0%|          | 0/179 [00:00<?, ?batch/s]  egma-math:   1%|          | 1/179 [00:04<14:00,  4.72s/batch]  egma-math:   1%|          | 2/179 [00:08<12:43,  4.31s/batch]  egma-math:   2%|▏         | 3/179 [00:12<11:53,  4.06s/batch]  egma-math:   2%|▏         | 4/179 [00:16<11:23,  3.90s/batch]  egma-math:   3%|▎         | 5/179 [00:21<12:46,  4.40s/batch]  egma-math:   3%|▎         | 6/179 [00:26<13:04,  4.53s/batch]  egma-math:   4%|▍         | 7/179 [00:30<12:21,  4.31s/batch]  egma-math:   4%|▍         | 8/179 [00:35<12:55,  4.54s/batch]  egma-math:   5%|▌         | 9/179 [00:39<12:24,  4.38s/batch]  egma-math:   6%|▌         | 10/179 [00:43<12:12,  4.34s/batch]  egma-math:   6%|▌         | 11/179 [00:47<11:58,  4.28s/batch]  egma-math:   7%|▋         | 12/179 [00:51<11:48,  4.24s/batch]  egma-math:   7%|▋         | 13/179 [00:55<11:34,  4.18s/batch]  egma-math:   8%|▊         | 14/179 [01:00<11:51,  4.31s/batch]  egma-math:   8%|▊         | 15/179 [01:04<11:37,  4.25s/batch]  egma-math:   9%|▉         | 16/179 [01:08<11:32,  4.25s/batch]  egma-math:   9%|▉         | 17/179 [01:13<11:55,  4.42s/batch]  egma-math:  10%|█         | 18/179 [01:18<12:00,  4.47s/batch]  egma-math:  11%|█         | 19/179 [01:23<12:36,  4.73s/batch]  egma-math:  11%|█         | 20/179 [01:27<11:44,  4.43s/batch]  egma-math:  12%|█▏        | 21/179 [01:31<11:53,  4.51s/batch]  egma-math:  12%|█▏        | 22/179 [01:36<11:34,  4.42s/batch]  egma-math:  13%|█▎        | 23/179 [01:40<11:17,  4.34s/batch]  egma-math:  13%|█▎        | 24/179 [01:44<10:45,  4.17s/batch]  egma-math:  14%|█▍        | 25/179 [01:47<10:19,  4.03s/batch]  egma-math:  15%|█▍        | 26/179 [01:51<09:50,  3.86s/batch]  egma-math:  15%|█▌        | 27/179 [01:55<09:58,  3.94s/batch]  egma-math:  16%|█▌        | 28/179 [01:59<10:15,  4.08s/batch]  egma-math:  16%|█▌        | 29/179 [02:03<10:01,  4.01s/batch]  egma-math:  17%|█▋        | 30/179 [02:07<09:47,  3.95s/batch]  egma-math:  17%|█▋        | 31/179 [02:11<09:50,  3.99s/batch]  egma-math:  18%|█▊        | 32/179 [02:15<09:50,  4.02s/batch]  egma-math:  18%|█▊        | 33/179 [02:19<09:31,  3.92s/batch]  egma-math:  19%|█▉        | 34/179 [02:23<09:36,  3.98s/batch]  egma-math:  20%|█▉        | 35/179 [02:27<09:36,  4.00s/batch]  egma-math:  20%|██        | 36/179 [02:31<09:53,  4.15s/batch]  egma-math:  21%|██        | 37/179 [02:36<09:55,  4.19s/batch]  egma-math:  21%|██        | 38/179 [02:40<10:06,  4.30s/batch]  egma-math:  22%|██▏       | 39/179 [02:45<10:29,  4.50s/batch]  egma-math:  22%|██▏       | 40/179 [02:50<10:40,  4.61s/batch]  egma-math:  23%|██▎       | 41/179 [02:55<10:48,  4.70s/batch]  egma-math:  23%|██▎       | 42/179 [02:59<10:10,  4.45s/batch]  egma-math:  24%|██▍       | 43/179 [03:03<10:06,  4.46s/batch]  egma-math:  25%|██▍       | 44/179 [03:07<09:27,  4.20s/batch]  egma-math:  25%|██▌       | 45/179 [03:12<09:55,  4.45s/batch]  egma-math:  26%|██▌       | 46/179 [03:16<09:43,  4.39s/batch]  egma-math:  26%|██▋       | 47/179 [03:20<09:17,  4.22s/batch]  egma-math:  27%|██▋       | 48/179 [03:24<09:07,  4.18s/batch]  egma-math:  27%|██▋       | 49/179 [03:28<08:39,  3.99s/batch]  egma-math:  28%|██▊       | 50/179 [03:32<08:33,  3.98s/batch]  egma-math:  28%|██▊       | 51/179 [03:36<08:33,  4.01s/batch]  egma-math:  29%|██▉       | 52/179 [03:39<08:17,  3.92s/batch]  egma-math:  30%|██▉       | 53/179 [03:44<08:21,  3.98s/batch]  egma-math:  30%|███       | 54/179 [03:49<08:58,  4.31s/batch]  egma-math:  31%|███       | 55/179 [03:52<08:35,  4.16s/batch]  egma-math:  31%|███▏      | 56/179 [03:57<08:44,  4.26s/batch]  egma-math:  32%|███▏      | 57/179 [04:01<08:35,  4.23s/batch]  egma-math:  32%|███▏      | 58/179 [04:05<08:34,  4.26s/batch]  egma-math:  33%|███▎      | 59/179 [04:10<08:41,  4.35s/batch]  egma-math:  34%|███▎      | 60/179 [04:15<09:05,  4.59s/batch]  egma-math:  34%|███▍      | 61/179 [04:19<08:40,  4.41s/batch]  egma-math:  35%|███▍      | 62/179 [04:23<08:21,  4.28s/batch]  egma-math:  35%|███▌      | 63/179 [04:27<08:19,  4.31s/batch]  egma-math:  36%|███▌      | 64/179 [04:32<08:06,  4.23s/batch]  egma-math:  36%|███▋      | 65/179 [04:36<07:54,  4.16s/batch]  egma-math:  37%|███▋      | 66/179 [04:40<07:48,  4.15s/batch]  egma-math:  37%|███▋      | 67/179 [04:44<07:56,  4.25s/batch]  egma-math:  38%|███▊      | 68/179 [04:48<07:41,  4.16s/batch]  egma-math:  39%|███▊      | 69/179 [04:52<07:40,  4.19s/batch]  egma-math:  39%|███▉      | 70/179 [04:56<07:28,  4.11s/batch]  egma-math:  40%|███▉      | 71/179 [05:00<07:21,  4.08s/batch]  egma-math:  40%|████      | 72/179 [05:06<07:54,  4.44s/batch]  egma-math:  41%|████      | 73/179 [05:10<07:43,  4.37s/batch]  egma-math:  41%|████▏     | 74/179 [05:14<07:36,  4.35s/batch]  egma-math:  42%|████▏     | 75/179 [05:18<07:19,  4.23s/batch]  egma-math:  42%|████▏     | 76/179 [05:22<07:05,  4.13s/batch]  egma-math:  43%|████▎     | 77/179 [05:26<06:45,  3.98s/batch]  egma-math:  44%|████▎     | 78/179 [05:29<06:24,  3.81s/batch]  egma-math:  44%|████▍     | 79/179 [05:33<06:32,  3.92s/batch]  egma-math:  45%|████▍     | 80/179 [05:37<06:29,  3.93s/batch]  egma-math:  45%|████▌     | 81/179 [05:41<06:38,  4.07s/batch]  egma-math:  46%|████▌     | 82/179 [05:45<06:33,  4.06s/batch]  egma-math:  46%|████▋     | 83/179 [05:50<06:38,  4.16s/batch]  egma-math:  47%|████▋     | 84/179 [05:54<06:36,  4.17s/batch]  egma-math:  47%|████▋     | 85/179 [05:58<06:36,  4.22s/batch]  egma-math:  48%|████▊     | 86/179 [06:02<06:12,  4.00s/batch]  egma-math:  49%|████▊     | 87/179 [06:06<06:09,  4.02s/batch]  egma-math:  49%|████▉     | 88/179 [06:10<05:58,  3.94s/batch]  egma-math:  50%|████▉     | 89/179 [06:14<05:55,  3.95s/batch]  egma-math:  50%|█████     | 90/179 [06:17<05:43,  3.86s/batch]  egma-math:  51%|█████     | 91/179 [06:23<06:14,  4.25s/batch]  egma-math:  51%|█████▏    | 92/179 [06:26<06:02,  4.17s/batch]  egma-math:  52%|█████▏    | 93/179 [06:31<06:09,  4.30s/batch]  egma-math:  53%|█████▎    | 94/179 [06:35<05:51,  4.14s/batch]  egma-math:  53%|█████▎    | 95/179 [06:38<05:34,  3.98s/batch]  egma-math:  54%|█████▎    | 96/179 [06:42<05:27,  3.94s/batch]  egma-math:  54%|█████▍    | 97/179 [06:46<05:12,  3.81s/batch]  egma-math:  55%|█████▍    | 98/179 [06:50<05:06,  3.78s/batch]  egma-math:  55%|█████▌    | 99/179 [06:54<05:30,  4.13s/batch]  egma-math:  56%|█████▌    | 100/179 [06:58<05:15,  4.00s/batch]  egma-math:  56%|█████▋    | 101/179 [07:02<05:09,  3.97s/batch]  egma-math:  57%|█████▋    | 102/179 [07:07<05:35,  4.35s/batch]  egma-math:  58%|█████▊    | 103/179 [07:11<05:20,  4.21s/batch]  egma-math:  58%|█████▊    | 104/179 [07:15<05:10,  4.13s/batch]  egma-math:  59%|█████▊    | 105/179 [07:19<04:57,  4.01s/batch]  egma-math:  59%|█████▉    | 106/179 [07:23<04:56,  4.06s/batch]  egma-math:  60%|█████▉    | 107/179 [07:28<05:12,  4.34s/batch]  egma-math:  60%|██████    | 108/179 [07:32<05:09,  4.36s/batch]  egma-math:  61%|██████    | 109/179 [07:36<04:53,  4.19s/batch]  egma-math:  61%|██████▏   | 110/179 [07:41<04:51,  4.22s/batch]  egma-math:  62%|██████▏   | 111/179 [07:45<04:59,  4.40s/batch]  egma-math:  63%|██████▎   | 112/179 [07:49<04:33,  4.09s/batch]  egma-math:  63%|██████▎   | 113/179 [07:52<04:18,  3.92s/batch]  egma-math:  64%|██████▎   | 114/179 [07:56<04:12,  3.88s/batch]  egma-math:  64%|██████▍   | 115/179 [08:01<04:26,  4.17s/batch]  egma-math:  65%|██████▍   | 116/179 [08:05<04:13,  4.02s/batch]  egma-math:  65%|██████▌   | 117/179 [08:08<04:07,  3.99s/batch]  egma-math:  66%|██████▌   | 118/179 [08:12<04:01,  3.95s/batch]  egma-math:  66%|██████▋   | 119/179 [08:16<03:47,  3.79s/batch]  egma-math:  67%|██████▋   | 120/179 [08:20<03:55,  3.98s/batch]  egma-math:  68%|██████▊   | 121/179 [08:24<03:49,  3.96s/batch]  egma-math:  68%|██████▊   | 122/179 [08:28<03:44,  3.94s/batch]  egma-math:  69%|██████▊   | 123/179 [08:32<03:49,  4.10s/batch]  egma-math:  69%|██████▉   | 124/179 [08:37<03:45,  4.09s/batch]  egma-math:  70%|██████▉   | 125/179 [08:41<03:41,  4.11s/batch]  egma-math:  70%|███████   | 126/179 [08:45<03:34,  4.04s/batch]  egma-math:  71%|███████   | 127/179 [08:49<03:34,  4.13s/batch]  egma-math:  72%|███████▏  | 128/179 [08:53<03:26,  4.05s/batch]  egma-math:  72%|███████▏  | 129/179 [08:57<03:21,  4.04s/batch]  egma-math:  73%|███████▎  | 130/179 [09:00<03:11,  3.90s/batch]  egma-math:  73%|███████▎  | 131/179 [09:05<03:21,  4.20s/batch]  egma-math:  74%|███████▎  | 132/179 [09:10<03:20,  4.27s/batch]  egma-math:  74%|███████▍  | 133/179 [09:13<03:08,  4.10s/batch]  egma-math:  75%|███████▍  | 134/179 [09:18<03:09,  4.21s/batch]  egma-math:  75%|███████▌  | 135/179 [09:21<02:56,  4.02s/batch]  egma-math:  76%|███████▌  | 136/179 [09:26<02:59,  4.17s/batch]  egma-math:  77%|███████▋  | 137/179 [09:30<02:48,  4.02s/batch]  egma-math:  77%|███████▋  | 138/179 [09:34<02:45,  4.04s/batch]  egma-math:  78%|███████▊  | 139/179 [09:38<02:49,  4.24s/batch]  egma-math:  78%|███████▊  | 140/179 [09:43<02:52,  4.43s/batch]  egma-math:  79%|███████▉  | 141/179 [09:47<02:43,  4.30s/batch]  egma-math:  79%|███████▉  | 142/179 [09:51<02:33,  4.14s/batch]  egma-math:  80%|███████▉  | 143/179 [09:55<02:28,  4.12s/batch]  egma-math:  80%|████████  | 144/179 [10:00<02:27,  4.21s/batch]  egma-math:  81%|████████  | 145/179 [10:04<02:23,  4.23s/batch]  egma-math:  82%|████████▏ | 146/179 [10:08<02:22,  4.32s/batch]  egma-math:  82%|████████▏ | 147/179 [10:13<02:23,  4.48s/batch]  egma-math:  83%|████████▎ | 148/179 [10:17<02:15,  4.38s/batch]  egma-math:  83%|████████▎ | 149/179 [10:21<02:08,  4.27s/batch]  egma-math:  84%|████████▍ | 150/179 [10:26<02:07,  4.39s/batch]  egma-math:  84%|████████▍ | 151/179 [10:32<02:15,  4.85s/batch]  egma-math:  85%|████████▍ | 152/179 [10:37<02:13,  4.93s/batch]  egma-math:  85%|████████▌ | 153/179 [10:43<02:12,  5.08s/batch]  egma-math:  86%|████████▌ | 154/179 [10:48<02:12,  5.29s/batch]  egma-math:  87%|████████▋ | 155/179 [10:53<02:01,  5.08s/batch]  egma-math:  87%|████████▋ | 156/179 [10:58<01:54,  4.99s/batch]  egma-math:  88%|████████▊ | 157/179 [11:04<01:58,  5.37s/batch]  egma-math:  88%|████████▊ | 158/179 [11:11<02:01,  5.77s/batch]  egma-math:  89%|████████▉ | 159/179 [11:17<01:56,  5.81s/batch]  egma-math:  89%|████████▉ | 160/179 [11:20<01:37,  5.13s/batch]  egma-math:  90%|████████▉ | 161/179 [11:25<01:29,  4.96s/batch]  egma-math:  91%|█████████ | 162/179 [11:29<01:19,  4.70s/batch]  egma-math:  91%|█████████ | 163/179 [11:32<01:10,  4.42s/batch]  egma-math:  92%|█████████▏| 164/179 [11:36<01:03,  4.25s/batch]  egma-math:  92%|█████████▏| 165/179 [11:40<00:58,  4.20s/batch]  egma-math:  93%|█████████▎| 166/179 [11:44<00:52,  4.01s/batch]  egma-math:  93%|█████████▎| 167/179 [11:48<00:49,  4.12s/batch]  egma-math:  95%|█████████▍| 170/179 [11:53<00:23,  2.63s/batch]  egma-math:  96%|█████████▌| 171/179 [11:57<00:23,  2.94s/batch]  egma-math:  96%|█████████▌| 172/179 [12:01<00:22,  3.22s/batch]  egma-math: 100%|██████████| 179/179 [12:01<00:00,  4.03s/batch]
  gemini_pro/egma-math: wrote /home/david/levante/levante-bench/results_api_retry_metadata/v1/gemini_pro/egma-math-by-type.csv
  gemini_pro/egma-math: 0.9665 (173/179)
  matrix-reasoning: loaded human proportions for 157 items
  matrix-reasoning:   0%|          | 0/80 [00:00<?, ?batch/s]  matrix-reasoning:   1%|▏         | 1/80 [00:04<05:56,  4.51s/batch]  matrix-reasoning:   2%|▎         | 2/80 [00:10<07:01,  5.40s/batch]  matrix-reasoning:   4%|▍         | 3/80 [00:15<06:44,  5.26s/batch]  matrix-reasoning:   5%|▌         | 4/80 [00:20<06:28,  5.12s/batch]  matrix-reasoning:   6%|▋         | 5/80 [00:26<06:44,  5.39s/batch]  matrix-reasoning:   8%|▊         | 6/80 [00:33<07:12,  5.84s/batch]  matrix-reasoning:   9%|▉         | 7/80 [00:38<06:48,  5.59s/batch]  matrix-reasoning:  10%|█         | 8/80 [00:42<06:24,  5.34s/batch]  matrix-reasoning:  11%|█▏        | 9/80 [00:48<06:19,  5.34s/batch]  matrix-reasoning:  12%|█▎        | 10/80 [00:53<06:08,  5.26s/batch]  matrix-reasoning:  14%|█▍        | 11/80 [00:58<06:09,  5.36s/batch]  matrix-reasoning:  15%|█▌        | 12/80 [01:04<06:16,  5.54s/batch]  matrix-reasoning:  16%|█▋        | 13/80 [01:09<05:53,  5.27s/batch]  matrix-reasoning:  18%|█▊        | 14/80 [01:16<06:15,  5.69s/batch]  matrix-reasoning:  19%|█▉        | 15/80 [01:25<07:22,  6.81s/batch]  matrix-reasoning:  20%|██        | 16/80 [01:31<07:00,  6.57s/batch]  matrix-reasoning:  21%|██▏       | 17/80 [01:37<06:33,  6.25s/batch]  matrix-reasoning:  22%|██▎       | 18/80 [01:42<06:07,  5.92s/batch]  matrix-reasoning:  24%|██▍       | 19/80 [01:46<05:36,  5.52s/batch]  matrix-reasoning:  25%|██▌       | 20/80 [01:51<05:20,  5.35s/batch]  matrix-reasoning:  26%|██▋       | 21/80 [01:56<05:09,  5.24s/batch]  matrix-reasoning:  28%|██▊       | 22/80 [02:03<05:21,  5.54s/batch]  matrix-reasoning:  29%|██▉       | 23/80 [02:07<05:02,  5.31s/batch]  matrix-reasoning:  30%|███       | 24/80 [02:13<04:58,  5.33s/batch]  matrix-reasoning:  31%|███▏      | 25/80 [02:18<04:54,  5.36s/batch]  matrix-reasoning:  32%|███▎      | 26/80 [02:25<05:10,  5.75s/batch]  matrix-reasoning:  34%|███▍      | 27/80 [02:31<05:11,  5.89s/batch]  matrix-reasoning:  35%|███▌      | 28/80 [02:37<05:00,  5.78s/batch]  matrix-reasoning:  36%|███▋      | 29/80 [02:42<04:55,  5.79s/batch]  matrix-reasoning:  38%|███▊      | 30/80 [02:48<04:44,  5.70s/batch]  matrix-reasoning:  39%|███▉      | 31/80 [02:52<04:22,  5.36s/batch]  matrix-reasoning:  40%|████      | 32/80 [02:57<04:10,  5.22s/batch]  matrix-reasoning:  41%|████▏     | 33/80 [03:04<04:25,  5.65s/batch]  matrix-reasoning:  42%|████▎     | 34/80 [03:10<04:21,  5.67s/batch]  matrix-reasoning:  44%|████▍     | 35/80 [03:15<04:12,  5.61s/batch]  matrix-reasoning:  45%|████▌     | 36/80 [03:21<04:08,  5.66s/batch]  matrix-reasoning:  46%|████▋     | 37/80 [03:26<03:57,  5.52s/batch]  matrix-reasoning:  48%|████▊     | 38/80 [03:32<03:58,  5.68s/batch]  matrix-reasoning:  49%|████▉     | 39/80 [03:37<03:40,  5.38s/batch]  matrix-reasoning:  50%|█████     | 40/80 [03:43<03:47,  5.69s/batch]  matrix-reasoning:  51%|█████▏    | 41/80 [03:49<03:40,  5.66s/batch]  matrix-reasoning:  52%|█████▎    | 42/80 [03:55<03:43,  5.89s/batch]  matrix-reasoning:  54%|█████▍    | 43/80 [04:00<03:29,  5.67s/batch]  matrix-reasoning:  55%|█████▌    | 44/80 [04:05<03:14,  5.41s/batch]  matrix-reasoning:  56%|█████▋    | 45/80 [04:22<05:07,  8.78s/batch]  matrix-reasoning:  57%|█████▊    | 46/80 [04:27<04:24,  7.77s/batch]  matrix-reasoning:  59%|█████▉    | 47/80 [04:32<03:48,  6.92s/batch]  matrix-reasoning:  60%|██████    | 48/80 [04:38<03:27,  6.48s/batch]  matrix-reasoning:  61%|██████▏   | 49/80 [04:46<03:36,  6.99s/batch]  matrix-reasoning:  62%|██████▎   | 50/80 [04:53<03:27,  6.93s/batch]  matrix-reasoning:  64%|██████▍   | 51/80 [04:58<03:08,  6.49s/batch]  matrix-reasoning:  65%|██████▌   | 52/80 [05:03<02:47,  5.99s/batch]  matrix-reasoning:  66%|██████▋   | 53/80 [05:08<02:33,  5.69s/batch]  matrix-reasoning:  68%|██████▊   | 54/80 [05:14<02:28,  5.70s/batch]  matrix-reasoning:  69%|██████▉   | 55/80 [05:20<02:25,  5.83s/batch]  matrix-reasoning:  70%|███████   | 56/80 [05:25<02:12,  5.52s/batch]  matrix-reasoning:  71%|███████▏  | 57/80 [05:30<02:07,  5.56s/batch]  matrix-reasoning:  72%|███████▎  | 58/80 [05:35<01:59,  5.41s/batch]  matrix-reasoning:  74%|███████▍  | 59/80 [05:41<01:54,  5.46s/batch]  matrix-reasoning:  75%|███████▌  | 60/80 [05:48<01:57,  5.85s/batch]  matrix-reasoning:  76%|███████▋  | 61/80 [05:53<01:51,  5.84s/batch]  matrix-reasoning:  78%|███████▊  | 62/80 [05:58<01:40,  5.59s/batch]  matrix-reasoning:  79%|███████▉  | 63/80 [06:05<01:37,  5.75s/batch]  matrix-reasoning:  80%|████████  | 64/80 [06:09<01:27,  5.48s/batch]  matrix-reasoning:  81%|████████▏ | 65/80 [06:16<01:25,  5.70s/batch]  matrix-reasoning:  82%|████████▎ | 66/80 [06:20<01:13,  5.28s/batch]  matrix-reasoning:  84%|████████▍ | 67/80 [06:26<01:10,  5.41s/batch]  matrix-reasoning:  85%|████████▌ | 68/80 [06:31<01:03,  5.30s/batch]  matrix-reasoning:  86%|████████▋ | 69/80 [06:36<00:56,  5.14s/batch]  matrix-reasoning:  88%|████████▊ | 70/80 [06:41<00:51,  5.13s/batch]  matrix-reasoning:  89%|████████▉ | 71/80 [06:45<00:43,  4.86s/batch]  matrix-reasoning:  90%|█████████ | 72/80 [06:51<00:42,  5.33s/batch]  matrix-reasoning:  91%|█████████▏| 73/80 [06:57<00:37,  5.37s/batch]  matrix-reasoning:  92%|█████████▎| 74/80 [07:02<00:31,  5.21s/batch]  matrix-reasoning:  94%|█████████▍| 75/80 [07:07<00:26,  5.27s/batch]  matrix-reasoning:  95%|█████████▌| 76/80 [07:13<00:21,  5.38s/batch]  matrix-reasoning:  96%|█████████▋| 77/80 [07:18<00:16,  5.35s/batch]  matrix-reasoning:  98%|█████████▊| 78/80 [07:23<00:10,  5.18s/batch]  matrix-reasoning:  99%|█████████▉| 79/80 [07:28<00:05,  5.11s/batch]  matrix-reasoning: 100%|██████████| 80/80 [07:34<00:00,  5.34s/batch]  matrix-reasoning: 100%|██████████| 80/80 [07:34<00:00,  5.68s/batch]
  gemini_pro/matrix-reasoning: 0.5500 (44/80)
  mental-rotation: loaded human proportions for 12 items
  mental-rotation:   0%|          | 0/83 [00:00<?, ?batch/s]  mental-rotation:   1%|          | 1/83 [00:04<06:44,  4.93s/batch]  mental-rotation:   2%|▏         | 2/83 [00:09<06:31,  4.83s/batch]  mental-rotation:   4%|▎         | 3/83 [00:14<06:23,  4.80s/batch]  mental-rotation:   5%|▍         | 4/83 [00:18<06:08,  4.67s/batch]  mental-rotation:   6%|▌         | 5/83 [00:23<06:03,  4.66s/batch]  mental-rotation:   7%|▋         | 6/83 [00:28<05:55,  4.61s/batch]  mental-rotation:   8%|▊         | 7/83 [00:34<06:32,  5.16s/batch]  mental-rotation:  10%|▉         | 8/83 [00:39<06:37,  5.30s/batch]  mental-rotation:  11%|█         | 9/83 [00:44<06:13,  5.05s/batch]  mental-rotation:  12%|█▏        | 10/83 [00:48<05:51,  4.81s/batch]  mental-rotation:  13%|█▎        | 11/83 [00:53<05:38,  4.70s/batch]  mental-rotation:  14%|█▍        | 12/83 [00:57<05:33,  4.70s/batch]  mental-rotation:  16%|█▌        | 13/83 [01:03<05:47,  4.96s/batch]  mental-rotation:  22%|██▏       | 18/83 [01:07<02:21,  2.18s/batch]  mental-rotation:  24%|██▍       | 20/83 [01:12<02:15,  2.15s/batch]  mental-rotation:  27%|██▋       | 22/83 [01:17<02:19,  2.28s/batch]  mental-rotation:  28%|██▊       | 23/83 [01:21<02:39,  2.65s/batch]  mental-rotation:  29%|██▉       | 24/83 [01:28<03:20,  3.40s/batch]  mental-rotation:  30%|███       | 25/83 [01:33<03:37,  3.75s/batch]  mental-rotation:  31%|███▏      | 26/83 [01:39<04:05,  4.31s/batch]  mental-rotation:  33%|███▎      | 27/83 [01:43<04:05,  4.39s/batch]  mental-rotation:  34%|███▎      | 28/83 [01:48<03:57,  4.32s/batch]  mental-rotation:  35%|███▍      | 29/83 [01:52<03:57,  4.40s/batch]  mental-rotation:  36%|███▌      | 30/83 [01:58<04:11,  4.74s/batch]  mental-rotation:  37%|███▋      | 31/83 [02:03<04:17,  4.96s/batch]  mental-rotation:  39%|███▊      | 32/83 [02:14<05:36,  6.59s/batch]  mental-rotation:  40%|███▉      | 33/83 [02:19<05:12,  6.25s/batch]  mental-rotation:  41%|████      | 34/83 [02:25<05:02,  6.17s/batch]  mental-rotation:  43%|████▎     | 36/83 [02:30<03:24,  4.34s/batch]  mental-rotation:  45%|████▍     | 37/83 [02:34<03:23,  4.42s/batch]  mental-rotation:  53%|█████▎    | 44/83 [02:39<01:09,  1.78s/batch]  mental-rotation:  54%|█████▍    | 45/83 [02:45<01:25,  2.24s/batch]  mental-rotation:  55%|█████▌    | 46/83 [02:49<01:34,  2.55s/batch]  mental-rotation:  57%|█████▋    | 47/83 [02:54<01:49,  3.04s/batch]  mental-rotation:  58%|█████▊    | 48/83 [02:59<01:56,  3.32s/batch]  mental-rotation:  59%|█████▉    | 49/83 [03:04<02:11,  3.88s/batch]  mental-rotation:  60%|██████    | 50/83 [03:10<02:20,  4.26s/batch]  mental-rotation:  61%|██████▏   | 51/83 [03:15<02:22,  4.46s/batch]  mental-rotation:  63%|██████▎   | 52/83 [03:19<02:17,  4.45s/batch]  mental-rotation:  64%|██████▍   | 53/83 [03:26<02:29,  4.97s/batch]  mental-rotation:  65%|██████▌   | 54/83 [03:31<02:28,  5.11s/batch]  mental-rotation:  66%|██████▋   | 55/83 [03:36<02:24,  5.17s/batch]  mental-rotation:  67%|██████▋   | 56/83 [03:41<02:15,  5.02s/batch]  mental-rotation:  69%|██████▊   | 57/83 [03:46<02:06,  4.86s/batch]  mental-rotation:  70%|██████▉   | 58/83 [03:51<02:05,  5.01s/batch]  mental-rotation:  71%|███████   | 59/83 [03:56<01:59,  4.99s/batch]  mental-rotation:  72%|███████▏  | 60/83 [04:01<01:53,  4.94s/batch]  mental-rotation:  73%|███████▎  | 61/83 [04:05<01:46,  4.83s/batch]  mental-rotation:  75%|███████▍  | 62/83 [04:10<01:39,  4.73s/batch]  mental-rotation:  76%|███████▌  | 63/83 [04:15<01:35,  4.79s/batch]  mental-rotation:  77%|███████▋  | 64/83 [04:20<01:34,  4.96s/batch]  mental-rotation:  78%|███████▊  | 65/83 [04:25<01:27,  4.88s/batch]  mental-rotation:  80%|███████▉  | 66/83 [04:30<01:22,  4.86s/batch]  mental-rotation:  81%|████████  | 67/83 [04:34<01:15,  4.71s/batch]  mental-rotation:  82%|████████▏ | 68/83 [04:39<01:12,  4.86s/batch]  mental-rotation:  83%|████████▎ | 69/83 [04:44<01:06,  4.74s/batch]  mental-rotation:  84%|████████▍ | 70/83 [04:48<01:00,  4.66s/batch]  mental-rotation:  86%|████████▌ | 71/83 [04:53<00:58,  4.90s/batch]  mental-rotation:  87%|████████▋ | 72/83 [04:58<00:53,  4.82s/batch]  mental-rotation:  88%|████████▊ | 73/83 [05:03<00:48,  4.82s/batch]  mental-rotation:  89%|████████▉ | 74/83 [05:08<00:44,  4.93s/batch]  mental-rotation:  90%|█████████ | 75/83 [05:13<00:38,  4.85s/batch]  mental-rotation:  92%|█████████▏| 76/83 [05:18<00:34,  4.92s/batch]  mental-rotation:  93%|█████████▎| 77/83 [05:23<00:29,  4.99s/batch]  mental-rotation:  94%|█████████▍| 78/83 [05:28<00:25,  5.11s/batch]  mental-rotation:  95%|█████████▌| 79/83 [05:34<00:20,  5.18s/batch]  mental-rotation:  96%|█████████▋| 80/83 [05:38<00:14,  4.99s/batch]  mental-rotation:  98%|█████████▊| 81/83 [05:43<00:09,  4.94s/batch]  mental-rotation:  99%|█████████▉| 82/83 [05:48<00:04,  4.93s/batch]  mental-rotation: 100%|██████████| 83/83 [05:53<00:00,  5.01s/batch]  mental-rotation: 100%|██████████| 83/83 [05:53<00:00,  4.26s/batch]
  gemini_pro/mental-rotation: 0.5060 (42/83)
  synthetic-vocab:   0%|          | 0/100 [00:00<?, ?batch/s]  synthetic-vocab:   1%|          | 1/100 [00:07<12:19,  7.47s/batch]  synthetic-vocab:   2%|▏         | 2/100 [00:14<12:08,  7.43s/batch]  synthetic-vocab:   3%|▎         | 3/100 [00:20<10:48,  6.69s/batch]  synthetic-vocab:   4%|▍         | 4/100 [00:27<10:47,  6.75s/batch]  synthetic-vocab:   5%|▌         | 5/100 [00:34<10:43,  6.77s/batch]  synthetic-vocab:   6%|▌         | 6/100 [00:41<10:58,  7.01s/batch]  synthetic-vocab:   7%|▋         | 7/100 [00:48<10:47,  6.96s/batch]  synthetic-vocab:   8%|▊         | 8/100 [00:55<10:35,  6.91s/batch]  synthetic-vocab:   9%|▉         | 9/100 [01:01<10:14,  6.75s/batch]  synthetic-vocab:  10%|█         | 10/100 [01:08<10:08,  6.77s/batch]  synthetic-vocab:  11%|█         | 11/100 [01:15<10:15,  6.91s/batch]  synthetic-vocab:  12%|█▏        | 12/100 [01:22<09:51,  6.73s/batch]  synthetic-vocab:  13%|█▎        | 13/100 [01:29<10:02,  6.92s/batch]  synthetic-vocab:  14%|█▍        | 14/100 [01:35<09:32,  6.66s/batch]  synthetic-vocab:  15%|█▌        | 15/100 [01:43<09:53,  6.99s/batch]  synthetic-vocab:  16%|█▌        | 16/100 [01:51<10:10,  7.26s/batch]  synthetic-vocab:  17%|█▋        | 17/100 [01:57<09:44,  7.05s/batch]  synthetic-vocab:  18%|█▊        | 18/100 [02:04<09:25,  6.89s/batch]  synthetic-vocab:  19%|█▉        | 19/100 [02:11<09:21,  6.94s/batch]  synthetic-vocab:  20%|██        | 20/100 [02:19<09:35,  7.20s/batch]  synthetic-vocab:  21%|██        | 21/100 [02:25<09:13,  7.01s/batch]  synthetic-vocab:  22%|██▏       | 22/100 [02:32<08:56,  6.87s/batch]  synthetic-vocab:  23%|██▎       | 23/100 [02:38<08:30,  6.63s/batch]  synthetic-vocab:  24%|██▍       | 24/100 [02:45<08:30,  6.71s/batch]  synthetic-vocab:  25%|██▌       | 25/100 [02:54<09:14,  7.40s/batch]  synthetic-vocab:  26%|██▌       | 26/100 [03:01<09:00,  7.30s/batch]  synthetic-vocab:  27%|██▋       | 27/100 [03:08<08:41,  7.14s/batch]  synthetic-vocab:  28%|██▊       | 28/100 [03:14<08:26,  7.04s/batch]  synthetic-vocab:  29%|██▉       | 29/100 [03:21<08:11,  6.93s/batch]  synthetic-vocab:  30%|███       | 30/100 [03:30<08:37,  7.39s/batch]  synthetic-vocab:  31%|███       | 31/100 [03:36<08:02,  6.99s/batch]  synthetic-vocab:  32%|███▏      | 32/100 [03:43<08:06,  7.16s/batch]  synthetic-vocab:  33%|███▎      | 33/100 [03:50<07:55,  7.10s/batch]  synthetic-vocab:  34%|███▍      | 34/100 [03:58<08:06,  7.37s/batch]  synthetic-vocab:  35%|███▌      | 35/100 [04:06<08:01,  7.40s/batch]  synthetic-vocab:  36%|███▌      | 36/100 [04:13<07:57,  7.46s/batch]  synthetic-vocab:  37%|███▋      | 37/100 [04:21<07:47,  7.42s/batch]  synthetic-vocab:  38%|███▊      | 38/100 [04:28<07:32,  7.30s/batch]  synthetic-vocab:  39%|███▉      | 39/100 [04:35<07:33,  7.43s/batch]  synthetic-vocab:  40%|████      | 40/100 [04:42<07:10,  7.18s/batch]  synthetic-vocab:  41%|████      | 41/100 [04:49<06:53,  7.01s/batch]  synthetic-vocab:  42%|████▏     | 42/100 [04:55<06:45,  7.00s/batch]  synthetic-vocab:  43%|████▎     | 43/100 [05:03<06:44,  7.10s/batch]  synthetic-vocab:  44%|████▍     | 44/100 [05:10<06:35,  7.07s/batch]  synthetic-vocab:  45%|████▌     | 45/100 [05:16<06:11,  6.75s/batch]  synthetic-vocab:  46%|████▌     | 46/100 [05:22<05:55,  6.59s/batch]  synthetic-vocab:  47%|████▋     | 47/100 [05:29<05:50,  6.62s/batch]  synthetic-vocab:  48%|████▊     | 48/100 [05:37<06:14,  7.21s/batch]  synthetic-vocab:  49%|████▉     | 49/100 [05:44<06:00,  7.06s/batch]  synthetic-vocab:  50%|█████     | 50/100 [05:50<05:43,  6.88s/batch]  synthetic-vocab:  51%|█████     | 51/100 [05:57<05:31,  6.76s/batch]  synthetic-vocab:  52%|█████▏    | 52/100 [06:04<05:29,  6.87s/batch]  synthetic-vocab:  53%|█████▎    | 53/100 [06:12<05:33,  7.09s/batch]  synthetic-vocab:  54%|█████▍    | 54/100 [06:18<05:14,  6.85s/batch]  synthetic-vocab:  55%|█████▌    | 55/100 [06:25<05:06,  6.82s/batch]  synthetic-vocab:  56%|█████▌    | 56/100 [06:32<05:09,  7.04s/batch]  synthetic-vocab:  57%|█████▋    | 57/100 [06:39<05:00,  6.99s/batch]  synthetic-vocab:  58%|█████▊    | 58/100 [06:47<05:04,  7.25s/batch]  synthetic-vocab:  59%|█████▉    | 59/100 [06:54<04:49,  7.05s/batch]  synthetic-vocab:  60%|██████    | 60/100 [07:00<04:30,  6.76s/batch]  synthetic-vocab:  61%|██████    | 61/100 [07:07<04:32,  6.98s/batch]  synthetic-vocab:  62%|██████▏   | 62/100 [07:14<04:17,  6.79s/batch]  synthetic-vocab:  63%|██████▎   | 63/100 [07:21<04:16,  6.94s/batch]  synthetic-vocab:  64%|██████▍   | 64/100 [07:28<04:17,  7.16s/batch]  synthetic-vocab:  65%|██████▌   | 65/100 [07:35<04:08,  7.11s/batch]  synthetic-vocab:  66%|██████▌   | 66/100 [07:42<03:52,  6.85s/batch]  synthetic-vocab:  67%|██████▋   | 67/100 [07:49<03:51,  7.03s/batch]  synthetic-vocab:  68%|██████▊   | 68/100 [07:56<03:40,  6.91s/batch]  synthetic-vocab:  69%|██████▉   | 69/100 [08:03<03:38,  7.03s/batch]  synthetic-vocab:  70%|███████   | 70/100 [08:10<03:26,  6.90s/batch]  synthetic-vocab:  71%|███████   | 71/100 [08:16<03:16,  6.78s/batch]  synthetic-vocab:  72%|███████▏  | 72/100 [08:23<03:14,  6.94s/batch]  synthetic-vocab:  73%|███████▎  | 73/100 [08:30<03:00,  6.70s/batch]  synthetic-vocab:  74%|███████▍  | 74/100 [08:37<02:57,  6.83s/batch]  synthetic-vocab:  75%|███████▌  | 75/100 [08:43<02:42,  6.50s/batch]  synthetic-vocab:  76%|███████▌  | 76/100 [08:49<02:35,  6.47s/batch]  synthetic-vocab:  77%|███████▋  | 77/100 [08:56<02:32,  6.63s/batch]  synthetic-vocab:  78%|███████▊  | 78/100 [09:03<02:26,  6.66s/batch]  synthetic-vocab:  79%|███████▉  | 79/100 [09:09<02:17,  6.53s/batch]  synthetic-vocab:  80%|████████  | 80/100 [09:16<02:14,  6.74s/batch]  synthetic-vocab:  81%|████████  | 81/100 [09:23<02:10,  6.89s/batch]  synthetic-vocab:  82%|████████▏ | 82/100 [09:30<02:03,  6.87s/batch]  synthetic-vocab:  83%|████████▎ | 83/100 [09:37<01:57,  6.88s/batch]  synthetic-vocab:  84%|████████▍ | 84/100 [09:44<01:50,  6.88s/batch]  synthetic-vocab:  85%|████████▌ | 85/100 [09:50<01:41,  6.76s/batch]  synthetic-vocab:  86%|████████▌ | 86/100 [09:58<01:37,  6.96s/batch]  synthetic-vocab:  87%|████████▋ | 87/100 [10:07<01:38,  7.57s/batch]  synthetic-vocab:  88%|████████▊ | 88/100 [10:13<01:27,  7.27s/batch]  synthetic-vocab:  89%|████████▉ | 89/100 [10:20<01:16,  6.94s/batch]  synthetic-vocab:  90%|█████████ | 90/100 [10:26<01:07,  6.73s/batch]  synthetic-vocab:  91%|█████████ | 91/100 [10:34<01:05,  7.26s/batch]  synthetic-vocab:  92%|█████████▏| 92/100 [10:40<00:55,  6.89s/batch]  synthetic-vocab:  93%|█████████▎| 93/100 [10:47<00:47,  6.75s/batch]  synthetic-vocab:  94%|█████████▍| 94/100 [10:54<00:41,  6.97s/batch]  synthetic-vocab:  95%|█████████▌| 95/100 [11:01<00:34,  6.93s/batch]  synthetic-vocab:  96%|█████████▌| 96/100 [11:10<00:29,  7.44s/batch]  synthetic-vocab:  97%|█████████▋| 97/100 [11:35<00:38, 12.68s/batch]  synthetic-vocab:  98%|█████████▊| 98/100 [11:42<00:22, 11.05s/batch]  synthetic-vocab:  99%|█████████▉| 99/100 [11:48<00:09,  9.58s/batch]  synthetic-vocab: 100%|██████████| 100/100 [11:54<00:00,  8.49s/batch]  synthetic-vocab: 100%|██████████| 100/100 [11:54<00:00,  7.14s/batch]
  gemini_pro/synthetic-vocab: 0.9900 (99/100)
  theory-of-mind: loaded human proportions for 95 items
  theory-of-mind:   0%|          | 0/37 [00:00<?, ?batch/s]  theory-of-mind:   3%|▎         | 1/37 [00:04<02:34,  4.30s/batch]  theory-of-mind:   5%|▌         | 2/37 [00:08<02:24,  4.13s/batch]  theory-of-mind:   8%|▊         | 3/37 [00:12<02:22,  4.19s/batch]  theory-of-mind:  11%|█         | 4/37 [00:18<02:40,  4.85s/batch]  theory-of-mind:  14%|█▎        | 5/37 [00:22<02:21,  4.43s/batch]  theory-of-mind:  16%|█▌        | 6/37 [00:26<02:12,  4.27s/batch]  theory-of-mind:  19%|█▉        | 7/37 [00:30<02:06,  4.21s/batch]  theory-of-mind:  22%|██▏       | 8/37 [00:34<01:59,  4.11s/batch]  theory-of-mind:  24%|██▍       | 9/37 [00:37<01:51,  3.99s/batch]  theory-of-mind:  27%|██▋       | 10/37 [00:43<01:59,  4.41s/batch]  theory-of-mind:  30%|██▉       | 11/37 [00:48<01:58,  4.57s/batch]  theory-of-mind:  32%|███▏      | 12/37 [00:52<01:54,  4.59s/batch]  theory-of-mind:  35%|███▌      | 13/37 [00:56<01:47,  4.46s/batch]  theory-of-mind:  38%|███▊      | 14/37 [01:02<01:48,  4.73s/batch]  theory-of-mind:  41%|████      | 15/37 [01:06<01:41,  4.63s/batch]  theory-of-mind:  43%|████▎     | 16/37 [01:10<01:33,  4.45s/batch]  theory-of-mind:  46%|████▌     | 17/37 [01:14<01:25,  4.29s/batch]  theory-of-mind:  49%|████▊     | 18/37 [01:18<01:19,  4.18s/batch]  theory-of-mind:  51%|█████▏    | 19/37 [01:24<01:23,  4.65s/batch]  theory-of-mind:  54%|█████▍    | 20/37 [01:28<01:15,  4.44s/batch]  theory-of-mind:  57%|█████▋    | 21/37 [01:32<01:08,  4.27s/batch]  theory-of-mind:  59%|█████▉    | 22/37 [01:36<01:05,  4.35s/batch]  theory-of-mind:  62%|██████▏   | 23/37 [01:41<01:01,  4.41s/batch]  theory-of-mind:  65%|██████▍   | 24/37 [01:45<00:58,  4.47s/batch]  theory-of-mind:  68%|██████▊   | 25/37 [01:50<00:55,  4.61s/batch]  theory-of-mind:  70%|███████   | 26/37 [01:55<00:51,  4.70s/batch]  theory-of-mind:  73%|███████▎  | 27/37 [02:00<00:49,  4.90s/batch]  theory-of-mind:  76%|███████▌  | 28/37 [02:05<00:42,  4.71s/batch]  theory-of-mind:  78%|███████▊  | 29/37 [02:08<00:35,  4.41s/batch]  theory-of-mind:  81%|████████  | 30/37 [02:13<00:31,  4.51s/batch]  theory-of-mind:  84%|████████▍ | 31/37 [02:18<00:27,  4.53s/batch]  theory-of-mind:  86%|████████▋ | 32/37 [02:23<00:23,  4.65s/batch]  theory-of-mind:  89%|████████▉ | 33/37 [02:27<00:18,  4.66s/batch]  theory-of-mind:  92%|█████████▏| 34/37 [02:34<00:15,  5.22s/batch]  theory-of-mind:  95%|█████████▍| 35/37 [02:39<00:10,  5.18s/batch]  theory-of-mind:  97%|█████████▋| 36/37 [02:45<00:05,  5.37s/batch]  theory-of-mind: 100%|██████████| 37/37 [02:50<00:00,  5.36s/batch]  theory-of-mind: 100%|██████████| 37/37 [02:50<00:00,  4.61s/batch]
  gemini_pro/theory-of-mind: wrote /home/david/levante/levante-bench/results_api_retry_metadata/v1/gemini_pro/theory-of-mind-by-type.csv
  gemini_pro/theory-of-mind: 0.7568 (28/37)
  trog: loaded human proportions for 99 items
  trog:   0%|          | 0/99 [00:00<?, ?batch/s]  trog:   1%|          | 1/99 [00:04<06:56,  4.25s/batch]  trog:   2%|▏         | 2/99 [00:08<07:10,  4.44s/batch]  trog:   3%|▎         | 3/99 [00:13<07:21,  4.60s/batch]  trog:   4%|▍         | 4/99 [00:18<07:10,  4.53s/batch]  trog:   5%|▌         | 5/99 [00:22<07:08,  4.56s/batch]  trog:   6%|▌         | 6/99 [00:27<07:02,  4.54s/batch]  trog:   7%|▋         | 7/99 [00:31<06:47,  4.43s/batch]  trog:   8%|▊         | 8/99 [00:35<06:38,  4.38s/batch]  trog:   9%|▉         | 9/99 [00:40<06:47,  4.53s/batch]  trog:  10%|█         | 10/99 [00:44<06:41,  4.51s/batch]  trog:  11%|█         | 11/99 [00:50<07:10,  4.90s/batch]  trog:  12%|█▏        | 12/99 [00:54<06:43,  4.64s/batch]  trog:  13%|█▎        | 13/99 [00:59<06:29,  4.53s/batch]  trog:  14%|█▍        | 14/99 [01:03<06:12,  4.38s/batch]  trog:  15%|█▌        | 15/99 [01:07<06:10,  4.41s/batch]  trog:  16%|█▌        | 16/99 [01:11<05:59,  4.33s/batch]  trog:  17%|█▋        | 17/99 [01:19<07:08,  5.22s/batch]  trog:  18%|█▊        | 18/99 [01:23<06:41,  4.96s/batch]  trog:  19%|█▉        | 19/99 [01:27<06:14,  4.68s/batch]  trog:  20%|██        | 20/99 [01:31<06:05,  4.62s/batch]  trog:  21%|██        | 21/99 [01:36<06:00,  4.62s/batch]  trog:  22%|██▏       | 22/99 [01:40<05:44,  4.48s/batch]  trog:  23%|██▎       | 23/99 [01:45<05:38,  4.45s/batch]  trog:  24%|██▍       | 24/99 [01:49<05:32,  4.44s/batch]  trog:  25%|██▌       | 25/99 [01:54<05:36,  4.55s/batch]  trog:  26%|██▋       | 26/99 [01:58<05:25,  4.46s/batch]  trog:  27%|██▋       | 27/99 [02:02<05:07,  4.27s/batch]  trog:  28%|██▊       | 28/99 [02:07<05:30,  4.66s/batch]  trog:  29%|██▉       | 29/99 [02:13<05:36,  4.81s/batch]  trog:  30%|███       | 30/99 [02:17<05:14,  4.56s/batch]  trog:  31%|███▏      | 31/99 [02:21<05:06,  4.51s/batch]  trog:  32%|███▏      | 32/99 [02:26<05:10,  4.63s/batch]  trog:  33%|███▎      | 33/99 [02:30<04:59,  4.53s/batch]  trog:  34%|███▍      | 34/99 [02:36<05:10,  4.78s/batch]  trog:  35%|███▌      | 35/99 [02:40<05:08,  4.81s/batch]  trog:  36%|███▋      | 36/99 [02:45<05:01,  4.79s/batch]  trog:  37%|███▋      | 37/99 [02:49<04:37,  4.47s/batch]  trog:  38%|███▊      | 38/99 [02:53<04:25,  4.35s/batch]  trog:  39%|███▉      | 39/99 [02:58<04:32,  4.54s/batch]  trog:  40%|████      | 40/99 [03:03<04:46,  4.85s/batch]  trog:  41%|████▏     | 41/99 [03:08<04:33,  4.72s/batch]  trog:  42%|████▏     | 42/99 [03:13<04:27,  4.70s/batch]  trog:  43%|████▎     | 43/99 [03:18<04:27,  4.78s/batch]  trog:  44%|████▍     | 44/99 [03:22<04:17,  4.68s/batch]  trog:  45%|████▌     | 45/99 [03:26<04:09,  4.63s/batch]  trog:  46%|████▋     | 46/99 [03:32<04:22,  4.95s/batch]  trog:  47%|████▋     | 47/99 [03:37<04:07,  4.77s/batch]  trog:  48%|████▊     | 48/99 [03:41<03:57,  4.65s/batch]  trog:  49%|████▉     | 49/99 [03:45<03:51,  4.64s/batch]  trog:  51%|█████     | 50/99 [03:50<03:41,  4.52s/batch]  trog:  52%|█████▏    | 51/99 [03:54<03:33,  4.45s/batch]  trog:  53%|█████▎    | 52/99 [03:58<03:23,  4.32s/batch]  trog:  54%|█████▎    | 53/99 [04:02<03:20,  4.36s/batch]  trog:  55%|█████▍    | 54/99 [04:08<03:25,  4.56s/batch]  trog:  56%|█████▌    | 55/99 [04:12<03:19,  4.53s/batch]  trog:  57%|█████▋    | 56/99 [04:16<03:12,  4.47s/batch]  trog:  58%|█████▊    | 57/99 [04:21<03:05,  4.42s/batch]  trog:  59%|█████▊    | 58/99 [04:24<02:54,  4.26s/batch]  trog:  60%|█████▉    | 59/99 [04:29<02:48,  4.22s/batch]  trog:  61%|██████    | 60/99 [04:32<02:40,  4.11s/batch]  trog:  62%|██████▏   | 61/99 [04:37<02:43,  4.31s/batch]  trog:  63%|██████▎   | 62/99 [04:41<02:37,  4.25s/batch]  trog:  64%|██████▎   | 63/99 [04:46<02:34,  4.30s/batch]  trog:  65%|██████▍   | 64/99 [04:49<02:21,  4.06s/batch]  trog:  66%|██████▌   | 65/99 [04:54<02:21,  4.17s/batch]  trog:  67%|██████▋   | 66/99 [04:58<02:22,  4.32s/batch]  trog:  68%|██████▊   | 67/99 [05:02<02:13,  4.16s/batch]  trog:  69%|██████▊   | 68/99 [05:06<02:07,  4.11s/batch]  trog:  70%|██████▉   | 69/99 [05:11<02:07,  4.26s/batch]  trog:  71%|███████   | 70/99 [05:15<01:59,  4.13s/batch]  trog:  72%|███████▏  | 71/99 [05:20<02:03,  4.42s/batch]  trog:  73%|███████▎  | 72/99 [05:24<01:56,  4.30s/batch]  trog:  74%|███████▎  | 73/99 [05:28<01:52,  4.32s/batch]  trog:  75%|███████▍  | 74/99 [05:33<01:51,  4.46s/batch]  trog:  76%|███████▌  | 75/99 [05:37<01:46,  4.43s/batch]  trog:  77%|███████▋  | 76/99 [05:44<01:57,  5.09s/batch]  trog:  78%|███████▊  | 77/99 [05:48<01:45,  4.79s/batch]  trog:  79%|███████▉  | 78/99 [05:52<01:35,  4.56s/batch]  trog:  80%|███████▉  | 79/99 [05:57<01:34,  4.74s/batch]  trog:  81%|████████  | 80/99 [06:01<01:27,  4.62s/batch]  trog:  82%|████████▏ | 81/99 [06:08<01:31,  5.08s/batch]  trog:  83%|████████▎ | 82/99 [06:12<01:21,  4.81s/batch]  trog:  84%|████████▍ | 83/99 [06:17<01:19,  5.00s/batch]  trog:  85%|████████▍ | 84/99 [06:22<01:12,  4.86s/batch]  trog:  86%|████████▌ | 85/99 [06:27<01:09,  5.00s/batch]  trog:  87%|████████▋ | 86/99 [06:31<01:02,  4.81s/batch]  trog:  88%|████████▊ | 87/99 [06:36<00:55,  4.66s/batch]  trog:  89%|████████▉ | 88/99 [06:40<00:49,  4.50s/batch]  trog:  90%|████████▉ | 89/99 [06:45<00:46,  4.70s/batch]  trog:  91%|█████████ | 90/99 [06:51<00:46,  5.18s/batch]  trog:  92%|█████████▏| 91/99 [06:55<00:38,  4.85s/batch]  trog:  93%|█████████▎| 92/99 [07:01<00:34,  4.98s/batch]  trog:  94%|█████████▍| 93/99 [07:05<00:28,  4.74s/batch]  trog:  95%|█████████▍| 94/99 [07:09<00:22,  4.54s/batch]  trog:  96%|█████████▌| 95/99 [07:14<00:18,  4.60s/batch]  trog:  97%|█████████▋| 96/99 [07:17<00:13,  4.35s/batch]  trog:  98%|█████████▊| 97/99 [07:21<00:08,  4.25s/batch]  trog:  99%|█████████▉| 98/99 [07:27<00:04,  4.53s/batch]  trog: 100%|██████████| 99/99 [07:32<00:00,  4.89s/batch]  trog: 100%|██████████| 99/99 [07:32<00:00,  4.57s/batch]
  gemini_pro/trog: 0.9394 (93/99)
  vocab: loaded human proportions for 164 items
  vocab:   0%|          | 0/170 [00:00<?, ?batch/s]  vocab:   1%|          | 1/170 [00:04<11:41,  4.15s/batch]  vocab:   1%|          | 2/170 [00:08<11:22,  4.06s/batch]  vocab:   2%|▏         | 3/170 [00:12<11:09,  4.01s/batch]  vocab:   2%|▏         | 4/170 [00:16<11:36,  4.20s/batch]  vocab:   3%|▎         | 5/170 [00:20<11:19,  4.12s/batch]  vocab:   4%|▎         | 6/170 [00:25<12:23,  4.53s/batch]  vocab:   4%|▍         | 7/170 [00:29<11:53,  4.38s/batch]  vocab:   5%|▍         | 8/170 [00:34<11:57,  4.43s/batch]  vocab:   5%|▌         | 9/170 [00:38<11:30,  4.29s/batch]  vocab:   6%|▌         | 10/170 [00:42<11:22,  4.27s/batch]  vocab:   6%|▋         | 11/170 [00:47<11:23,  4.30s/batch]  vocab:   7%|▋         | 12/170 [00:52<12:10,  4.62s/batch]  vocab:   8%|▊         | 13/170 [00:58<13:35,  5.19s/batch]  vocab:   8%|▊         | 14/170 [01:03<13:00,  5.01s/batch]  vocab:   9%|▉         | 15/170 [01:08<13:04,  5.06s/batch]  vocab:   9%|▉         | 16/170 [01:12<12:23,  4.83s/batch]  vocab:  10%|█         | 17/170 [01:17<12:25,  4.88s/batch]  vocab:  11%|█         | 18/170 [01:22<12:07,  4.78s/batch]  vocab:  11%|█         | 19/170 [01:27<12:15,  4.87s/batch]  vocab:  12%|█▏        | 20/170 [01:33<13:13,  5.29s/batch]  vocab:  12%|█▏        | 21/170 [01:37<12:09,  4.90s/batch]  vocab:  13%|█▎        | 22/170 [01:43<12:32,  5.09s/batch]  vocab:  14%|█▎        | 23/170 [01:47<11:49,  4.83s/batch]  vocab:  14%|█▍        | 24/170 [01:52<11:35,  4.76s/batch]  vocab:  15%|█▍        | 25/170 [01:56<11:01,  4.56s/batch]  vocab:  15%|█▌        | 26/170 [02:00<10:43,  4.47s/batch]  vocab:  16%|█▌        | 27/170 [02:05<10:48,  4.54s/batch]  vocab:  16%|█▋        | 28/170 [02:09<10:22,  4.39s/batch]  vocab:  17%|█▋        | 29/170 [02:13<10:31,  4.48s/batch]  vocab:  18%|█▊        | 30/170 [02:18<10:20,  4.43s/batch]  vocab:  18%|█▊        | 31/170 [02:23<10:45,  4.64s/batch]  vocab:  19%|█▉        | 32/170 [02:27<10:11,  4.43s/batch]  vocab:  19%|█▉        | 33/170 [02:32<10:19,  4.52s/batch]  vocab:  20%|██        | 34/170 [02:38<11:19,  5.00s/batch]  vocab:  21%|██        | 35/170 [02:43<11:30,  5.11s/batch]  vocab:  21%|██        | 36/170 [02:48<11:22,  5.10s/batch]  vocab:  22%|██▏       | 37/170 [02:53<10:51,  4.90s/batch]  vocab:  22%|██▏       | 38/170 [02:57<10:20,  4.70s/batch]  vocab:  23%|██▎       | 39/170 [03:01<09:49,  4.50s/batch]  vocab:  24%|██▎       | 40/170 [03:05<09:40,  4.47s/batch]  vocab:  24%|██▍       | 41/170 [03:11<10:38,  4.95s/batch]  vocab:  25%|██▍       | 42/170 [03:16<10:32,  4.94s/batch]  vocab:  25%|██▌       | 43/170 [03:21<10:33,  4.99s/batch]  vocab:  26%|██▌       | 44/170 [03:28<11:24,  5.43s/batch]  vocab:  26%|██▋       | 45/170 [03:33<11:13,  5.39s/batch]  vocab:  27%|██▋       | 46/170 [03:38<10:45,  5.20s/batch]  vocab:  28%|██▊       | 47/170 [03:43<10:46,  5.26s/batch]  vocab:  28%|██▊       | 48/170 [03:48<10:09,  4.99s/batch]  vocab:  29%|██▉       | 49/170 [03:52<09:49,  4.88s/batch]  vocab:  29%|██▉       | 50/170 [03:56<09:17,  4.65s/batch]  vocab:  30%|███       | 51/170 [04:01<09:17,  4.68s/batch]  vocab:  31%|███       | 52/170 [04:06<09:15,  4.70s/batch]  vocab:  31%|███       | 53/170 [04:11<09:23,  4.82s/batch]  vocab:  32%|███▏      | 54/170 [04:17<09:45,  5.04s/batch]  vocab:  32%|███▏      | 55/170 [04:23<10:18,  5.38s/batch]  vocab:  33%|███▎      | 56/170 [04:27<09:43,  5.12s/batch]  vocab:  34%|███▎      | 57/170 [04:32<09:25,  5.01s/batch]  vocab:  34%|███▍      | 58/170 [04:37<09:15,  4.96s/batch]  vocab:  35%|███▍      | 59/170 [04:41<08:49,  4.77s/batch]  vocab:  35%|███▌      | 60/170 [04:45<08:17,  4.52s/batch]  vocab:  36%|███▌      | 61/170 [04:50<08:31,  4.69s/batch]  vocab:  36%|███▋      | 62/170 [04:54<08:06,  4.51s/batch]  vocab:  37%|███▋      | 63/170 [04:58<07:51,  4.40s/batch]  vocab:  38%|███▊      | 64/170 [05:03<07:41,  4.36s/batch]  vocab:  38%|███▊      | 65/170 [05:08<07:55,  4.53s/batch]  vocab:  39%|███▉      | 66/170 [05:13<08:04,  4.66s/batch]  vocab:  39%|███▉      | 67/170 [05:17<07:47,  4.54s/batch]  vocab:  40%|████      | 68/170 [05:22<07:50,  4.61s/batch]  vocab:  41%|████      | 69/170 [05:27<08:07,  4.83s/batch]  vocab:  41%|████      | 70/170 [05:31<07:54,  4.74s/batch]  vocab:  42%|████▏     | 71/170 [05:36<07:36,  4.61s/batch]  vocab:  42%|████▏     | 72/170 [05:50<12:05,  7.40s/batch]  vocab:  43%|████▎     | 73/170 [05:55<10:47,  6.68s/batch]  vocab:  44%|████▎     | 74/170 [06:00<09:58,  6.23s/batch]  vocab:  44%|████▍     | 75/170 [06:05<09:09,  5.79s/batch]  vocab:  45%|████▍     | 76/170 [06:10<08:41,  5.55s/batch]  vocab:  45%|████▌     | 77/170 [06:14<08:07,  5.25s/batch]  vocab:  46%|████▌     | 78/170 [06:18<07:30,  4.89s/batch]  vocab:  46%|████▋     | 79/170 [06:22<06:52,  4.53s/batch]  vocab:  47%|████▋     | 80/170 [06:29<07:48,  5.20s/batch]  vocab:  48%|████▊     | 81/170 [06:33<07:11,  4.84s/batch]  vocab:  48%|████▊     | 82/170 [06:38<07:12,  4.92s/batch]  vocab:  49%|████▉     | 83/170 [06:42<06:54,  4.77s/batch]  vocab:  49%|████▉     | 84/170 [06:48<07:15,  5.07s/batch]  vocab:  50%|█████     | 85/170 [06:53<07:19,  5.17s/batch]  vocab:  51%|█████     | 86/170 [06:58<06:57,  4.96s/batch]  vocab:  51%|█████     | 87/170 [07:05<07:40,  5.55s/batch]  vocab:  52%|█████▏    | 88/170 [07:10<07:21,  5.39s/batch]  vocab:  52%|█████▏    | 89/170 [07:14<06:57,  5.15s/batch]  vocab:  53%|█████▎    | 90/170 [07:19<06:29,  4.86s/batch]  vocab:  54%|█████▎    | 91/170 [07:23<06:09,  4.67s/batch]  vocab:  54%|█████▍    | 92/170 [07:27<06:03,  4.66s/batch]  vocab:  55%|█████▍    | 93/170 [07:32<05:53,  4.60s/batch]  vocab:  55%|█████▌    | 94/170 [07:38<06:14,  4.92s/batch]  vocab:  56%|█████▌    | 95/170 [07:42<05:53,  4.72s/batch]  vocab:  56%|█████▋    | 96/170 [07:47<05:56,  4.82s/batch]  vocab:  57%|█████▋    | 97/170 [07:51<05:45,  4.73s/batch]  vocab:  58%|█████▊    | 98/170 [07:56<05:44,  4.78s/batch]  vocab:  58%|█████▊    | 99/170 [08:02<05:58,  5.05s/batch]  vocab:  59%|█████▉    | 100/170 [08:08<06:24,  5.49s/batch]  vocab:  59%|█████▉    | 101/170 [08:13<06:08,  5.34s/batch]  vocab:  60%|██████    | 102/170 [08:18<05:57,  5.26s/batch]  vocab:  61%|██████    | 103/170 [08:23<05:40,  5.09s/batch]  vocab:  61%|██████    | 104/170 [08:28<05:29,  4.99s/batch]  vocab:  62%|██████▏   | 105/170 [08:33<05:23,  4.97s/batch]  vocab:  62%|██████▏   | 106/170 [08:37<05:04,  4.76s/batch]  vocab:  63%|██████▎   | 107/170 [08:43<05:11,  4.95s/batch]  vocab:  64%|██████▎   | 108/170 [08:48<05:10,  5.01s/batch]  vocab:  64%|██████▍   | 109/170 [08:53<05:15,  5.16s/batch]  vocab:  65%|██████▍   | 110/170 [08:58<05:07,  5.12s/batch]  vocab:  65%|██████▌   | 111/170 [09:02<04:42,  4.80s/batch]  vocab:  66%|██████▌   | 112/170 [09:07<04:31,  4.67s/batch]  vocab:  66%|██████▋   | 113/170 [09:12<04:37,  4.87s/batch]  vocab:  67%|██████▋   | 114/170 [09:17<04:30,  4.83s/batch]  vocab:  68%|██████▊   | 115/170 [09:22<04:28,  4.88s/batch]  vocab:  68%|██████▊   | 116/170 [09:27<04:33,  5.07s/batch]  vocab:  69%|██████▉   | 117/170 [09:31<04:14,  4.80s/batch]  vocab:  69%|██████▉   | 118/170 [09:36<04:03,  4.69s/batch]  vocab:  70%|███████   | 119/170 [09:40<03:56,  4.65s/batch]  vocab:  71%|███████   | 120/170 [09:46<04:06,  4.93s/batch]  vocab:  71%|███████   | 121/170 [09:51<03:59,  4.89s/batch]  vocab:  72%|███████▏  | 122/170 [09:55<03:45,  4.70s/batch]  vocab:  72%|███████▏  | 123/170 [09:59<03:32,  4.53s/batch]  vocab:  73%|███████▎  | 124/170 [10:04<03:25,  4.47s/batch]  vocab:  74%|███████▎  | 125/170 [10:09<03:29,  4.65s/batch]  vocab:  74%|███████▍  | 126/170 [10:13<03:23,  4.61s/batch]  vocab:  75%|███████▍  | 127/170 [10:19<03:34,  4.99s/batch]  vocab:  75%|███████▌  | 128/170 [10:24<03:29,  4.99s/batch]  vocab:  76%|███████▌  | 129/170 [10:28<03:17,  4.82s/batch]  vocab:  76%|███████▋  | 130/170 [10:33<03:06,  4.66s/batch]  vocab:  77%|███████▋  | 131/170 [10:38<03:14,  4.98s/batch]  vocab:  78%|███████▊  | 132/170 [10:43<03:07,  4.94s/batch]  vocab:  78%|███████▊  | 133/170 [10:48<02:59,  4.85s/batch]  vocab:  79%|███████▉  | 134/170 [10:53<03:00,  5.02s/batch]  vocab:  79%|███████▉  | 135/170 [10:58<02:50,  4.87s/batch]  vocab:  80%|████████  | 136/170 [11:03<02:53,  5.11s/batch]  vocab:  81%|████████  | 137/170 [11:08<02:46,  5.06s/batch]  vocab:  81%|████████  | 138/170 [11:13<02:37,  4.94s/batch]  vocab:  82%|████████▏ | 139/170 [11:18<02:28,  4.80s/batch]  vocab:  82%|████████▏ | 140/170 [11:23<02:26,  4.89s/batch]  vocab:  83%|████████▎ | 141/170 [11:29<02:30,  5.19s/batch]  vocab:  84%|████████▎ | 142/170 [11:34<02:23,  5.14s/batch]  vocab:  84%|████████▍ | 143/170 [11:38<02:14,  4.98s/batch]  vocab:  85%|████████▍ | 144/170 [11:43<02:07,  4.92s/batch]  vocab:  85%|████████▌ | 145/170 [11:48<02:03,  4.93s/batch]  vocab:  86%|████████▌ | 146/170 [11:52<01:54,  4.76s/batch]  vocab:  86%|████████▋ | 147/170 [11:57<01:51,  4.85s/batch]  vocab:  87%|████████▋ | 148/170 [12:02<01:43,  4.70s/batch]  vocab:  88%|████████▊ | 149/170 [12:07<01:40,  4.78s/batch]  vocab:  88%|████████▊ | 150/170 [12:11<01:32,  4.61s/batch]  vocab:  89%|████████▉ | 151/170 [12:15<01:25,  4.49s/batch]  vocab:  89%|████████▉ | 152/170 [12:19<01:20,  4.46s/batch]  vocab:  90%|█████████ | 153/170 [12:33<02:02,  7.23s/batch]  vocab:  91%|█████████ | 154/170 [12:38<01:46,  6.63s/batch]  vocab:  91%|█████████ | 155/170 [12:43<01:30,  6.01s/batch]  vocab:  92%|█████████▏| 156/170 [12:47<01:17,  5.52s/batch]  vocab:  92%|█████████▏| 157/170 [12:53<01:12,  5.61s/batch]  vocab:  93%|█████████▎| 158/170 [12:58<01:03,  5.32s/batch]  vocab:  94%|█████████▎| 159/170 [13:02<00:55,  5.07s/batch]  vocab:  94%|█████████▍| 160/170 [13:09<00:54,  5.46s/batch]  vocab:  95%|█████████▍| 161/170 [13:14<00:48,  5.35s/batch]  vocab:  95%|█████████▌| 162/170 [13:19<00:41,  5.21s/batch]  vocab:  96%|█████████▌| 163/170 [13:23<00:35,  5.03s/batch]  vocab:  96%|█████████▋| 164/170 [13:29<00:30,  5.11s/batch]  vocab:  97%|█████████▋| 165/170 [13:34<00:25,  5.09s/batch]  vocab:  98%|█████████▊| 166/170 [13:39<00:21,  5.28s/batch]  vocab:  98%|█████████▊| 167/170 [13:44<00:14,  4.97s/batch]  vocab:  99%|█████████▉| 168/170 [13:48<00:09,  4.72s/batch]  vocab:  99%|█████████▉| 169/170 [13:52<00:04,  4.75s/batch]  vocab: 100%|██████████| 170/170 [13:57<00:00,  4.55s/batch]  vocab: 100%|██████████| 170/170 [13:57<00:00,  4.92s/batch]
  gemini_pro/vocab: 0.9941 (169/170)
  Summary: /home/david/levante/levante-bench/results_api_retry_metadata/v1/gemini_pro/summary.csv
Success: wrote 2 file(s)
  gpt53	/home/david/levante/levante-bench/results_api_retry_metadata/v1/gpt53/summary.csv
  gemini_pro	/home/david/levante/levante-bench/results_api_retry_metadata/v1/gemini_pro/summary.csv