Skip to content

Commit 38990b3

Browse files
pkmandkepkmandke
authored
Add Bits Per Weight (BPW) info for model cards (#17)
* add bits per weight info for model cards * re-order bpw column * add bpw with embedding included
1 parent bffc38f commit 38990b3

6 files changed

Lines changed: 38 additions & 34 deletions

File tree

‎models/gemma3/README.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -79,11 +79,11 @@ Defaults: 512 prompt tokens, 1024 generation tokens, 5 trials. Override with `-p
7979

8080
Perplexity score on the [`WikiText-2`](https://huggingface.co/datasets/EleutherAI/wikitext_document_level) dataset computed using the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/wikitext/README.md) with the Core AI PyTorch models.
8181

82-
| Model | Compression | Platform | Perplexity Score |
83-
| ----------- | ------------------------- | -------- | ---------------- |
84-
| Gemma 3 4B | none (`float16`) | macOS | 17.90 |
85-
| Gemma 3 4B | [4-bit quantized][p-4bit] | macOS | 19.28 |
86-
| Gemma 3 12B | none (`float16`) | macOS | 11.24 |
87-
| Gemma 3 12B | [4-bit quantized][p-4bit] | macOS | 11.75 |
82+
| Model | Compression | Bits Per Weight (BPW) | Platform | Perplexity Score |
83+
| ----------- | ------------------------- | --------------------- | -------- | ---------------- |
84+
| Gemma 3 4B | none (`float16`) | 16.00 | macOS | 17.90 |
85+
| Gemma 3 4B | [4-bit quantized][p-4bit] | 4.50 | macOS | 19.28 |
86+
| Gemma 3 12B | none (`float16`) | 16.00 | macOS | 11.24 |
87+
| Gemma 3 12B | [4-bit quantized][p-4bit] | 4.50 | macOS | 11.75 |
8888

8989
[p-4bit]: ../README.md#quantization-options

‎models/mistral/README.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -69,9 +69,9 @@ Defaults: 512 prompt tokens, 1024 generation tokens, 5 trials. Override with `-p
6969

7070
Perplexity score on the [`WikiText-2`](https://huggingface.co/datasets/EleutherAI/wikitext_document_level) dataset computed using the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/wikitext/README.md) with the Core AI PyTorch models.
7171

72-
| Model | Compression | Platform | Perplexity Score |
73-
| ------------------- | ------------------------------------------ | -------- | ---------------- |
74-
| Mistral 7B Instruct | none (`float16`) | macOS | 8.29 |
75-
| Mistral 7B Instruct | [4-bit quantized][p-4bit] | macOS | 8.41 |
72+
| Model | Compression | Bits Per Weight (BPW) | Platform | Perplexity Score |
73+
| ------------------- | ------------------------------------------ | --------------------- | -------- | ---------------- |
74+
| Mistral 7B Instruct | none (`float16`) | 16.00 | macOS | 8.29 |
75+
| Mistral 7B Instruct | [4-bit quantized][p-4bit] | 4.50 | macOS | 8.41 |
7676

7777
[p-4bit]: ../README.md#quantization-options

‎models/mixtral/README.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -69,9 +69,9 @@ Defaults: 512 prompt tokens, 1024 generation tokens, 5 trials. Override with `-p
6969

7070
Perplexity score on the [`WikiText-2`](https://huggingface.co/datasets/EleutherAI/wikitext_document_level) dataset computed using the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/wikitext/README.md) with the Core AI PyTorch models.
7171

72-
| Model | Compression | Platform | Perplexity Score |
73-
| ------------ | ---------------------------------------------------- | -------- | ---------------- |
74-
| Mixtral 8x7B | none (`float16`) | macOS | 5.72 |
75-
| Mixtral 8x7B | [4-bit quantized][p-4bit] | macOS | 6.19 |
72+
| Model | Compression | Bits Per Weight (BPW) | Platform | Perplexity Score |
73+
| ------------ | ---------------------------------------------------- | --------------------- | -------- | ---------------- |
74+
| Mixtral 8x7B | none (`float16`) | 16.00 | macOS | 5.72 |
75+
| Mixtral 8x7B | [4-bit quantized][p-4bit] | 4.50 | macOS | 6.19 |
7676

7777
[p-4bit]: ../README.md#quantization-options

‎models/qwen2/README.md‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -75,11 +75,13 @@ Defaults: 512 prompt tokens, 1024 generation tokens, 5 trials. Override with `-p
7575

7676
Perplexity score on the [`WikiText-2`](https://huggingface.co/datasets/EleutherAI/wikitext_document_level) dataset computed using the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/wikitext/README.md) with the Core AI PyTorch models.
7777

78-
| Model | Compression | Platform | Perplexity Score |
79-
| --------------------- | ----------------------------------------- | -------- | ---------------- |
80-
| Qwen2.5 1.5B Instruct | none (`float16`) | macOS | 12.21 |
81-
| Qwen2.5 1.5B Instruct | [4-bit quantized][p-4bit] | macOS | 14.79 |
82-
| Qwen2.5 1.5B Instruct | none (`float16`) | iOS | 12.21 |
83-
| Qwen2.5 1.5B Instruct | [4-bit palettized (group size 8)][p-4bit] | iOS | 14.64 |
78+
| Model | Compression | Bits Per Weight (BPW) | Platform | Perplexity Score |
79+
| --------------------- | ----------------------------------------- | --------------------- | -------- | ---------------- |
80+
| Qwen2.5 1.5B Instruct | none (`float16`) | 16.00 | macOS | 12.21 |
81+
| Qwen2.5 1.5B Instruct | [4-bit quantized][p-4bit] | 4.50 | macOS | 14.79 |
82+
| Qwen2.5 1.5B Instruct | none (`float16`) | 16.00 | iOS | 12.21 |
83+
| Qwen2.5 1.5B Instruct | [4-bit palettized (group size 8)][p-4bit] | 4.63\* | iOS | 14.64 |
84+
85+
\* BPW includes the Embedding which is quantized to INT8 per-tensor.
8486

8587
[p-4bit]: ../README.md#quantization-options

‎models/qwen3/README.md‎

Lines changed: 12 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -77,16 +77,18 @@ Defaults: 512 prompt tokens, 1024 generation tokens, 5 trials. Override with `-p
7777

7878
Perplexity score on the [`WikiText-2`](https://huggingface.co/datasets/EleutherAI/wikitext_document_level) dataset computed using the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/wikitext/README.md) with the Core AI PyTorch models.
7979

80-
| Model | Compression | Platform | Perplexity Score |
81-
| ---------- | ---------------------------------------------------- | -------- | ---------------- |
82-
| Qwen3 0.6B | none (`float16`) | iOS | 26.16 |
83-
| Qwen3 0.6B | [Mixed 4-bit/8-bit palettized][mixed-4bit-8bit-yaml] | iOS | 30.90 |
84-
| Qwen3 4B | none (`float16`) | macOS | 16.41 |
85-
| Qwen3 4B | [4-bit quantized][presets-info] | macOS | 18.33 |
86-
| Qwen3 4B | none (`float16`) | iOS | 16.41 |
87-
| Qwen3 4B | [Mixed 4-bit/8-bit palettized][qwen3-4b-mixed-yaml] | iOS | 18.80 |
88-
| Qwen3 8B | none (`float16`) | macOS | 12.19 |
89-
| Qwen3 8B | [4-bit quantized][presets-info] | macOS | 12.90 |
80+
| Model | Compression | Bits Per Weight (BPW) | Platform | Perplexity Score |
81+
| ---------- | ---------------------------------------------------- | --------------------- | -------- | ---------------- |
82+
| Qwen3 0.6B | none (`float16`) | 16.00 | iOS | 26.16 |
83+
| Qwen3 0.6B | [Mixed 4-bit/8-bit palettized][mixed-4bit-8bit-yaml] | 5.71\* | iOS | 30.90 |
84+
| Qwen3 4B | none (`float16`) | 16.00 | macOS | 16.41 |
85+
| Qwen3 4B | [4-bit quantized][presets-info] | 4.50 | macOS | 18.33 |
86+
| Qwen3 4B | none (`float16`) | 16.00 | iOS | 16.41 |
87+
| Qwen3 4B | [Mixed 4-bit/8-bit palettized][qwen3-4b-mixed-yaml] | 4.89\* | iOS | 18.80 |
88+
| Qwen3 8B | none (`float16`) | 16.00 | macOS | 12.19 |
89+
| Qwen3 8B | [4-bit quantized][presets-info] | 4.50 | macOS | 12.90 |
90+
91+
\* BPW includes the Embedding which is quantized to INT8 per-tensor.
9092

9193
[presets-info]: ../README.md#quantization-options
9294
[mixed-4bit-8bit-yaml]: qwen3_0_6b_mixed_4bit_8bit.yaml

‎models/qwen3_moe/README.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -51,9 +51,9 @@ Defaults: 512 prompt tokens, 1024 generation tokens, 5 trials. Override with `-p
5151

5252
Perplexity score on the [`WikiText-2`](https://huggingface.co/datasets/EleutherAI/wikitext_document_level) dataset computed using the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/wikitext/README.md) with the Core AI PyTorch models.
5353

54-
| Model | Compression | Platform | Perplexity Score |
55-
| ---------------------------- | ------------------------- | -------- | ---------------- |
56-
| Qwen3 Coder 30B-A3B Instruct | none (`float16`) | macOS | 11.06 |
57-
| Qwen3 Coder 30B-A3B Instruct | [4-bit quantized][p-4bit] | macOS | 11.90 |
54+
| Model | Compression | Bits Per Weight (BPW) | Platform | Perplexity Score |
55+
| ---------------------------- | ------------------------- | --------------------- | -------- | ---------------- |
56+
| Qwen3 Coder 30B-A3B Instruct | none (`float16`) | 16.00 | macOS | 11.06 |
57+
| Qwen3 Coder 30B-A3B Instruct | [4-bit quantized][p-4bit] | 4.50 | macOS | 11.90 |
5858

5959
[p-4bit]: ../README.md#quantization-options

0 commit comments

Comments
 (0)