[v0.12.3] Llama 3.2 3B Q40 on 4 x Raspberry Pi 5 8GB

[b4rtaz]

Llama 3.2 3B Q40

Model: dllama_model_llama3_2_3b_instruct_q40.m
Version: 0.12.3

	Evaluation	Prediction
4 x Raspberry Pi 5 8GB	27.74 tok/s	13.68 tok/s
2 x Raspberry Pi 5 8GB	23.63 tok/s	10.33 tok/s
1 x Raspberry Pi 5 8GB	14.30 tok/s	5.95 tok/s

4 x Raspberry Pi 5 8GB

...
🔶 P   64 ms S   571 kB R   911 kB import
🔶 P   59 ms S   571 kB R   911 kB  torch
🔶 P   83 ms S   571 kB R   911 kB 

🔶 P   64 ms S   571 kB R   911 kB from
🔶 P   59 ms S   571 kB R   911 kB  transformers
🔶 P   83 ms S   571 kB R   911 kB  import
🔶 P   84 ms S   571 kB R   911 kB  Auto

Evaluation
   nBatches: 32
    nTokens: 19
   tokens/s: 27.74 (36.05 ms/tok)
Prediction
    nTokens: 77
   tokens/s: 13.68 (73.08 ms/tok)

2 x Raspberry Pi 5 8GB

...
🔶 P   97 ms S   190 kB R   429 kB  IoT
🔶 P   97 ms S   190 kB R   429 kB  infrastructure
🔶 P   97 ms S   190 kB R   429 kB ,
🔶 P   97 ms S   190 kB R   429 kB  allowing
🔶 P   97 ms S   190 kB R   429 kB  for
🔶 P   97 ms S   190 kB R   429 kB  more
🔶 P   97 ms S   190 kB R   429 kB  efficient
🔶 P   97 ms S   190 kB R   429 kB  use

Evaluation
   nBatches: 32
    nTokens: 19
   tokens/s: 23.63 (42.32 ms/tok)
Prediction
    nTokens: 77
   tokens/s: 10.33 (96.79 ms/tok)
⭕ Network is closed

1 x Raspberry Pi 5 8GB

🔶 P  168 ms S     0 kB R     0 kB  a
🔶 P  169 ms S     0 kB R     0 kB  powerful
🔶 P  168 ms S     0 kB R     0 kB  cluster
🔶 P  169 ms S     0 kB R     0 kB .


🔶 P  168 ms S     0 kB R     0 kB ##

Evaluation
   nBatches: 32
    nTokens: 19
   tokens/s: 14.30 (69.95 ms/tok)
Prediction
    nTokens: 77
   tokens/s: 5.95 (168.06 ms/tok)

[Fuckingnameless]

1x Rock-5B

model	size	params	backend	threads	fa	test	t/s
llama 3B Q4_0	1.99 GiB	3.61 B	BLAS	4	1	pp64	50.33 ± 0.05
llama 3B Q4_0	1.99 GiB	3.61 B	BLAS	4	1	pp128	50.95 ± 0.05
llama 3B Q4_0	1.99 GiB	3.61 B	BLAS	4	1	pp256	48.29 ± 0.05
llama 3B Q4_0	1.99 GiB	3.61 B	BLAS	4	1	tg64	12.02 ± 0.01
llama 3B Q4_0	1.99 GiB	3.61 B	BLAS	4	1	tg128	11.83 ± 0.02
llama 3B Q4_0	1.99 GiB	3.61 B	BLAS	4	1	tg256	11.04 ± 0.36

[Fuckingnameless]

bitnet new model for the lolz

model	size	params	backend	threads	rtr	test	t/s
============ Repacked 211 tensors
bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	pp64	81.23 ± 0.07
bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	pp128	79.22 ± 0.04
bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	pp256	74.89 ± 0.06
bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	pp512	68.75 ± 0.08
bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	tg64	28.37 ± 0.04
bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	tg128	28.14 ± 0.01
bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	tg256	27.75 ± 0.04
bitnet-25 2B IQ2_BN - 2.00 bpw Bitnet	934.16 MiB	2.74 B	CPU	4	1	tg512	25.45 ± 0.56