.bin .pt and .safetensors

[arnoldtheaardvark]

Hi,

Very new to this so apologies for my lack of understanding.

When I try to use the built in models such as facebook-opt6.7 or others from hugging face which are .bin files.
I get en error saying "could not find the quantized model in .pt or .safetensors format"

Could someone please explain either what I need to do to get .bin files working or point me in the direction to learn about the difference please? I have searched a lot and I cant quite get the answer.

I imagine they are different formats.
So can .bin models be used in this tool?
Or is it possible to convert a .bin int a .pt?

Thanks

[BetaDoggo]

Based on the error it sounds like you are trying to load a full fp16 model with the 4bit command line arguments, which won't work since the files have to be pre-quantized before they can be used in 4bit mode.

The main difference between safetensors and the .bin/.pt formats is that safetensors can't execute code so they are safer to distribute. There can also be some loading speed benefits but I don't know if this project takes advantage of those yet. I believe .bin and .pt are both pytorch checkpoints, just with different extensions.

[mcmonkey4eva]

Safetensors speed benefits are basically free. If you use a safetensors file, it just loads faster, not much project impl at all needed. It's just load-times though, and only matters when the bottleneck isn't your datadrive's throughput rate.

[BetaDoggo]

Yeah it seems like I was misremembering #298.

[arnoldtheaardvark]

Thanks for the replies, I have been trying to learn a lot more.
I now understand the differences between the bit size in the models.

My final question if, if I have the appropriate Model and identify it size, can I use .bin model files in ooba?

[BetaDoggo]

Yes, if they have all of the required supporting files they should work.