Help document models minimum requirements (VRAM, RAM, OS etc.)

[geroldmeisinger]

Please help document minimum requirements of different models and their resource usage across different systems:

Template:

Model:
Options:
VRAM:
OS:
GPU:
CUDA version:
Driver version:
Size:

on Linux:
use watch nvidia-smi to get version and resource usage.
cat /proc/version or cat /etc/*-release to get your distribution.
models are located in ~/.cache/huggingface/hub

(also see #161)

[geroldmeisinger]

Model: THUDM/cogagent-vqa-hf
Options: [x] 4-bit, export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
VRAM: ~12GB (approx.)
OS: Debian Linux 12 (bookworm)
GPU: NVIDIA RTX 3060
CUDA version: 12.3
Driver version: 545.23.08
Size: 36.6 GB

Model: THUDM/cogvlm-chat-hf
Options: [x] 4-bit, export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
VRAM: ~12GB (approx.)
OS: Debian Linux 12 (bookworm)
GPU: NVIDIA RTX 3060
CUDA version: 12.3
Driver version: 545.23.08
Size: 35.3 GB

Model: THUDM/cogvlm2-llama3-chat-19B-int4
Options: [x] 4-bit, export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
VRAM: 15112MiB
OS: Debian Linux 12 (bookworm)
GPU: NVIDIA GeForce RTX 4060 Ti
CUDA version: 12.3
Driver version: 545.23.08
Size: 25.1 GB

[geroldmeisinger]

prompt: "describe this image" (or <MORE_DETAILED_CAPTION> for Florence-2)

Meatball

System is

OS: Debian Linux 12 (bookworm)
GPU: NVIDIA GeForce RTX 4060 Ti
CUDA version: 12.3
Driver version: 545.23.08

Options are the following unless stated otherwise

Options: [x] Load in 4-bit, export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

Notable models

• microsoft/Florence-2-large-ft (2GB VRAM, 2GB disk, very tiny, very fast, short but precise description)
• THUDM/cogvlm2-llama3-chat-19B-int4 (16GB VRAM, 26GB disk, huge and slow, but very detailed and most precise description)
• xtuner/llava-llama-3-8b-v1_1-transformers (7GB VRAM, 17GB disk, medium size, detailed description)
• vikhyatk/moondream2 (4GB VRAM, 5GB disk, small size, balanced description)

Considered legacy:

• llava-1.5
• cogvlm-chat-hf, cogagent-vqa-hf
• moondream 1
• internlm-xcomposer2 family
• salesforce-instructblip family
• (Florence-2-base is only meant for training I think)

Benchmark

Model: internlm/internlm-xcomposer2-vl-7b-4bit
VRAM: 11552MiB
Size: 7.0 GB

A man with glasses is holding a fork with spaghetti on it in front of his face.

Model: internlm/internlm-xcomposer2-vl-7b
VRAM: OOM (>16GB)
Size: 17.3 GB
Options: [ ] Load in 4-bit

Model: internlm/internlm-xcomposer2-vl-7b
VRAM: 8112MiB
Options: [ ] Load in 4-bit

A man in a tan jacket and black turtleneck holding a silver fork with a skewer of food sticking out of it.

Model: THUDM/cogvlm-chat-hf
VRAM: 11356MiB
Size: 35.3 GB

This image showcases a man wearing glasses holding a fork with spaghetti and peas attached to it in a playful manner. The man is dressed in a light-colored jacket and a black turtleneck. The background is dark with small white specks giving the impression of a starry night.

Model: THUDM/cogagent-vqa-hf
VRAM: 11174MiB
Size: 36.6 GB

The image showcases a man with glasses holding a fork that has a decorative design resembling an atom or molecule. The man is wearing a black turtleneck and a beige blazer. The background is dark with white specks giving the impression of a starry night or space.

Model: THUDM/cogvlm2-llama3-chat-19B-int4
Options: [x] 4-bit, export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
VRAM: 15112MiB
Size: 25.1 GB

a man in a photograph style, with a playful and whimsical demeanor. He is holding a fork that has a piece of food, possibly a meatball, skewered on it. The fork is positioned in a way that it appears to be a makeshift telescope, with the food acting as the focal point. The man is wearing glasses and a beige jacket over a black turtleneck sweater. The background is a dark, textured surface with specks that resemble stars, giving the impression of a night sky. The lighting is soft and evenly distributed, highlighting the man's face and the food on the fork, while the background remains in shadow. The style of this image is a blend of candid and posed, with an artistic touch that adds a sense of humor and creativity.

Model: llava-hf/llava-v1.6-mistral-7b-hf
VRAM: 5986MiB (OOM >16GB with Options: [ ] Load in 4-bit)
Size: 15.1 GB

The image is a creative and artistic portrait of a man. He is wearing glasses and a black turtleneck. The man is holding a fork with a piece of food on it which appears to be a skewer with various items on it. The skewer is positioned in front of his face and the food items are arranged in a way that they resemble a star or a flower. The background is a dark textured surface that could be a chalkboard or a similar material which adds a dramatic effect to the image. The lighting is soft and appears to come from the upper left side of the frame casting a gentle shadow on the man's face and the skewer. The overall composition of the image is visually striking and suggests a theme of culinary artistry.

Model: llava-hf/llava-v1.6-vicuna-7b-hf
VRAM: 7662MiB
Size: 14.1 GB

The image features a man with a playful and humorous expression. He is wearing glasses and a black turtleneck and he has a fork with a star-shaped arrangement of what appears to be chopsticks stuck in his forehead. The background is a dark textured surface that resembles a chalkboard with white speckles which adds to the whimsical and artistic feel of the image. The man's smile and the absurdity of the fork in his head suggest a lighthearted and possibly satirical tone.

Model: llava-hf/llava-v1.6-vicuna-13b-hf
VRAM: 12548MiB
Size: 26.7 GB

The image is a creative portrait of a man with a playful and artistic twist. The man is wearing glasses and a black turtleneck sweater. He is holding a fork with a star-shaped skewer on it which is piercing his nose. The skewer is adorned with what appears to be a piece of broccoli at the top adding a touch of whimsy to the scene. The background is a dark textured surface that contrasts with the subject making him stand out. The overall composition of the image is striking and unusual with a focus on the man's face and the skewer which draws the viewer's attention.

Model: xtuner/llava-llama-3-8b-v1_1-transformers
VRAM: 6350MiB (OOM >16GB with Options: [ ] Load in 4-bit)
Size: 16.8 GB

In the center of the image a man with a beard and glasses is the main subject. He is dressed in a beige jacket adding a touch of elegance to his appearance. His gaze is directed straight at the camera creating an engaging connection with the viewer.

The man is holding a fork in his right hand which is not just any ordinary fork. This fork is adorned with a unique decoration - a star made of yellow and orange flowers. The star with its vibrant colors stands out against the black background drawing attention to itself.

The background itself is a stark contrast to the man and his decorated fork. It's black but not just plain black. It has white splatters scattered across it adding an artistic touch to the overall composition.

The image is a blend of simplicity and complexity with each element carefully placed to create a visually appealing scene. The man the decorated fork and the black and white background come together to create an image that is both intriguing and aesthetically pleasing.

Model: vikhyatk/moondream2
VRAM: 4316MiB (OOM >16GB with Options: [ ] Load in 4-bit)
Size: 3.7 GB

The image features a man wearing glasses and holding a fork up to his eye with a piece of spaghetti on the fork. He is smiling and looking directly into the camera creating a playful and lighthearted atmosphere. The man is wearing a black shirt and the spaghetti on his fork is positioned in the center of the image. The scene appears to be a creative and fun composition.

Model: vikhyatk/moondream1
VRAM: 7272MiB
Size: 3.7 GB

The image shows a man wearing glasses and a black shirt holding a fork up to his face as if it were a lens. He has a smile on his face and the fork is positioned in such a way that it covers part of his face. The man appears to be looking through the fork creating a playful and imaginative scene. The background features a wall and a chair.

Model: microsoft/Florence-2-large-ft
VRAM: 1190MiB 
Size: 1.5 GB

A man is holding a fork with a piece of food on it. The food is brown and yellow and appears to be spaghetti. The man is wearing black glasses and a beige jacket. He is smiling and has blonde hair. The background behind the man is black.

Model: microsoft/Florence-2-large-ft
VRAM: 2138MiB
Options: [ ] Load in 4-bit

A man is wearing a black turtle neck and a beige jacket. He is holding a silver fork in his hand. He has black glasses on his face. There is a piece of food on top of the fork.

Model: microsoft/Florence-2-large
VRAM: 1196MiB
Size: 1.5 GB

The image is a portrait of a middle-aged man with a fork in his hand. He is wearing a beige jacket and black turtleneck sweater. He has a pair of glasses on and is holding the fork with both hands. The man is looking directly at the camera with a slight smile on his face. On his forehead there is a small piece of food that appears to be a piece of meatball with yellow and green lines wrapped around it. The background is a dark grey wall with small white dots scattered across it.

Model: microsoft/Florence-2-large
VRAM: 2122MiB
Options: [ ] Load in 4-bit

The image is a portrait of a middle-aged man with a serious expression on his face. He is wearing a beige jacket and black glasses. The man is holding a fork in front of his face with the fork resting on his chin. On his forehead there is a small piece of meatball with yellow and green lines wrapped around it. The background is a dark grey with small white dots scattered across it.

Model: microsoft/Florence-2-base-ft
VRAM: 656MiB
Size: 467.1 MB

A man is wearing black glasses. He is holding a fork with food on it. The food is made up of sticks. The man is smiling. He has a black shirt under his jacket.

Model: microsoft/Florence-2-base-ft
VRAM: 916MiB
Options: [ ] Load in 4-bit

A man is wearing black glasses. He is holding a fork in his hand. There are sticks sticking out of the top of the fork.

Model: microsoft/Florence-2-base
VRAM: 658MiB
Size: 467.1 MB

The image is a portrait of a middle-aged man with a fork in his hand. He is wearing a beige jacket and black glasses. The man has blonde hair and is smiling at the camera. On his head there is a small piece of food that appears to be a burger or a dessert. The background is a dark grey color with small white dots scattered across it. The overall mood of the image is playful and whimsical.

Model: microsoft/Florence-2-base
VRAM: 920MiB
Options: [ ] Load in 4-bit

The image is a portrait of a middle-aged man with a fork in his hand. He is wearing a beige jacket and a black turtle neck sweater. The man has short blonde hair and is wearing black-rimmed glasses. On his head there is a small piece of food that appears to be a burger or a dessert. The fork is made up of yellow sticks that are arranged in a way that they form a star-like shape. The background is a dark grey color with small white dots scattered throughout. The overall mood of the image is playful and whimsical.

Model: llava-hf/llava-1.5-7b-hf
VRAM: 4822MiB
Size: 14.1 GB

The image features a man wearing glasses and a suit holding a fork with a piece of food on it. The food appears to be a piece or a small portion of a sandwich. The man is smiling possibly enjoying the meal or posing for a picture. The scene is set against a black background which adds a contrasting element to the image.

Model: llava-hf/llava-1.5-7b-hf
VRAM: 14348MiB
Options: [ ] Load in 4-bit

The image features a man wearing glasses and a black shirt holding a fork with a piece of food on it. The food appears to be a piece or a small portion of broccoli. The man is smiling possibly enjoying the meal or the experience of eating the broccolli. The scene is set against a black background which adds a dramatic touch to the image.

Model: llava-hf/llava-1.5-13b-hf
VRAM: 8368MiB
Size: 26.7 GB

The image features a man wearing glasses and a black shirt holding a fork with a piece of food on it. The food appears to be a piece or a small portion of a pizza. The man is smiling possibly enjoying the taste of the food. The scene is set against a black background which adds a dramatic touch to the image.

Model: llava-hf/bakLlava-v1-hf
VRAM: 5088MiB
Size: 15.7 GB

The image features a man wearing glasses and a black shirt holding a fork with food on it. He appears to be posing for a picture. The food on the fork seems to be an apple and there is a spider on the apple making the scene a bit unusual. The man's glasses are positioned halfway down his nose and he is smiling giving a friendly and approachable vibe.

Model: llava-hf/bakLlava-v1-hf
VRAM: 15368MiB
Options: [ ] Load in 4-bit

The image features a man wearing glasses a black shirt and a tan jacket. He is holding a fork with a piece of food on it possibly a pieceof broccoli. The man appears to be smiling and posing for the camera giving a close-up view of the food on the fork. The scene seems to be set in a dining or kitchen area with a chair visible in the background.

Model: Salesforce/instructblip-vicuna-7b
VRAM: 4858MiB
Size: 31.7 GB

a man with glasses holding a fork with a star on it

Model: Salesforce/instructblip-vicuna-7b
VRAM: 15340MiB
Options: [ ] Load in 4-bit

a man with glasses holding a fork with a star on it

Model: Salesforce/instructblip-vicuna-13b
VRAM: 8306MiB
Size: 56.8 GB

a man is holding a fork in his hand and has a star shaped object on top of it 1 the man is wearing glasses and a jacket and is holding the fork in front of his face. 2 there are several other utensils in the image including a knife and a spoon. 3 the man's hand is holding onto the fork which is positioned in front and to the left of his mouth.

Model: Salesforce/instructblip-flan-t5-xl
VRAM: 4184MiB
Size: 16.1 GB

The image features a man holding a fork with a star-shaped piece of spaghetti in his mouth. The man is wearing glasses and a suit and the fork is placed in his face. The star-shape spaghetti is visible in the upper left corner of the image. The fork can be seen in the lower right corner of this image.

Model: Salesforce/blip2-opt-2.7b
VRAM: 2390MiB
Size: 15.5 GB

with words
?

Model: microsoft/kosmos-2-patch14-224
VRAM: 1322MiB
Size: 6.7 GB

: man holding a fork in front of his face

Not utilizing GPU

• SmilingWolf/wd-swinv2-tagger-v3 size=467.8 MB
• SmilingWolf/wd-convnext-tagger-v3 size=395.3 MB
• SmilingWolf/wd-vit-tagger-v3 size=378.9 MB
• SmilingWolf/wd-v1-4-moat-tagger-v2 size=326.5 MB
• SmilingWolf/wd-v1-4-swinv2-tagger-v2 size=455.7 MB
• SmilingWolf/wd-v1-4-convnext-tagger-v2 size=388.1 MB
• SmilingWolf/wd-v1-4-convnextv2-tagger-v2 size=388.8 MB
• SmilingWolf/wd-v1-4-vit-tagger-v2 size=373.6 MB