Hugging Face Demo

[Mandalorian007]

Motivation & Examples

This project looks absolutely incredible, but I have been really struggling to get it running locally due to not having a dedicated GPU and remote on a colab notebook is a bit messy trying to upload content through tunnels with gradio. I was thinking it would be pretty epic if this project had a Hugging Face style demo to actually try out the project easier. No pressure, but just some feedback from a novice AI vision user.

All the same incredible work this has to be one of the better looking promising projects that has potential to incorporate into other software.

[luca-medeiros]

Appreciate the kind words.
Yea, thats something I thought about it before. I'll update you once I find some time to set up a HF space.

In the meantime, you might find lightning ai's studio a good option to try it out out of the box: https://lightning.ai/studios?section=featured

[Mandalorian007]

Appreciate the kind words. Yea, thats something I thought about it before. I'll update you once I find some time to set up a HF space.

In the meantime, you might find lightning ai's studio a good option to try it out out of the box: https://lightning.ai/studios?section=featured

Thank you I will definitely try this out!

[ld-xy]

Hello, I followed the instructions to install it on the docker server. After the installation is complete, I run python app.py, but I cannot access it on the local browser? Why?

[Mandalorian007]

Appreciate the kind words. Yea, thats something I thought about it before. I'll update you once I find some time to set up a HF space.

In the meantime, you might find lightning ai's studio a good option to try it out out of the box: https://lightning.ai/studios?section=featured

Hey Luca I hope your holidays are going great.

I took your advice and have been trying to get this project up and running on lightning.ai like suggested, but I have been hitting some issues I am struggling to debug.

After swapping env to python 3.11.11 and following installation instructions on repo I leverage the gradio app to run the app.py file (I change the port in the startup to 8000) and then it loads up and I get the litserve running on the base "/" url.

When I copy the public url and go to "gradio" I keep getting http 307s. Any idea what I should check on?

GRADIO_SERVER_PORT=8000 gradio lang-segment-anything/app.py
⚡ ~ GRADIO_SERVER_PORT=8000 gradio lang-segment-anything/app.py

Warning: Cannot statically find a gradio demo called demo. Reload work may fail.
Watching: '/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/gradio', '/teamspace/studios/this_studio/lang-segment-anything'

WARNING:root:No GPU found, using CPU instead
WARNING:root:No GPU found, using CPU instead
Starting LitServe and Gradio server on port 8000...
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
Swagger UI is available at http://0.0.0.0:8000/docs
INFO:     Started server process [20226]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
IMPORTANT: You are using gradio version 3.50.2, however version 4.44.1 is available, please upgrade.
--------
INFO:     130.211.236.75:23616 - "GET / HTTP/1.1" 200 OK
WARNING:root:No GPU found, using CPU instead
WARNING:root:No GPU found, using CPU instead
IMPORTANT: You are using gradio version 3.50.2, however version 4.44.1 is available, please upgrade.
--------
INFO:     130.211.236.75:23616 - "GET / HTTP/1.1" 200 OK
LangSAM model initialized.
Setup complete for worker 0.
INFO:     130.211.236.75:23616 - "GET / HTTP/1.1" 200 OK
INFO:     130.211.236.75:23616 - "GET / HTTP/1.1" 200 OK
INFO:     130.211.236.75:23616 - "GET / HTTP/1.1" 200 OK
INFO:     130.211.236.75:46360 - "GET / HTTP/1.1" 200 OK
INFO:     130.211.236.75:23616 - "GET / HTTP/1.1" 200 OK
INFO:     130.211.236.75:23616 - "GET /gradio HTTP/1.1" 307 Temporary Redirect
INFO:     130.211.236.75:23616 - "GET / HTTP/1.1" 200 OK
INFO:     130.211.236.75:23616 - "GET / HTTP/1.1" 200 OK
INFO:     130.211.236.75:23616 - "GET / HTTP/1.1" 200 OK

[luca-medeiros]

You seem to be running GRADIO_SERVER_PORT=8000 gradio lang-segment-anything/app.py, mind trying python lang-segment-anything/app.py?

You may find better reproducibility by using docker and Dockefile on root.

[luca-medeiros]

does /docs work?

[Mandalorian007]

does /docs work?

That was a good catch I guess the gradio plugin enforces the gradio start. I swapped the application to be fully setup by terminal and then using the Port Viewer Plugin, but I am seeing this behavior:

⚡ ~ python lang-segment-anything/app.py
/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/sam2/modeli
/sam/transformer.py:23: UserWarning: Flash Attention is disabled as it require
a GPU with Ampere (8.0) CUDA capability.
  OLD_GPU, USE_FLASH_ATTN, MATH_KERNEL_ON = get_sdpa_settings()
Starting LitServe and Gradio server on port 8000...
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
Swagger UI is available at http://0.0.0.0:8000/docs
INFO:     Started server process [5365]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/sam2/modeli
/sam/transformer.py:23: UserWarning: Flash Attention is disabled as it require
a GPU with Ampere (8.0) CUDA capability.
  OLD_GPU, USE_FLASH_ATTN, MATH_KERNEL_ON = get_sdpa_settings()
LangSAM model initialized.
Setup complete for worker 0.
INFO:     130.211.236.75:48621 - "GET / HTTP/1.1" 200 OK
INFO:     130.211.236.75:48621 - "GET / HTTP/1.1" 200 OK
INFO:     130.211.236.75:48621 - "GET / HTTP/1.1" 200 OK
INFO:     130.211.236.75:48621 - "GET / HTTP/1.1" 200 OK
INFO:     130.211.236.75:48621 - "GET / HTTP/1.1" 200 OK
INFO:     130.211.236.75:48621 - "GET /gradio HTTP/1.1" 307 Temporary Redirect
INFO:     130.211.236.75:47700 - "GET / HTTP/1.1" 200 OK
INFO:     130.211.236.75:47700 - "GET / HTTP/1.1" 200 OK
INFO:     130.211.236.75:47700 - "GET / HTTP/1.1" 200 OK
INFO:     130.211.236.75:47700 - "GET / HTTP/1.1" 200 OK
INFO:     130.211.236.75:47700 - "GET / HTTP/1.1" 200 OK

Using the /docs endpoint seems to work correctly giving me a FastAPI Swagger page:

INFO:     130.211.236.75:46487 - "GET /openapi.json HTTP/1.1" 200 OK
INFO:     130.211.236.75:46974 - "GET / HTTP/1.1" 200 OK
INFO:     130.211.236.75:46974 - "GET / HTTP/1.1" 200 OK

[jahnvi-sikligar-tcgls]

Hi I tried running the code on Colab, but am facing issues. Firsly, because of the code.

code:

from PIL import Image
from lang_sam import LangSAM
from lang_sam.utils import draw_image

model = LangSAM()
image_pil = Image.open("/content/car1.jpg").convert("RGB")
text_prompt = "wheel"
#masks, boxes, labels, logits = model.predict([image_pil], [text_prompt])
results = model.predict([image_pil], [text_prompt]) #Extract masks, boxes, and labels from results
masks = results[0]["masks"] #Assuming results is a list of dictionaries
boxes = results[0]["boxes"]
labels = results[0]["labels"]

image = draw_image(image_pil, masks, boxes, labels

error:
TypeError Traceback (most recent call last)
in <cell line: 0>()
12 labels = results[0]["labels"]
13
---> 14 image = draw_image(image_pil, masks, boxes, labels)

TypeError: draw_image() missing 1 required positional argument: 'labels'

Apparently, the code is able to generate output as follows with the above mentioned error for display of image with segmentation mask.

How do I fix the error?

[luca-medeiros]

@jahnvi-sikligar-tcgls if you check the code for draw_images, it requires 5 parameters
def draw_image(image_rgb, masks, xyxy, probs, labels).
make sure to include probs.

[jahnvi-sikligar-tcgls]

from PIL import Image
from lang_sam import LangSAM
from lang_sam.utils import draw_image

model = LangSAM()
image_pil = Image.open("/content/car1.jpg").convert("RGB")
text_prompt = "wheel"
#masks, boxes, labels, logits = model.predict([image_pil], [text_prompt])
results = model.predict([image_pil], [text_prompt]) #Extract masks, boxes, and labels from results
masks = results[0]["masks"] #Assuming results is a list of dictionaries
print(masks)
boxes = results[0]["boxes"]
print(boxes)
labels = results[0]["labels"]
print(labels)
probs = results[0]["probs"]
print(probs)
#image = draw_image(image_pil, masks, boxes, probs,labels)

error:
KeyError Traceback (most recent call last)
in <cell line: 0>()
14 labels = results[0]["labels"]
15 print(labels)
---> 16 probs = results[0]["probs"]
17 print(probs)
18 #image = draw_image(image_pil, masks, boxes, probs,labels)

KeyError: 'probs'

The issue of image display is still not solved. Additionally, I also faced issues when running the app.py as well.

[luca-medeiros]

probs = results[0]["probs"]
there is no key probs. Please read the code.

lang-segment-anything/lang_sam/server.py

Line 88 in 04eaddf

output_image = draw_image(

[jahnvi-sikligar-tcgls]

initially, it was mentioned by you to check the probs parameter as the draw image takes 5 parameters. So I do not understand which is correct. The code has been taken from the blog and is not changed much except for the input image. It would be great to have one inference.py which also mentions the environment in which it is able to work and necessary dependencies.

[Learningm]

from PIL import Image
from lang_sam import LangSAM
from lang_sam.utils import draw_image

model = LangSAM()
image_pil = Image.open("./assets/car.jpeg").convert("RGB")
text_prompt = "wheel."
results = model.predict([image_pil], [text_prompt])
masks = results[0]["masks"] #Assuming results is a list of dictionaries
boxes = results[0]["boxes"]
labels = results[0]["labels"]
probs = results[0]["mask_scores"]

image = draw_image(image_pil, masks, boxes, probs, labels)

Minimum script (inference.py) above can work.