UI available now

[byjlw]

Simple UI available with drag and drop.
Just view the instructions in the UI Directory

Install
pip install video-analyzer-ui

Run
video-analyzer-ui

[road2empire]

hey man, great work indeed!

I'm actively working on modifying this project for my needs for last 3 days.

it works perfect with "anthropic/claude-3.5-sonnet:beta", but it's quite costly, 10 frame analyse costs around $0.1.

Did you have any chance with the open source LLMs such as qwen/qwen-2-vl-7b-instruct?

In huggingface benchmarks, they show good results, but it hallucinating a lot.

Here's the result for 1 frame video(attached the frame too).

It happens with multi-frame videos too.

[byjlw]

Yeah that output is pretty bad. I'll run a couple samples with that model to see if I can see what's going on. they might have different pre-proc requirements since based on the output it looks like it's accounting for a small part of the frame.
Did you try it with llama 11b? It's pretty cheap and does pretty well.

[road2empire]

I did couple of llama 11b , it's better than qwen.
But I felt some missing context on "describe.txt" part. Frame descriptions looks okay tough.

I found this one performing best: "google/gemini-flash-1.5-8b"

$0.0375/M input tokens $0.15/M output tokens

Give it a try, it's a bit expensive than llama, but I think it is better

[byjlw]

Missing context from describe.txt?

[road2empire]

What I mean is the LLama was not fully able to describe the captured frame, thus having the missing context from the description of the video(describe.txt).

So, the issue for me was in LLama performance.

I did 5 different videos with both LLama and Gemini, Gemini gave me the desired result in all of them, while LLama maybe 2 good results.