(Reaching out to the mentors) Interest in GSOC 2025 Project 9 - Interactive Multimodal Data Explorer

[geeky33]

Dear @samet-akcay, @laurenshogewegintel

I am Aarya Pandey, a second-year Computer Science student at VJTI, with an experience in web development and a strong interest in Machine Learning and AI. I have previously contributed to OpenVINO, and here are some of my merged and resolved issues: PR1, PR2, PR3, PR4, PR5, PR6.
I will be working on more issues by this weekend, because of my university exams, I took a small break.

I am very interested in working on the GSOC 2025 Project 9 "Interactive Multimodal Data Explorer" project and would love to contribute to enhancing Datumaro’s dataset visualization capabilities. To get started, I am considering using Streamlit as the front end to create an interactive interface where users can explore joint embedding spaces generated from models like CLIP, LLaVa, and GPT-4V.

My initial thoughts are:

Embedding Visualization: Using Streamlit and Plotly to display a 3D interactive visualization of embeddings.
Filtering & Navigation: Implementing pan, zoom, and filter functionalities for better dataset exploration.

Annotation Interface: Allowing users to tag noisy or incorrect data directly from the interface.

Integration with Datumaro: Utilizing Datumaro for dataset handling and OTX for embedding computation.

Similarity Search Feature: Implementing a feature where users can select a sample and retrieve the most visually or semantically similar images/texts within the dataset. This can be useful for finding duplicate data, spotting anomalies, and understanding dataset clusters.

Does this direction align with your vision for the project? If so, I would be happy to refine it further and proceed with the proposal.

Looking forward to your feedback!

Thank you for your time and consideration.

Warm regards,
Aarya.

[geeky33]

@p-wysocki @rkazants @dmitry-gorokhov @andrei-kochin @adrianboguszewski

Can I get a connection with a reply from the mentors here?
Thanks.

[adrianboguszewski]

They should respond within a few days :)

[rajeshgangireddy]

Hi @geeky33
I am a co-mentor for other topics and will give my initial response until Laurens is back to work.
Overall, you ideas sound good and are well aligned with the project.

Few points from my side :

1. Please use fully open source models.
2. You can also do a bit of research in finding existing annotation tools to see which annotations features would be best for this topic :)
3. Please also consider if your methods would still work smoothly while scaling to large datasets (more than 50K data points?)
4. You can also have a look at the FAISS library for similarity search.

We look forward to your proposal.
Thanks and let us know if you have any questions.

[laurenshogewegintel]

Dear Aarya,

Your approach and demo looks very good. This is the approach we're interested in.

We're especially interested in novel approaches for computing, visualising, and exploring joint embeddings. I think CLIP like models are an excellent start for the computation. On the visualization side I think we can brainstorm about possibilities when you join the project.

Best,

Laurens

[geeky33]

Dear @laurenshogewegintel @rajeshgangireddy @samet-akcay

Thank you for showing such a positive feedback over my ideation!

I have made the first draft of the proposal and would love to know if there are any prerequisite tasks required to complete it along with the proposals!

would be happy to know your response at the earliest!

Best.

Aarya :)

[geeky33]

Hello @rajeshgangireddy @laurenshogewegintel @adrianboguszewski @samet-akcay
I hope my proposals have reached you through GSoC site
I have sent you my proposals on mail
please let me know if you have any questions on my proposal
would be happy to address them

Best,

Aarya.