Tunning Proposal for GSoC26 Project #1 - GUI Agent with VLM/LLM and OpenVINO.

[Sahilbhatane]

Referring ongoing discussion - discussioncomment-15986519 .

This is My current Proposal and I need Guide to tune it further before I draft my final proposal on GSoC official site -
GUI Agent Proposal.pdf

Quick discussion summary of above linked discussion -

1. Showed interest and give quick overview of what I thought how we can build this project.
2. made a MVP which works near same (some functionalities) as UL-TARS level.
3. Q/A answered by @zhuo-yoyowz, @openvino-dev-samples and adrianboguszewski.
4. Prerequisites done with 1 open issue ([Bug]: GPU_ENABLE_LARGE_ALLOCATIONS ignored for large constants during GPU compile (ProgramBuilder before program) #34967), and merge PR ([GPU] Improve max-allocation error guidance and enable_large_allocations docs #34894) and open PR ([GPU] Forward CMAKE_SYSROOT and CMAKE_FIND_ROOT_PATH to oneDNN GPU ExternalProject #34861 and [GPU] Apply enable_large_allocations before ProgramBuilder constant GPU uploads #35022)

Any feedback so I can make it more polish or more aligned with what you want it to be like.

cc: @openvino-dev-samples and @zhuo-yoyowz

[Sahilbhatane]

@openvino-dev-samples and @zhou-yoyowz I have submitted my proposal on GSoC portal, its same as above with slight change in Prerequisites where PR #34894 has been merged.

[KarSri7694]

Referring ongoing discussion - discussioncomment-15986519 .

This is My current Proposal and I need Guide to tune it further before I draft my final proposal on GSoC official site - GUI Agent Proposal.pdf

Quick discussion summary of above linked discussion -

1. Showed interest and give quick overview of what I thought how we can build this project.

2. made a MVP which works near same (some functionalities) as UL-TARS level.

3. Q/A answered by @zhuo-yoyowz, @openvino-dev-samples  and **adrianboguszewski**.

4. Prerequisites done with 1 open issue ([[Bug]: GPU_ENABLE_LARGE_ALLOCATIONS ignored for large constants during GPU compile (ProgramBuilder before program) #34967](https://github.com/openvinotoolkit/openvino/issues/34967)), and merge queue PR ([[GPU] Improve max-allocation error guidance and enable_large_allocations docs #34894](https://github.com/openvinotoolkit/openvino/pull/34894)) and open PR ([[GPU] Forward CMAKE_SYSROOT and CMAKE_FIND_ROOT_PATH to oneDNN GPU ExternalProject #34861](https://github.com/openvinotoolkit/openvino/pull/34861) and [[GPU] Apply enable_large_allocations before ProgramBuilder constant GPU uploads #35022](https://github.com/openvinotoolkit/openvino/pull/35022))

Any feedback so I can make it more polish or more aligned with what you want it to be like.

cc: @openvino-dev-samples and @zhuo-yoyowz

Hey @Sahilbhatane , thanks for your PR #34894 I was actually hitting the same CL_DEVICE_MAX_MEM_ALLOC_SIZE when single tensor size was >4gb, I never knew the workaround already existed😅. It would now make it easier to have higher context window with GPU_ENABLE_LARGE_ALLOCATIONS flag enabled.

[Sahilbhatane]

Hey @Sahilbhatane , thanks for your PR #34894 I was actually hitting the same CL_DEVICE_MAX_MEM_ALLOC_SIZE when single tensor size was >4gb, I never knew the workaround already existed😅. It would now make it easier to have higher context window with GPU_ENABLE_LARGE_ALLOCATIONS flag enabled.

Well Thank you @KarSri7694. There is also PR #35022 open which will permanently fix the issue.

[Sahilbhatane]

@adrianboguszewski, @openvino-dev-samples and @zhuo-yoyowz I have submitted the proposal. I was not getting any replies on previous discussion that's why I have created this discussion, any suggestions on the proposal or MVP?

[Sahilbhatane]

I have stopped working on MVP as suggested by @adrianboguszewski till the point it was a working MVP.
Additional I have used "OpenVINO's" model from huggingface as primary VLM for the MVP so that it will be more aligned with organization in future.
One more question - is there any other communication channel that OpenVINO's dev use for communicating as communicating through discussion tabs is quite public.
This is also same case for If we get selected which communication channel are we going to use?

[adrianboguszewski]

When you're selected you will use a private channel with your mentor (most probable it will be an email)

[Sahilbhatane]

Hey @adrianboguszewski, @zhuo-yoyowz, @openvino-dev-samples. I was not selected for this project little sad but its OK, I am and will still contribute to this and other project, will be in communication with you all in PRs as contributor.
Just a question - I have made the MVP for this project and made quite a progress on it and I would like to keep the project for my portfolio, so can I keep it as my project? I know it's a open source project, but because it was organization's idea and it's always respectful to ask.

• I'll close this discussion after the answer, thanks for all guide till now and in future as well

[adrianboguszewski]

Thanks @Sahilbhatane being part of this. Unfortunately, the winner takes all in that kind of programs.
I'll let the mentors to make a decision on this @openvino-dev-samples and @zhuo-yoyowz