GSoC 2026: High-Performance C++ Multimodal Ingestion Pipeline for Deep Search (Idea 2)

[reeseliao]

Hi @zhaohb and @18582088138,

I am writing to express my strong interest in the "OpenVINO Deep Search AI Assistant on Multimodal Personal Database" (Idea 2) for GSoC 2026. I've been diving into your work on the ollama_openvino repository and really appreciate the robust Go-based backend architecture you've built for AIPC deployment.

Recently, I developed a localized multimodal semantic search engine (using LLaVA and Gemma2) designed to extract and analyze non-standard video data, such as animated videos with subtitles. A critical bottleneck I encountered in practice was severe multimodal hallucinations and memory overhead when handling high-frequency video frames and OCR tasks purely through Python wrappers.

While pure Python/FAISS pipelines are excellent for proof-of-concept, I believe building a production-ready standalone desktop application that handles "Word, PowerPoint, PDF, images, and videos" on AI PCs requires stricter resource control.

To complement your existing ollama_openvino backend, I propose building a C++ Native Multimodal Ingestion Client:

• C++ & OpenCV Preprocessing: Utilizing C++ and OpenCV to handle heavy video decoding, keyframe extraction, and heatmap analysis locally, bypassing Python's GIL bottlenecks and keeping RAM usage minimal.
• Hallucination Mitigation: Enforcing strict Structured Outputs (JSON Schema) at the data ingestion layer to ensure the accuracy of OCR and VLM translations before they enter the RAG database.
• Decoupled Architecture: Operating as a lightweight C++ frontend/pipeline that cleans and vectorizes the data, then sends structured API payloads to your existing Go/OpenVINO backend, ensuring seamless integration without altering your core server logic.

Before I draft my formal proposal, I would greatly value your feedback. Does this hybrid architecture—combining a C++/OpenCV native preprocessing pipeline with your ollama_openvino backend—align well with your vision for the AIPC desktop assistant?

Looking forward to your thoughts!

Best regards,

Reese Liao
https://github.com/reeseliao

[18582088138]

@reeseliao
Hi Reese Liao
Thanks a lot for your interest. The C++ architecture design is indeed very helpful for AIPC deployment, but also it means a huge development work. If you're confident in your abilities, I will have high hopes for you.
About your "localized multimodal semantic search engine" project, could you please send me a link?
If not, a resume would also be helpful.

[reeseliao]

Hi @18582088138 (Kunda),

Thank you for the detailed feedback! I’ve taken some time to organize my PoC and document the technical challenges I encountered.

I have just made the repository public:
https://github.com/reeseliao/Multimodal_Video_Search

I also wrote a technical retrospective on my blog regarding the memory and GIL bottlenecks I identified during this Python implementation:
https://reeseliao.github.io/2026/02/23/Building-a-Local-Multimodal-Video-RAG-System/
As you'll see in the demo, while the LLaVA + Gemma 2 pipeline works for semantic search, the local inference overhead is the primary reason I am proposing a C++ Native / OpenVINO migration. I am eager to discuss how we can achieve near-real-time performance on AIPC hardware.

I have also attached my updated resume for your review：
LiaoYudanReese_EngCV_latest.pdf

Looking forward to your thoughts!

Best regards,
Reese Liao

[reeseliao]

Hi @zhaohb @18582088138,

Hope you are having a great week! Just a quick update: I have finalized a comprehensive proposal draft focusing heavily on the C++ native architecture we discussed earlier.

To ensure the feasibility of the pipeline and address the "huge development work" concern, I built an independent C++ Native Ingestion PoC over the past few days. It successfully hit 727 effective FPS on a local AIPC, completely bypassing the Python memory bloat.

I just sent the complete proposal draft and the PoC data to your emails for an early architectural review. Would love to hear your thoughts and critiques whenever you have a moment! Let me know if you need any adjustments to the core C++ design.

[zhaohb]

Hi @reeseliao ,
Do you have any experience with large open-source projects, such as contributing PRs, or maintaining a project that has received stars from other users?

Thank you very much.

[reeseliao]

Hi @zhaohb,
Thank you for getting back to me so quickly!
To be completely transparent, I don't yet have a history of merged PRs to large open-source projects. However, my relevant background includes a 6-month Research Assistantship optimizing PointNet++ for 3D point cloud transmission, a Kaggle Bronze Medal (global top 8%) in LLM Prompt Optimization, and most recently, the C++ Native Ingestion PoC I built specifically for this idea:
https://github.com/reeseliao/GSoC26-OpenVINO-CPP-PoC
I am currently diving into the repository's codebase to prepare my first PR and plan to submit it shortly.
Happy to discuss the technical details of the architecture!

[reeseliao]

Quick update @zhaohb — I dove straight into the codebase today and have just opened my first two PRs:

#34566: Documentation fixes in the CPU and GPU plugins.
#34577: Added PyTorch frontend support for the aten::_fake_quantize_learnable_per_channel_affine operator in C++.

Getting hands-on with the frontend C++ architecture today — right after building the 727 FPS C++ PoC — gives me a lot of confidence in tackling the Native C++ Ingestion Pipeline for this GSoC project. Both PRs are currently running through CI.
Looking forward to your thoughts on the proposal!

[18582088138]

@reeseliao
Thank you very much for your positive feedback recently; your enthusiasm for the project has left a strong impression on us. During our recent interactions, it became clear to me that you possess a very deep understanding of the project. Over the past few weeks, we have been engaged in internal discussions regarding the selection of suitable candidates; if possible, I would appreciate the opportunity to see some of your personal reflections on this specific project.

In this current era dominated by AI, many project tasks—as well as the conceptualization of solutions to problems—can be rapidly facilitated through AI tools. This makes it increasingly difficult to accurately assess a developer's true capabilities. My hope is that the GSoC program will serve to genuinely enhance your personal technical skills and your business-oriented thinking, rather than merely becoming a means to pad your resume.

Regarding this "AI PC" project, I would like you to approach it from the perspective of a real-world product. Consider how to design and optimize applications that are best suited for the Intel hardware platform—specifically, how to deploy these applications effectively within resource-constrained environments. This presents a significant contrast to deploying agents on servers, as you must contend with challenges such as limited memory resources, insufficient local computing power, and the computational limitations of running local LLMs.

Furthermore, a purely local AI PC experience can sometimes be less than user-friendly. If you are able to adopt a "Cloud + Client" hybrid approach for key components of the application, you could significantly enhance the overall user experience. However, doing so would require you to carefully address issues regarding user data security as well as the operational costs associated with calling Cloud LLM APIs. (Please note: the "Cloud + Client" development approach is an optional requirement, not a mandatory one.)

I look forward to reading your thoughts on this project and await your reply. Thank you very much.

[reeseliao]

Hi @18582088138 (Kunda),

Thank you for raising this — it's exactly the architectural question I've been working through since benchmarking my own pipeline on this hardware.

What the numbers actually show

Running LLaVA-7B via Ollama on my Core Ultra 9 185H (CPU path):

Stage	Result
Cold start (first frame, including model load)	16.39s
Stable inference average	~5.1s / frame (35 frames, 188s total)
Pure Python frame extraction	15.70s for 35 frames
C++ extraction (4× denser sampling)	5.62s for 137 frames

Extraction is solved. Inference is not — 5.1s/frame is unusable for a responsive desktop assistant. This directly informs the architecture below.

Proposed architecture: local-first with intelligent cloud escalation

User Query
    ↓
[Privacy Gate] → Tagged private? → Local only, hard stop
    ↓ Non-private
[Complexity Router — lightweight model on NPU]
    ↓ Simple factual              ↓ Complex / exceeds local context
Local LLaVA                   PII Redaction Layer
via OpenVINO                       ↓
                       Session-scoped mapping table (RAM only)
                       regex → fixed-format PII (phone, ID)
                       lightweight NER → unstructured PII
                                   ↓
                       Compressed anonymised prompt → Cloud API
                                   ↓
                       Reverse mapping → final response

Layer 1 — Privacy gate (user-controlled): Users tag data as private via UI. Private data never leaves the device — hard constraint, not a preference.

Layer 2 — Complexity router on NPU: A lightweight intent classifier distinguishes factual retrieval ("who wore red at 3:42?") — answered locally via OpenVINO-accelerated LLaVA — from complex multi-step reasoning exceeding local context limits, which escalates to cloud.

Layer 3 — Privacy-preserving cloud packaging: Before any cloud call, the system runs a two-stage PII redaction pipeline: regex for fixed-format identifiers (phone numbers, IDs), followed by a lightweight NER model for unstructured personal information. Redacted terms are stored in a session-scoped mapping table (RAM only, never persisted to disk). On receiving the cloud response, the system performs reverse mapping to restore context. The mapping table is destroyed on session close or timeout — semantically equivalent to a browser's private mode.

Solving the 5.1s local inference problem

1. Daemon + warm model: Lightweight intent classifier permanently resident on NPU. Heavy VLM async-loaded via OpenVINO with memory reuse — eliminating the 16s cold start penalty.
2. Zero-copy frame pipeline: cv::Mat buffers passed directly as ov::Tensor via pointer injection, no intermediate copies between extraction and inference.
3. Semantic keyframe filtering: MobileNet-scale model pre-filters frames before VLM inference — only frames exceeding a semantic divergence threshold reach LLaVA, reducing inference volume without sacrificing retrieval quality.

Storage and retrieval

I would integrate DiskANN for the vector index — load-on-demand rather than full in-memory FAISS, keeping RAM overhead flat as the personal database grows. Background indexing runs on NPU, leaving GPU headroom for RAG inference and UI rendering.

On Cloud + Client scope

I see this as genuinely optional but high-value. The core constraint is zero raw personal data leaving the device — only redacted, session-scoped semantic summaries. Cloud is a last resort for queries that genuinely exceed local capability, not a default path.

Happy to discuss which components belong in the GSoC scope versus longer-term roadmap.

Best regards,
Yudan Liao (Reese Liao)

[18582088138]

@reeseliao
The application period for GSoC contributors has officially begun. Please pay close attention to the application submission deadline.
If you encounter any uncertainties or wish to discuss specific details during the application process, please feel free to reach out to us via email.
- hongbo.zhao@intel.com
- kunda.xu@intel.com

I would like to add a specific reminder: the application system performs AI-generation checks on submitted content. While you may use AI tools as an aid in drafting your application, we place a much greater emphasis on your personal understanding of—and insights into—the project.

I am optimistic about your proposal; I encourage you to prepare your application with diligence. Please note that the final list of selected participants is subject to the official announcement by GSoC.

A quick reminder about the dates:
- March 16 – March 31 - GSoC contributor application period
- April 21 - 18:00 UTC - GSoC contributor slot requests due from Org Admins
- April 30 - 18:00 UTC - Accepted GSoC contributor projects announced
- May 1 - May 24 - Community Bonding Period
- May 25 - Coding officially begins

[18582088138]

@reeseliao
I have already reviewed the proposal you submitted for GSoC.

Additionally, I would like to remind you that the selection of the model/pipeline is a critical aspect of this project. In application scenarios where hardware resources are constrained, you must consider not only the model's functionality and capabilities but also its hardware resource footprint and response speed.
While AI-assisted programming can certainly facilitate the initial setup of a Proof of Concept (POC), a great deal of work remains to be done to evolve it into a full-fledged product—for instance:
- How to select the appropriate model? The latest LLMs often require specific enablement steps and cannot be directly enabled by openvino.
- How to strike a balance between model performance and accuracy? While optimization techniques—such as model compression—may loss the accuracy, they enable the deployment of models with significantly larger parameter model.
- How to accelerate the application startup and model loading times?
- When a task requires the simultaneous orchestration of multiple modules/models, how can the processing pipeline be optimized effectively to enhance the user experience?
- And so on…
While AI tools can typically provide a generalized design framework for such issues, addressing them effectively in real-world scenarios ultimately requires your own personal experience and a deep understanding of the specific problem at hand.
Furthermore, for this AI-PC project, the objective is not simply to maximize the number of features or the complexity of the application; rather, you must make strategic trade-offs regarding functionality and model selection to satisfy user requirements while utilizing the fewest possible resources.
The points I have raised here are not intended as formal task requirements for your project; rather, my hope is that by reflecting on these questions, you will gain a deeper and more profound understanding of what is involved in developing a real-world project.

[reeseliao]

Hi @18582088138 Kunda,

Thank you for the thoughtful feedback — these are exactly the trade-offs I've been actively benchmarking on real AI-PC hardware.

A few specific design decisions I've already integrated into my architecture in response to these exact constraints:

Model Selection & Enablement: I've scoped the proposal strictly to Small Vision Language Models (SVLMs) optimized for OpenVINO (e.g., Qwen2-VL-2B-Instruct, Florence-2, or Moondream), utilizing INT4/INT8 quantization via NNCF. I deliberately avoided generic, heavy LLMs because they exceed AI-PC memory budgets and cannot be directly enabled without severe conversion overhead.

Performance vs. Accuracy Balance: My initial profiling showed local VLM inference at ~5.1s/frame on a pure CPU baseline. Rather than chasing theoretical accuracy at the cost of UX, I designed a lightweight semantic keyframe filter (MobileNet-scale, NPU-friendly) to drastically reduce VLM call volume—preserving retrieval quality while cutting the compute load by over 70%.

Startup & Loading Time: To address the 16.39s cold-start penalty I measured, I designed a daemon-mode warm loading mechanism. A lightweight intent classifier stays resident on the NPU, while the heavy VLM is async-loaded with OpenVINO memory reuse. Critically, ov::CompiledModel serialises the compiled model to a .blob cache file after first compilation — subsequent loads skip the compilation step entirely, reducing cold-start from ~16s to approximately 1-2s.

Multi-Module Orchestration: The pipeline decouples video extraction, inference, and backend handoff into independent C++ components utilizing a lock-free ring buffer. This ensures the 727 FPS ingestion engine isn't blocked by the slower inference pipeline.

I recognise the gap between an AI-assisted PoC and a production-grade system. These decisions reflect my attempt to make resource-aware trade-offs rather than maximise feature count — which I believe is the right engineering philosophy for constrained AIPC deployment.

Looking forward to your thoughts.

Best regards,
Yudan Liao (Reese Liao)