[gsoc 26]: Reaching out to mentors of project#18 - "Agentic Toolkit for AI PC with OpenVINO"

[FAbdullah17]

Hi @freddychiu @ravi9 and Everyone! hope you're doing well.
I'm Fahad a 4th year BS Artificial Intelligence student with expertise in AI & ML and AI/ML Engineer with 2+ years of hands-on experience building and deploying generative AI systems, LLM pipelines, and scalable inference services using PyTorch, Hugging Face, FastAPI, Docker, and Kubernetes. some of my work is:

• Virtual-TryOn-System
• GRS-Gemini
• Barren-Plateau-Mitigation-QNN

'm particularly interested in Project #18 (Agentic Toolkit for AI PC with OpenVINO) as it aligns closely with my experience building agent-based systems with LangGraph and LangChain.

I've explored the openvino.genai repository https://github.com/openvinotoolkit/openvino.genai and documentation to understand how LLM pipelines are structured for agent integration. OpenVINO GenAI provides ready-to-use pipelines with C++/Python APIs on top of the OpenVINO runtime .

For example, an LLM pipeline for agent chat is structured as:

import openvino_genai as ov_genai

pipe = ov_genai.LLMPipeline(model_path, "CPU")
response = pipe.generate("What is the weather?", max_new_tokens=200, streamer=streamer)

This shows streaming and chat support already implemented (docs).

Recent work also demonstrates OpenVINO powering local AI agents – for instance, accelerating Qwen3-8B on Intel Core Ultra with speculative decoding and integrating it with 🤗 smolagents (blog).

From this, my architectural understanding is:

LLMPipeline provides the core text generation with chat modes, streaming, and structured output – all essential for agent function calling.

VLMPipeline and TextEmbeddingPipeline can power multimodal agents and RAG retrieval.

Agent frameworks (LangChain, smolagents) can wrap these pipelines as local tools.

Design implications for Project #18:

The toolkit would need to:

• Wrap OpenVINO pipelines into agent-compatible interfaces (e.g., LangChain's BaseLLM, BaseTool).
• Provide local-first tools: LocalLLM (text generation), LocalVLM (vision), LocalRetriever (embeddings).
• Include example agentic workflows (research assistant, coding agent) running fully locally.
• Benchmark local vs. cloud performance to demonstrate value.

The modular GenAI design makes this a natural extension – the same way existing pipelines are extended for new tasks.

I've done some research and built a rough architectural idea of how things will work and how we will implement the project:

To start contributing, I have a few questions:

Would issues related to agent tool structure or LangChain/LangGraph integration be a good place to begin? I've been looking at how LLMPipeline is implemented in src/cpp/pipelines/ and see how it handles streaming, chat modes, and structured output — would contributing to wrapping this as a LangChain component, building a simple LocalLLM tool, or working on benchmarking tools for agent latency make sense as a first step?

I'm hoping to start with something focused that builds toward the full Agentic Toolkit — whether that's extending a utility, adding a helper class, or prototyping a component. If there's an open issue (or planned task) along these lines, I'd love to take it on.
Looking forward to next steps from you guys!!

[FAbdullah17]

Hi @freddychiu @ravi9, just wanted to gently follow up on my previous message from two weeks ago about Project #18 (Agentic Toolkit for AI PC). I'm still very interested in contributing and would appreciate any initial thoughts or guidance when you have a moment. Thanks!

[ravi9]

Hi @FAbdullah17 , Apologies for the delayed response ! Thank you for your patience.
Please submit your proposal formally through the GSoC portal for a formal review.

Yes, LangChain/LangGraph integration, agent tool abstractions, and benchmarking utilities are all good entry points.
There are some existing workflows that you can refer to

1. https://github.com/openvinotoolkit/model_server/blob/main/demos/continuous_batching/agentic_ai/llama_index_agent.py
2. https://github.com/openvinotoolkit/model_server/blob/main/demos/continuous_batching/agentic_ai/openai_agent.py
3. https://github.com/11happy/langchain-openvino/tree/main

Building a simple LocalLLM adapter or a LangChain-compatible wrapper is probably the best first step.

At a higher-level, I'm thinking of having several components in a modular structure which will allow the end user to build agents.

1. CLI/Configuration Layer: ov-agent CLI, YAML Config parser, etc
2. Framework Integration Layer: LangChain/LangGraph, LlamaIndex, AutoGen, etc
3. Agent Core Layer: ReAct Agent, Tool-Calling Agent, etc
4. Provider Abstraction Layer: LLM, Embedder, Retriever Providers.
5. Tools/Functions Layer: MCP Server/Client, Built-in Tools, etc
6. Orchestration/Communication Layer: A2A, etc
7. Observability Layer: Telemetry, Profiler, etc
8. Deployment/Serving Layer: OVMS, FastAPI Server, A2A Server, Docker, etc
9. OpenVINO Runtime

The user could define the agentic workflow in a YAML file and run it via CLI

ovat run --config_file workflow.yml --input "Summarize my documents"

THe workflow.yaml could look like:

workflow:
  id: local_rag_agent
  framework: langchain

llm:
  provider: openvino
  class: LocalLLM
  model_path: ./models/qwen3-8b
  device: CPU
  generation_config:
    max_new_tokens: 256
    temperature: 0.1
    streaming: true

embeddings:
  provider: openvino
  class: LocalEmbeddings
  model_path: ./models/bge-small-en
  device: CPU

retriever:
  class: LocalRetriever
  vector_store: faiss
  persist_path: ./data/index
  top_k: 5

tools:
  - name: search_docs
    class: RetrieverTool
    description: Search local documents relevant to the query.

agent:
  type: react
  system_prompt: |
    You are a helpful local AI assistant.
    Use tools when necessary.
    Prefer local retrieval before answering factual questions.

inputs:
  input_key: question
  chat_history_key: chat_history

outputs:
  output_key: answer

The process could look like:

1. Workflow Engine: Reads YAML Config, creates Langchain
2. Agent Loop: Call LLM through OVMS, Parse/Execute tool calls from response, Repeat or return final response.
3. Tool Executor: Execute RAG (OVMS Embeddings + VectorDB).

Thanks,
Ravi.

cc: @freddychiu

[FAbdullah17]

@ravi9 @freddychiu – Following up on this discussion, I've opened issue #3524 and submitted PR #3530 in openvino.genai with the LangChain wrapper implementation, example agent, tests, and docs as discussed. Would appreciate your review when you have time. Thanks!