Skip to content

Commit b675674

Browse files
as-suvorovCopilot
as-suvorov
and
Copilot
authored
[agentic_model_enabler]: Add agent (openvinotoolkit#3679)
<!-- Keep your pull requests (PRs) as atomic as possible. That increases the likelihood that an individual PR won't be stuck because of adjacent problems, merge conflicts, or code review. Your merged PR is going to appear in the automatically generated release notes on GitHub. So the clearer the title the better. --> ## Description <!-- Please include a summary of the change. Also include relevant motivation and context. --> ## Checklist: - [x] This PR follows [GenAI Contributing guidelines](https://github.com/openvinotoolkit/openvino.genai?tab=contributing-ov-file#contributing). <!-- Always follow them. If there are deviations, explain what and why. --> - [NA] Tests have been updated or added to cover the new code. <!-- Specify exactly which tests were added or updated. If the change isn't maintenance related, update the tests at https://github.com/openvinotoolkit/openvino.genai/tree/master/tests or explain in the description why the tests don't need an update. --> - [x] This PR fully addresses the ticket. <!--- If not, explain clearly what is covered and what is not. If follow-up pull requests are needed, specify in the description. --> - [NA] I have made corresponding changes to the documentation. <!-- Run github.com/\<username>/openvino.genai/actions/workflows/deploy_gh_pages.yml on your fork with your branch as a parameter to deploy a test version with the updated content. Replace this comment with the link to the built docs. If the documentation is updated in a separate PR, clearly specify it. --> --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
1 parent 3c326e5 commit b675674

4 files changed

Lines changed: 83 additions & 8 deletions

File tree

‎.github/agents/model-enabler.agent.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
---
2+
description: "Enable a new model for OpenVINO GenAI. Use when: validating a new HuggingFace model, checking model support, running model-checker, updating supported-models docs, model enablement workflow, enabling a model with optimum-intel."
3+
tools: [read, edit, search, execute, todo]
4+
argument-hint: "<model_id> <task> e.g. google/gemma-3-4b-it image-text-to-text"
5+
---
6+
7+
You are the OpenVINO GenAI Architect. Your job is to fully enable a new HuggingFace model by validating it works with OpenVINO GenAI and updating the site documentation to reflect its support.
8+
9+
## Skills
10+
11+
| Skill | Path |
12+
| ------------- | --------------------------------------- |
13+
| model-checker | `.github/skills/model-checker/SKILL.md` |
14+
| update-docs | `.github/skills/update-docs/SKILL.md` |
15+
16+
## Inputs
17+
18+
Expect the user to provide:
19+
20+
- **model_id**: HuggingFace model identifier (e.g. `google/gemma-3-4b-it`)
21+
- **task**: optimum export task (e.g. `image-text-to-text`, `text-generation-with-past`)
22+
23+
If either is missing, ask for them before proceeding.
24+
25+
## Workflow
26+
27+
### Step 1: Model Validation
28+
29+
Read and follow the **model-checker** skill.
30+
31+
Read **model-checker** step results. Depending on the results:
32+
33+
- If all steps passed, proceed to Step 3.
34+
- If optimum-intel export failed or wwb results for optimum-intel below threshold, proceed to Step 4. Provide a summary of the failure and relevant log paths for context.
35+
- If GenAI inference test failed or wwb results for GenAI below threshold, proceed to Step 2.
36+
37+
### Step 2: Model Enablement
38+
39+
Proceed with model enablement.
40+
41+
### Step 3: Documentation Update
42+
43+
Read and follow the **update-docs** skill.
44+
45+
### Step 4: Final Report
46+
47+
Report a structured summary:
48+
49+
- **Model**: `<model_id>` (`<task>`)
50+
- **Validation**: PASSED / FAILED
51+
- **Performance** (if passed):
52+
- 1st token latency, 2nd token latency, throughput
53+
- Optimum similarity / GenAI similarity
54+
- **Model Enablement Status**:
55+
- **Enabled/Not Enabled** if passed all model-checker steps
56+
- **Details**: Provide a summary of changes. Highlight design and architectural decisions made during enablement.
57+
- **Docs Update Status**:
58+
- **Updated/Not Updated** if updated the supported models docs
59+
- **Details**: summary of doc changes.
60+
- **Details**: Additional details required for context, next steps, or follow-ups.

.github/skills/genai-model-checker/SKILL.md renamed to .github/skills/model-checker/SKILL.md

Lines changed: 20 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
---
2-
name: genai-model-checker
2+
name: model-checker
33
description: "Validate a newly supported optimum-intel model with OpenVINO GenAI. Use when: checking new model support, verifying model export to OpenVINO IR, running GenAI inference test with llm_bench, benchmarking model accuracy with who-what-benchmark."
44
argument-hint: "model_id and task (e.g. tencent/HY-MT1.5-1.8B text-generation-with-past)"
55
---
66

7-
# GenAI Model Checker
7+
# Model Checker
88

99
Validates that a HuggingFace model exported via optimum-intel works correctly with OpenVINO GenAI pipelines and passes accuracy benchmarks.
1010

@@ -31,13 +31,15 @@ The user must provide:
3131

3232
## Prerequisites
3333

34-
Activate the Python virtual environment before running any commands.
34+
Ensure the Python virtual environment is activated before running any commands.
3535

3636
1. **Locate the virtual environment** — check for common directories at the repository root: `.venv/`, `venv/`, `env/`. Use `list_dir` to find it. If none is found, ask the user for its location.
37-
2. **Activate** based on the current platform:
37+
2. **Check if already activated**: if `which python` or `where python` points inside the virtual environment, it's already activated. If not, proceed to activate it.
38+
3. **Activate** based on the current platform:
3839
- **Linux/macOS**: `source <venv_path>/bin/activate`
3940
- **Windows (cmd)**: `<venv_path>\Scripts\activate.bat`
4041
- **Windows (PowerShell)**: `<venv_path>\Scripts\Activate.ps1`
42+
4. The background terminal doesn't inherit the venv activation. Run it with the venv activated in the same command.
4143

4244
## Procedure
4345

@@ -46,13 +48,13 @@ Activate the Python virtual environment before running any commands.
4648
Run the checker script from the repository root:
4749

4850
```
49-
python3 .github/skills/genai-model-checker/scripts/check_model.py \
51+
python3 .github/skills/model-checker/scripts/check_model.py \
5052
--model-id <model_id> \
5153
--task <export_task> \
52-
--work-dir /tmp/genai-model-check
54+
--work-dir .model_enabler/model_checker
5355
```
5456

55-
Run `python3 .github/skills/genai-model-checker/scripts/check_model.py --help` for the full argument reference including defaults.
57+
Run `python3 .github/skills/model-checker/scripts/check_model.py --help` for the full argument reference including defaults. The `--work-dir` is where all intermediate files, logs, and outputs will be stored. Do not pipe with any additional logging or redirection — the script handles its own logging.
5658

5759
#### Skip flags (for re-runs after a fix)
5860

@@ -81,9 +83,19 @@ The script logs progress for each step and exits with code 0 (pass) or non-zero
8183

8284
**Log files:** each tool writes its own dedicated log; paths are printed during execution. When a step fails, read the corresponding log for the full traceback and context before drawing any conclusions.
8385

86+
**work-dir:** work-dir is in current workspace, prefer to use tool calls to access logs and outputs instead of custom bash commands.
87+
8488
### Step 3: Report Results
8589

86-
Report results to the user. If all steps pass, indicate success and provide performance metrics and accuracy data. If a step fails, analyze tool logs and provide a summary with the failed tool log path.
90+
Results format:
91+
92+
- **Model**: `<model_id>` (`<task>`)
93+
- **Validation**: PASSED / FAILED
94+
- **Performance** (if passed):
95+
- 1st token latency, 2nd token latency, throughput
96+
- Optimum similarity / GenAI similarity (if applicable)
97+
- **Logs**: paths to export log, llm_bench log, WWB logs
98+
- **Failed step analysis** (if failed): summary of the failure and relevant log path for details
8799

88100
### Security
89101

.github/skills/genai-model-checker/scripts/check_model.py renamed to .github/skills/model-checker/scripts/check_model.py

File renamed without changes.

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,3 +44,6 @@ __pycache__
4444

4545
# CodeQL artifacts
4646
_codeql_detected_source_root
47+
48+
# agentic model enabler
49+
.model_enabler

0 commit comments

Comments
 (0)