Skip to content

Pi: emit "input": ["text", "image"] for vision-capable models #2128

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
davanstrien wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Pi: emit "input": ["text", "image"] for vision-capable models #2128

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7b0e996
Pi: emit "input": ["text", "image"] for vision-capable models
davanstrien Apr 27, 2026
c4a9eb0
Pi vision test: also assert "text" in input array
davanstrien Apr 27, 2026
671b44f
Merge branch 'main' into pi-snippet-vision-input
davanstrien May 1, 2026
bf73c01
Merge branch 'main' into pi-snippet-vision-input
davanstrien May 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions packages/tasks/src/local-apps.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,26 @@ curl -X POST "http://localhost:8000/v1/chat/completions" \\
expect(snippet[2].content).toContain("pi");
});

it("pi - vision", async () => {
const { snippet: snippetFunc } = LOCAL_APPS["pi"];
const model: ModelData = {
id: "unsloth/Qwen3.6-35B-A3B-GGUF",
pipeline_tag: "image-text-to-text",
tags: ["conversational"],
gguf: { total: 1, context_length: 4096, chat_template: "{% if tools %}" },
inference: "",
};
const snippet = snippetFunc(model);

expect(snippet[0].content).toContain(`llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:{{QUANT_TAG}} --jinja`);

@cursor cursor Bot May 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test expects --jinja flag that implementation never produces

High Severity

The new vision test asserts that snippet[0].content contains --jinja in the llama-server command, but the snippetPi implementation at line 481 builds the content as `llama-server -hf ${model.id}${getQuantTag(filepath)}` with no --jinja flag appended — and no other code path adds it. The --jinja string appears nowhere in local-apps.ts. This test will fail at runtime. Given the PR discussion confirming jinja is now default in llama.cpp, the test expectation is likely stale and the assertion needs to drop --jinja.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit bf73c01. Configure here.

expect(snippet[1].setup).toContain("npm install -g @mariozechner/pi-coding-agent");
expect(snippet[1].content).toContain(`"id": "Qwen3.6-35B-A3B-GGUF"`);
expect(snippet[1].content).toContain(`"input"`);
expect(snippet[1].content).toContain(`"text"`);
expect(snippet[1].content).toContain(`"image"`);
Comment thread
cursor[bot] marked this conversation as resolved.
expect(snippet[2].content).toContain("pi");
});

it("pi - mlx", async () => {
const { snippet: snippetFunc } = LOCAL_APPS["pi"];
const model: ModelData = {
Expand Down
7 changes: 6 additions & 1 deletion packages/tasks/src/local-apps.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -466,6 +466,7 @@ const snippetMlxLm = (model: ModelData): LocalAppSnippet[] => {
const snippetPi = (model: ModelData, filepath?: string): LocalAppSnippet[] => {
const modelName = model.id.split("/").pop() ?? model.id;
const isMLX = isMlxModel(model);
const isVision = model.pipeline_tag === "image-text-to-text";

// Step 1: Server — differs by backend
const serverStep: LocalAppSnippet = isMLX
Expand All @@ -481,14 +482,18 @@ const snippetPi = (model: ModelData, filepath?: string): LocalAppSnippet[] => {
};

// Step 2: Pi config — port and provider name differ
const modelEntry: Record<string, unknown> = { id: isMLX ? model.id : modelName };

@cursor cursor Bot May 1, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing --jinja flag for vision models in server command

High Severity

The test at line 152 asserts that the llama-server command for vision models includes --jinja, but the implementation on line 481 of local-apps.ts never conditionally appends this flag. The isVision variable is computed but only used for the modelEntry.input field — it's never used to modify the server command. This means either the test will fail, or vision models won't get the --jinja flag they need for proper Jinja template handling with llama.cpp.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 671b44f. Configure here.

@davanstrien davanstrien May 1, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cursoragent IIUC jinja is default in llamacpp now?

@cursor cursor Bot May 1, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unable to authenticate your request. Please make sure to connect your GitHub account to Cursor. Go to Cursor

@julien-c julien-c May 1, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

if (isVision) {
modelEntry.input = ["text", "image"];
}
const modelsJson = JSON.stringify(
{
providers: {
[isMLX ? "mlx-lm" : "llama-cpp"]: {
baseUrl: "http://localhost:8080/v1",
api: "openai-completions",
apiKey: "none",
models: [{ id: isMLX ? model.id : modelName }],
models: [modelEntry],
},
},
},
Expand Down
Loading