Skip to content

Gate ZAYA1-VL tools on the actual template contract - #2540

Draft
jjang-ai wants to merge 3 commits into
mainfrom
fix/zaya-vl-existing-bundle-capability
Draft

Gate ZAYA1-VL tools on the actual template contract#2540
jjang-ai wants to merge 3 commits into
mainfrom
fix/zaya-vl-existing-bundle-capability

Conversation

@jjang-ai

Copy link
Copy Markdown
Contributor

Summary

  • honor explicit supports_tools=false before treating a parser name as tool capability
  • protect already-published zaya1_vl bundles whose JANG metadata accidentally inherited supports_tools=true / zaya_xml
  • require a ZAYA1-VL template to contain both a tools input and a tool-call output contract before exposing tools
  • allow a future genuinely repaired VL template to opt back in without a model-name allowlist or app release

Before / causal evidence

The installed JANGQ-AI/ZAYA1-VL-8B-JANGTQ4 bundle currently has:

  • config.json: model_type=zaya1_vl
  • jang_config.json: supports_tools=true, tool_parser=zaya_xml
  • chat_template.json and tokenizer_config.json: no tools input and no tool_call emission contract

The matched 3-trial AgentLoop run produced 0/3 valid spawn_batch calls. A bounded raw trace on the native upstream-shaped schema selected spawn_batch but emitted schema declarations in place of values and omitted required target/input; the parser correctly rejected it. This is not safely repairable with an argument-parser heuristic.

Artifacts from that reproduction:

  • /private/tmp/zaya-pr336-spawn-batch-repeat3.json
  • /private/tmp/zaya-pr336-spawn-batch-repeat3.transcripts/
  • /private/tmp/zaya-pr336-native-template-trace.json
  • /private/tmp/zaya-pr336-native-template-trace.transcripts/

Future bundle prevention is separately scoped in JANG PR #22. vMLX template truthfulness remains separately scoped in vMLX PR #336.

Fix contract

  • supports_tools=false is authoritative even when a parser string is present.
  • For zaya1_vl / zaya_vl, a legacy true stamp is insufficient: the actual template must consume tools and emit tool_call syntax.
  • Text-only or multimodal requests with no tools remain loadable; only unsupported tool-bearing requests are rejected at runtime policy preflight.
  • No prompt coercion, parser guessing, fabricated arguments, sampler override, or model-name permanent allowlist is added.

Current-head verification

Head: cf1622679b5b8336cebde328669464d583f41d76

  • MLXServiceRuntimePolicyTests: 18/18 PASS.
  • The focused test was run with OSAURUS_ZAYA_VL_TEST_MODEL=/Users/eric/models/JANGQ-AI/ZAYA1-VL-8B-JANGTQ4; the real installed bad-stamp bundle resolves tool support to false.
  • Synthetic regression rows prove legacy true + tool-less template = false, explicit false = false, and future true + real tools/tool_call template = true.

Honest status

PARTIAL / draft. Source and installed-bundle capability resolution are proven. Remaining live gates:

  • exact-head Release app visually shows the bundle as text/VLM-capable but does not hand it an unusable tool schema;
  • real image and video payloads still require VLM decode plus media-salted cache-hit telemetry;
  • raw CCA B=2 unequal-prefix/state-isolation proof remains independent of tool capability;
  • current CI must complete.

This PR prevents broken tool exposure for current users; it does not claim that the present ZAYA1-VL checkpoint has learned valid tool calling.

@github-actions github-actions Bot added the bug Something isn't working label Aug 29, 2026
@jjang-ai

Copy link
Copy Markdown
Contributor Author

Current live gate result — keep the fail-closed policy (979a62e5d)

I tested the tempting relaxation against the exact published bundle and rejected it.

Paired integration under test:

  • Osaurus capability experiment equivalent to commit b87aef657 (recognized the stamped vMLX fallback contract).
  • vMLX cache model capability #336 head 68b2627de71c034d297eb16f2701cc609036b6e7 (fixes dropped LMInput.toolSchemas).
  • Installed model: JANGQ-AI/ZAYA1-VL-8B-JANGTQ4.

The lower engine boundary passes a strict one-tool request: valid ZAYA XML, one structured get_weather(location=Tokyo), no marker leak. But the real Osaurus AgentLoop schema is the product gate:

  • agent_loop.list-folder-contents, 3 trials, 14 real tool schemas / 2,664 tool-schema tokens.
  • Result: 0/3; no tool calls in any trial; all exited finalResponse.
  • Outputs hallucinated directory contents or described how the user should invoke file_search; none grounded invoice.txt, photos, or notes.
  • Measured row: 14.1 tok/s, 97 ms TTFT, 2.8 GB peak process RAM, L2 +0 hit/+2 miss/+1 store.
  • Artifact: /private/tmp/zaya336-agentloop-3x-after-policy.json; transcripts: /private/tmp/zaya336-agentloop-3x-after-policy.transcripts/.

Therefore a valid stamped runtime fallback is not sufficient proof of harness tool capability. I reverted the relaxation in 979a62e5d; the PR once again blocks the published VL bundle exactly as intended. This remains the correct fail-closed product behavior until a bundle/template/model revision passes the broad AgentLoop tool surface. The vMLX schema-propagation bug is tracked separately in #336 and must not be presented as making ZAYA VL agent-ready.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant