docs: document the full sys__* reserved-variable surface (issue #37)

[zoharbabin]

Summary

• Adds a single canonical "Reserved Template Variables (sys__*)" table to API-REFERENCE.md, right after the Converse section, covering all 8 known reserved variables. None of the 8 had a dedicated table in API-REFERENCE.md before this PR — 5 (sys__thread_id, sys__message_id, sys__user_id, sys__user_message, secrets) were already named in source-level JSDoc (RESERVED_VARS in src/management/conversations.js, SYS_VARS in src/management/prompt-lint.js) but never spelled out for API consumers; the other 3 (sys__ks, sys__is_new_thread, sys__user_obj.*) are confirmed live via testing but aren't enforced or named anywhere else in the SDK yet.
• sys__ks carries an explicit, unambiguous security warning: never reference it in a prompt whose output could be echoed back to a user or logged, cross-linked to SECURITY.md's existing KS guidance.
• Adds a dated (2026-08-22), tracked note describing the current silent turn-failure behavior on sys__user_obj.* — confirmed backend behavior, not fixable from the SDK — cross-referencing this issue (Document and stabilize the full sys__* reserved-variable surface (includes a silent turn-failure bug) #37) and naming previewPrompt()'s reserved_user_attr_unresolved warning as the mitigation once PR Flag unresolvable reserved-variable references in previewPrompt() (#45) #49 (issue Harden previewPrompt() to flag unresolvable reserved-variable references #45), not yet merged, lands. Until then the warning doesn't exist and an unresolved sys__user_obj.* reference is rendered silently.
• Notes sys__user_id will resolve via Sessions.createConversationToken({ userId }) once PR Bind a session to a real end-user identity (userId on session creation) #48 (issue Bind a session to a real end-user identity (userId on session creation) #36), not yet merged, lands — until then, passing userId is silently ignored and the value stays empty on every session.

Fixed post-audit: audit-pr50-clean (independent, context-free subagent) found both notes above originally documented PR #48's and PR #49's capabilities as already shipped. Hedged in the file and here to reflect that neither PR is merged yet.

• Corrects an over-broad claim I initially drafted about which reserved names the SDK's request_vars pre-flight guard rejects — verified against src/management/conversations.js's RESERVED_VARS (only 5 of the 8 names are guarded client-side today) rather than assuming parity across all 8.

Why this doesn't duplicate PR #49's table

PR #49 (not yet merged) adds its own reserved-variable table scoped to previewPrompt()'s client-side simulation. This PR's table is the canonical, live-behavior reference (not simulation-scoped) and lives in a different section (right after Converse, not inside "Preview a Prompt"). Since PR #49 isn't merged into this branch's base, I can't edit its content directly — flagging for the human reviewer merging both: once PR #49 lands, its own table should either link to this canonical one instead of repeating it, or be trimmed to only the simulation-specific caveats (_meta, draftPrompts, etc.), to avoid two tables describing the same 8 variables.

Deviations from the harnessed-build plan

• Initial self-review was performed personally (re-read the diff cold, checked claims against the codebase and PR Bind a session to a real end-user identity (userId on session creation) #48/PR Flag unresolvable reserved-variable references in previewPrompt() (#45) #49 diffs, checked for duplicate tables, checked docs:gate's secret-pattern scan against text that names sys__ks). A separate, genuinely independent clean subagent (audit-pr50-clean, top-level, worktree-isolated, no prior context) then reviewed the merged diff and found two overclaiming defects the self-review missed — see the fix commit on this branch and the note above.

Test plan

• node tools/check-docs.mjs — all 12 doc-gate suites pass (secrets, GFM hygiene, cross-doc links, preview/loading annotation, etc.)
• node scripts/harness/run.mjs — all 3 gates pass (npm run verify, semgrep SAST, npm run docs:gate)
• Manually verified every SECURITY.md/issue/PR cross-reference and anchor slug resolves

🤖 Generated with Claude Code

This satisfies issue #37's documentation scope; left for the human reviewer to close manually after merge review.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[zoharbabin]

Metadata fix: removed a literal "Closes #37" from the PR body. This PR only satisfies issue #37's documentation scope — the backend fix and previewPrompt() hardening (PR #49) are separate, unmerged work — so issue #37 should stay open on merge, not auto-close. Left for the human reviewer to close manually once all of #37's success criteria are actually met.

[zoharbabin]

Independent audit: audit-pr50-clean

A fresh, context-free subagent re-derived every claim in this PR against the repo directly, not from the PR description.

Verified independently (not just re-read):

• grep -rn "sys__" src/ → confirmed the only client-side enforcement of reserved names on main is RESERVED_VARS in src/management/conversations.js (sys__thread_id, sys__message_id, sys__user_id, sys__user_message, secrets — 5 entries) and SYS_VARS in src/management/prompt-lint.js (derived from it). This exactly matches the doc's claim that the SDK's request_vars guard covers "5 of these 8 names," and I confirmed assertRequestVars() only checks against RESERVED_VARS — it does not name-check sys__ks, sys__is_new_thread, or sys__user_obj.*, as the doc states.
• Read src/core/session.js on main directly: createConversationToken(opts) / createAdminToken(opts = {}) have no userId handling at all today — any userId passed is silently dropped. Confirms the doc's "silently ignored... don't rely on it before Bind a session to a real end-user identity (userId on session creation) #48 lands" claim, not just PR Bind a session to a real end-user identity (userId on session creation) #48's own framing.
• Checked issue-45-preview-prompt-hardening (PR Flag unresolvable reserved-variable references in previewPrompt() (#45) #49, confirmed OPEN via gh pr view 49): classifyReservedVar(), RESERVED_EXTRA_SCALARS = ['sys__ks', 'sys__is_new_thread'], and RESERVED_USER_OBJ_PREFIX = 'sys__user_obj' exist there but not on main — confirms sys__ks/sys__is_new_thread/sys__user_obj.* and the reserved_user_attr_unresolved warning are real, specced, unmerged work, not invented, and that this PR's "not yet merged" hedging on both Bind a session to a real end-user identity (userId on session creation) #48 and Flag unresolvable reserved-variable references in previewPrompt() (#45) #49 matches actual repo state.
• gh pr diff 50 --name-only → only API-REFERENCE.md changed. Docs-only, as claimed.
• gh pr view 50 --json closingIssuesReferences → empty, both before and after my edit below (re-checked after the ~15s propagation delay). A prior PR comment already removed a literal "Closes Document and stabilize the full sys__* reserved-variable surface (includes a silent turn-failure bug) #37" from the body, so issue Document and stabilize the full sys__* reserved-variable surface (includes a silent turn-failure bug) #37 correctly stays open on merge.
• Anchors resolve to real headings: SECURITY.md#ks-kaltura-session-guidance-for-agents-ac-3--ac-6--ia-2, #configure-an-intellect, #secrets-write-only — confirmed by grepping the actual heading text, not trusting the link.

Real defect found and fixed: the PR body (and the underlying commit message) framed the new table as covering "8 known reserved variables (the 5 already documented plus 3 newly confirmed live)." I forked a subagent to grep every markdown doc on main (README, GETTING-STARTED, SDK_CONSTITUTION, SECURITY, API-REFERENCE, all of docs/*.md) for these 5 variable names — zero prior mentions anywhere. None of the 8 had any user-facing documentation before this PR; the "5 already documented" framing overstated how much of this was already exposed to API consumers, when at most those 5 were named in source-level JSDoc (RESERVED_VARS/SYS_VARS), never in a doc a developer would actually read. Fixed via gh pr edit 50 --body-file, rewording the Summary's first bullet to state precisely what existed before (source JSDoc naming only, no doc table) versus what's new (the first user-facing documentation of all 8, plus 3 that aren't enforced anywhere in the SDK yet). Re-verified closingIssuesReferences is still empty after the edit.

Gate results:

• node scripts/harness/run.mjs → PASS (3/3: npm run verify, semgrep SAST, npm run docs:gate)
• node tools/check-docs.mjs → PASS (12/12 suites, 28/28 tests, 0 failures)

No other defects found — no GFM issues, no broken cross-references, no invented variables or wrong behavior claims beyond the one fixed above.

🤖 Generated with Claude Code

[zoharbabin]

Live verification: remaining reserved-var claims — all confirmed

This PR's body already noted 3/8 vars (sys__ks, sys__is_new_thread, sys__user_obj.*) were live-tested and the other 5 were only source-verified. Ran a live check against the real backend covering all 8, using throwaway agents (provisioned and deleted in the same run).

Setup: intellects.setPrompts(configId, [], ks, {baseDirective}) to inject each {{var}} into a live base_directive, then conversations.send() on fresh/continued threads, cleanup via agents.delete/avatars.delete/intellects.delete.

Var	Live result
sys__thread_id	Echoed value == the real threadId returned by send().
sys__message_id	Non-empty, changes every turn (turn 1 ≠ turn 2 on the same thread).
sys__user_id	Resolves to empty — matches the doc's note that this is pending PR #48/issue #36.
sys__user_message	Exact match to the text sent, on two separate turns.
sys__is_new_thread	True on turn 1 of a new thread, False on turn 2 of the same thread.
sys__ks	Resolves to a non-empty value — checked presence-only ({{sys__ks}} fed into a model instruction that must never echo it back), consistent with the security warning in this PR's doc addition.
secrets.<NAME>	A throwaway secret set via intellects.secrets.set() resolved to its exact configured value when referenced as {{secrets.LIVE_TEST_SECRET}}.
sys__user_obj.first_name	Reproduced the documented silent turn failure live: send() returned an empty string with no thread created, no thrown error — matches the "Known issue, dated 2026-08-22" callout in this PR's doc exactly.

One caveat worth recording: an early pass on secrets/sys__ks got canned refusals from the model ("I'm not able to process token verification requests...") — that was the model's own safety alignment reacting to test wording that looked like a secret-exfiltration probe (PING-THREE, "token verification"), not a resolution failure. Re-running with benign wording and one fresh thread per var eliminated it. Not a doc issue, just a note in case anyone else reproduces this and mistakes it for a resolution bug.

No inaccuracies found — every row in the Reserved Template Variables table is accurate as written. No doc changes needed.