Skip to content

Commit b8d89e4

Browse files
borisbatclaude
andcommitted
MCP stdio audit fixes: daslib/logger + popen_argv_pipe + stdin isolation
Closes the stdio-leak audit thread that started from MCP server hangs. Every byte the MCP child process writes is now contained; every byte of the parent's stdin is unreachable to children. daslib/logger (new) - JSON Lines structured logger for daslang tools (MCP, daslang-live, etc). One JSON object per line to {get_das_root()}/logs/<name>.log with ISO 8601 UTC timestamp, level name, dotted category, message, optional fields object. All public API prefixed (logger_init, logger_info, logger_warning, logger_set_min_level, logger_install_hook, ...) so short verbs don't collide unqualified. - logger_install_hook() registers a global DapiDebugAgent that diverts print() / to_log() into the log file -- critical for stdio-transport processes where any accidental stdout write corrupts JSON-RPC framing. One install covers main + every new_thread worker. src/builtin/module_builtin_fio.cpp - Add iso8601_now() builtin (UTC, ms precision, ISO 8601). Used by daslib/logger; also generally useful. - Add popen_argv_pipe() builtin -- bidirectional subprocess pipes (writable stdin + readable stdout). Same shell-bypass semantics as popen_argv. - Harden popen_argv: explicitly isolate child stdin from parent. Windows: open NUL for read, pass as STARTUPINFO.hStdInput. Linux: open /dev/null, dup2(STDIN_FILENO) in child. Prevents a child from silently consuming bytes intended for the parent's JSON-RPC input. include/daScript/simulate/aot_builtin_{fio,time}.h - AOT-side declarations for the two new builtins so generated .cpp links. utils/mcp/* (MCP migration) - main.das + protocol.das: replace bespoke log_open/log_write/log_path infrastructure with daslib/logger. Dotted categories (mcp / mcp.req / mcp.res / mcp.tool / mcp.live / mcp.heap). - tools/live.das: replace two system() calls (powershell launcher, chmod+sh wrapper) with run_and_capture / popen_argv. system() inherited MCP's stdio; any launcher chatter (PowerShell errors, COM messages) would corrupt JSON-RPC. Output now captured + logged. daslib/command_line.das - Add missing `module command_line shared public` declaration. Without it, daslang's AOT codegen emitted only an empty stub for the module, triggering error[50101] hash desync when anything required it under AOT. This unblocks AOT for any future test/utility that uses get_das_exe. Tests - tests/daslib/test_logger.das -- 6 [test] / 12 subtests: JSON Lines write, level filter, dotted-category prefix override, JSON escaping + round-trip, fields serialization, no-op when unconfigured. - tests/mcp/test_mcp_jsonrpc.das -- spawns daslang.exe utils/mcp/main.das via popen_argv_pipe, drives initialize / tools/list / tools/call / method-not-found through JSON-RPC. Multi-chunk read loop for >16KB responses. - tests/mcp/test_popen_argv_pipe.das -- unit tests for the new builtin (echo round-trip, exit code propagation, stdin-EOF-after-block). - tests/fio/popen_argv.das -- new test confirming child stdin isolation (spawn echo fixture, no input -> exits cleanly, no output). - All new test dirs registered in tests/aot/CMakeLists.txt. Docs - daslib/logger added to das2rst.das with 5 groups (Setup / Level filters / Output control / Log calls / Stdout hook) + handmade module-logger.rst + sec_io.rst toctree entry. - iso8601_now categorized under "Time and date"; popen_argv_pipe under "OS specific routines" in builtin/fio doc groups. Handmade descriptions for both. - skills/daslib_modules.md updated with logger entry. Verified locally - Interpreter test suite: 9079 tests, 9073 passed (6 skipped). - AOT test suite: 8423 tests, 8417 passed (6 skipped). - Lint: 0 issues across all changed .das files. - Format: all changed .das files already-formatted. - Sphinx HTML build: build succeeded, no warnings. - MCP server smoke: starts, logs JSON Lines, exits cleanly on stdin EOF. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 4d83c4f commit b8d89e4

22 files changed

Lines changed: 1177 additions & 69 deletions

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,3 +140,4 @@ build-ubsan/.ninja_log
140140
doc/sphinx-build-latex/environment.pickle
141141
opencode.json
142142
test-results/
143+
logs/

daslib/command_line.das

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
options gen2
22
options no_unused_function_arguments = false
33

4+
module command_line shared public
5+
46
//! Command-line utilities for daslang scripts and standalone executables.
57
//! Provides functions for locating the daslang interpreter, which is essential
68
//! when a script or standalone executable needs to spawn daslang subprocesses.

daslib/logger.das

Lines changed: 330 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,330 @@
1+
options gen2
2+
options indenting = 4
3+
options no_global_variables = false
4+
options no_unused_block_arguments = false
5+
options no_unused_function_arguments = false
6+
7+
module logger shared public
8+
9+
//! Structured logging for daslang tools.
10+
//!
11+
//! Writes one JSON object per line ("JSON Lines" / ``ndjson``) to
12+
//! ``{get_das_root()}/logs/<name>.log``. Reuses the runtime ``LOG_TRACE`` /
13+
//! ``LOG_DEBUG`` / ``LOG_INFO`` / ``LOG_WARNING`` / ``LOG_ERROR`` /
14+
//! ``LOG_CRITICAL`` levels. Each record carries an ISO 8601 UTC timestamp
15+
//! (millisecond precision), level name, category, message, and optional
16+
//! ``fields`` object.
17+
//!
18+
//! All public names carry the ``logger_`` prefix — short verbs like
19+
//! ``flush`` / ``close`` / ``info`` / ``debug`` would collide with user
20+
//! code when called unqualified. The intended call shape is
21+
//! ``logger_info("mcp.tool", "...")``, not ``logger::info(...)``.
22+
//!
23+
//! **Categories** are free-form strings; the convention is dotted
24+
//! hierarchy from tool name down to subsystem:
25+
//!
26+
//! - ``"mcp"`` — top-level events for the MCP server
27+
//! - ``"mcp.req"`` / ``"mcp.res"`` — JSON-RPC request / response framing
28+
//! - ``"mcp.tool"`` — tool dispatch (start / complete / error)
29+
//! - ``"mcp.live"``, ``"mcp.live.launch"`` — live-reload subsystem
30+
//! - ``"mcp.heap"`` — heap-collection ticks
31+
//!
32+
//! Filters apply at the most-specific dotted prefix:
33+
//! :func:`logger_set_category_level` ``("mcp.live", LOG_DEBUG)`` enables
34+
//! debug for ``"mcp.live"``, ``"mcp.live.launch"``, etc. — anything else
35+
//! stays at the global :func:`logger_set_min_level`.
36+
//!
37+
//! :func:`logger_install_hook` plugs a GLOBAL debug agent into the runtime
38+
//! so accidental ``print()`` / ``to_log()`` calls are diverted into the
39+
//! log file instead of stdout/stderr — important for stdio-transport
40+
//! processes (MCP server, language servers) where any stdout write
41+
//! corrupts the wire format. One install covers every Context including
42+
//! future ``new_thread`` workers.
43+
44+
require daslib/json public
45+
require daslib/fio public
46+
require strings
47+
require daslib/jsonrpc
48+
require daslib/debugger
49+
require daslib/rtti
50+
51+
// ── State (per-Context) ──────────────────────────────────────────────
52+
53+
var private log_file : FILE const? = null
54+
var private log_path : string
55+
var private log_name : string
56+
var private log_open_attempted = false
57+
var private log_open_failed = false
58+
var private log_min_level = LOG_INFO
59+
var private log_category_overrides : table<string; int>
60+
var private log_fallback_to_stderr = true
61+
62+
// ── Internal helpers ─────────────────────────────────────────────────
63+
64+
def private level_name(level : int) : string {
65+
return "critical" if (level >= LOG_CRITICAL)
66+
return "error" if (level >= LOG_ERROR)
67+
return "warning" if (level >= LOG_WARNING)
68+
return "info" if (level >= LOG_INFO)
69+
return "debug" if (level >= LOG_DEBUG)
70+
return "trace"
71+
}
72+
73+
// Resolve filter level for a category. Most-specific dotted prefix wins —
74+
// "mcp.live.foo" first looks up the full path, then "mcp.live", then "mcp",
75+
// before falling back to the global min level.
76+
def private resolve_level(category : string) : int {
77+
var probe = category
78+
while (true) {
79+
if (key_exists(log_category_overrides, probe)) return log_category_overrides[probe]
80+
let dot = probe |> rfind(".")
81+
break if (dot < 0)
82+
probe = probe |> slice(0, dot)
83+
}
84+
return log_min_level
85+
}
86+
87+
def private should_log(level : int; category : string) : bool {
88+
return level >= resolve_level(category)
89+
}
90+
91+
def private ensure_dir(path : string) {
92+
let dir = dir_name(path)
93+
return if (empty(dir))
94+
let st = stat(dir)
95+
return if (st.is_valid && st.is_dir)
96+
mkdir(dir)
97+
}
98+
99+
def private ensure_open() {
100+
return if (log_file != null || log_open_failed || empty(log_path))
101+
log_open_attempted = true
102+
ensure_dir(log_path)
103+
log_file = fopen(log_path, "ab")
104+
if (log_file == null) {
105+
log_open_failed = true
106+
}
107+
}
108+
109+
// JSON-encode the four fixed fields directly into the writer, then append
110+
// the optional `fields` object via daslib/json's public `write_json` (no
111+
// json_boost reflection — fast path).
112+
def private write_record(var w : StringBuilderWriter; ts, lvl, cat, msg : string; fields : JsonValue?) {
113+
w |> write("\{\"ts\":\"")
114+
write_escape_string(w, ts)
115+
w |> write("\",\"level\":\"")
116+
write_escape_string(w, lvl)
117+
w |> write("\",\"cat\":\"")
118+
write_escape_string(w, cat)
119+
w |> write("\",\"msg\":\"")
120+
write_escape_string(w, msg)
121+
if (fields != null) {
122+
w |> write("\",\"fields\":")
123+
// write_json always pretty-prints (depth=0 still inserts \n/\t between
124+
// elements). One JSON object per line is load-bearing for JSON Lines,
125+
// so flatten via the jsonrpc framing helper (preserves whitespace
126+
// inside string literals).
127+
w |> write(compact_json_whitespace(write_json(fields)))
128+
w |> write("}\n")
129+
} else {
130+
w |> write("\"}\n")
131+
}
132+
}
133+
134+
def private emit(level : int; category, message : string; fields : JsonValue?) {
135+
return if (!should_log(level, category))
136+
ensure_open()
137+
let line = build_string() $(var w) {
138+
write_record(w, iso8601_now(), level_name(level), category, message, fields)
139+
}
140+
if (log_file != null) {
141+
fwrite(log_file, line)
142+
fflush(log_file)
143+
return
144+
}
145+
// Open failed (or path never set). stderr is always safe — it isn't the
146+
// JSON-RPC channel for stdio-transport servers, so a tail fallback can't
147+
// corrupt protocol framing.
148+
if (log_fallback_to_stderr) {
149+
fwrite(fstderr(), line)
150+
}
151+
}
152+
153+
// ── Public configuration ─────────────────────────────────────────────
154+
//
155+
// All public names carry the ``logger_`` prefix on purpose. Short verbs
156+
// like ``flush`` / ``close`` / ``info`` / ``debug`` collide with user
157+
// code when callers ``require daslib/logger`` and reach for the
158+
// unqualified form. The prefix keeps the call sites readable
159+
// (``logger_info("foo", "bar")``) without forcing ``logger::`` everywhere.
160+
161+
def logger_set_name(name : string) {
162+
//! Configure the log file as ``{get_das_root()}/logs/<name>.log``.
163+
//! Re-opens if the path changes; closes the previous handle. Each
164+
//! Context tracks its own state — call once per context.
165+
let new_path = path_join(get_das_root(), "logs/{name}.log")
166+
logger_set_path(new_path)
167+
log_name = name
168+
}
169+
170+
def logger_set_path(path : string) {
171+
//! Override the log file path explicitly. Pass an absolute path or a
172+
//! path relative to the current working directory.
173+
if (path == log_path) return
174+
if (log_file != null) {
175+
fclose(log_file)
176+
log_file = null
177+
}
178+
log_path = path
179+
log_open_attempted = false
180+
log_open_failed = false
181+
}
182+
183+
def logger_get_path() : string {
184+
//! Returns the currently configured log file path (empty if unset).
185+
return log_path
186+
}
187+
188+
def logger_get_name() : string {
189+
//! Returns the logical name passed to :func:`logger_set_name`
190+
//! (empty if the path was set directly via :func:`logger_set_path`).
191+
return log_name
192+
}
193+
194+
def logger_set_min_level(level : int) {
195+
//! Global minimum level. Records below this are dropped before
196+
//! formatting. Default ``LOG_INFO``.
197+
log_min_level = level
198+
}
199+
200+
def logger_get_min_level() : int {
201+
//! Returns the currently configured global minimum level.
202+
return log_min_level
203+
}
204+
205+
def logger_set_category_level(category : string; level : int) {
206+
//! Override the level filter for one category. Dotted prefix
207+
//! matching: an override on ``"mcp.live"`` applies to ``"mcp.live"``,
208+
//! ``"mcp.live.foo"``, ``"mcp.live.bar"`` (unless they have their own
209+
//! more specific override).
210+
log_category_overrides[category] = level
211+
}
212+
213+
def logger_clear_category_levels() {
214+
//! Remove all per-category overrides.
215+
log_category_overrides |> clear
216+
}
217+
218+
def logger_set_stderr_fallback(enabled : bool) {
219+
//! Enable / disable writing to stderr when the log file can't be
220+
//! opened. Defaults to true; set to false for strict stdio-transport
221+
//! servers that must NEVER write to either stream on file failure.
222+
log_fallback_to_stderr = enabled
223+
}
224+
225+
// ── Log calls ────────────────────────────────────────────────────────
226+
227+
def logger_log(level : int; category, message : string; fields : JsonValue? = null) {
228+
//! Generic log entry. Prefer the named-level helpers below.
229+
emit(level, category, message, fields)
230+
}
231+
232+
def logger_trace(category, message : string; fields : JsonValue? = null) {
233+
//! Verbose trace-level record. ``LOG_TRACE``.
234+
emit(LOG_TRACE, category, message, fields)
235+
}
236+
237+
def logger_debug(category, message : string; fields : JsonValue? = null) {
238+
//! Debug-level record. ``LOG_DEBUG``.
239+
emit(LOG_DEBUG, category, message, fields)
240+
}
241+
242+
def logger_info(category, message : string; fields : JsonValue? = null) {
243+
//! Informational record. ``LOG_INFO`` — the default level.
244+
emit(LOG_INFO, category, message, fields)
245+
}
246+
247+
def logger_warning(category, message : string; fields : JsonValue? = null) {
248+
//! Warning record. ``LOG_WARNING``.
249+
emit(LOG_WARNING, category, message, fields)
250+
}
251+
252+
def logger_error(category, message : string; fields : JsonValue? = null) {
253+
//! Error record. ``LOG_ERROR``.
254+
emit(LOG_ERROR, category, message, fields)
255+
}
256+
257+
def logger_critical(category, message : string; fields : JsonValue? = null) {
258+
//! Critical / fatal record. ``LOG_CRITICAL``.
259+
emit(LOG_CRITICAL, category, message, fields)
260+
}
261+
262+
// ── Lifecycle ────────────────────────────────────────────────────────
263+
264+
def logger_flush() {
265+
//! Explicit flush. Each write already flushes; only useful if you
266+
//! disable auto-flush in a fork.
267+
if (log_file != null) {
268+
fflush(log_file)
269+
}
270+
}
271+
272+
def logger_close() {
273+
//! Close the log file. Subsequent writes will lazy-reopen.
274+
if (log_file != null) {
275+
fclose(log_file)
276+
log_file = null
277+
}
278+
log_open_attempted = false
279+
log_open_failed = false
280+
}
281+
282+
// ── Debug-agent hook ─────────────────────────────────────────────────
283+
//
284+
// Intercepts ``Context::to_out`` (the underlying mechanism behind ``print``
285+
// and ``to_log``) and routes every message into the configured log file
286+
// instead of stdout/stderr. Critical for stdio-transport processes — MCP
287+
// servers, language servers, anything where stdout is the wire format.
288+
//
289+
// Registered as a GLOBAL debug agent (not thread-local) under the
290+
// ``"daslib_logger"`` category. One install covers every Context that
291+
// later spawns; ``new_thread`` workers don't need to re-install. The
292+
// agent's ``onLog`` is dispatched into its installing Context, which
293+
// owns the log file handle — cross-context invocations route writes
294+
// through that single handle.
295+
296+
let private LOGGER_AGENT_CATEGORY = "daslib_logger"
297+
298+
class private LogHookAgent : DapiDebugAgent {
299+
def override onLog(context : Context?; at : LineInfo const?; level : int; text : string#) : bool {
300+
// Fail-open when logging isn't configured — let the default
301+
// stdout/stderr path through instead of swallowing silently.
302+
// Makes accidental "hook before logger_set_name" calls harmless
303+
// rather than blackholing every print().
304+
if (empty(log_path) && !log_fallback_to_stderr) return false
305+
var fields : JsonValue? = null
306+
if (at != null && at.fileInfo != null) {
307+
fields = JV({"at" => JV("{at.fileInfo.name}:{at.line}")})
308+
}
309+
emit(level, "runtime", string(text), fields)
310+
return true
311+
}
312+
}
313+
314+
def logger_install_hook() {
315+
//! Install a global debug agent that diverts ``print`` and
316+
//! ``to_log`` (from any Context, current or future) into the
317+
//! configured log file. Idempotent — safe to call from every
318+
//! module's [init], from MCP main, etc. Worker threads spawned via
319+
//! ``new_thread`` inherit the hook automatically; no per-context
320+
//! re-install needed.
321+
return if (has_debug_agent_context(LOGGER_AGENT_CATEGORY))
322+
install_new_debug_agent(new LogHookAgent(), LOGGER_AGENT_CATEGORY)
323+
}
324+
325+
def logger_init(name : string) {
326+
//! Convenience: ``logger_set_name(name)`` + ``logger_install_hook()``.
327+
//! The common-case one-liner for tools that want both behaviors.
328+
logger_set_name(name)
329+
logger_install_hook()
330+
}

doc/reflections/das2rst.das

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,7 @@ require daslib/dynamic_cast_rtti
8888
require daslib/generic_return
8989
// require daslib/heartbeat // qmacro(heartbeat()) conflicts when used in das2rst context
9090
require daslib/lint
91+
require daslib/logger
9192
// require daslib/lint_everything // global_lint_macro causes lint errors in other modules
9293
require daslib/profiler
9394
require daslib/profiler_boost
@@ -169,7 +170,7 @@ get_value|insert_clone|emplace_new|insert_default|emplace_default|get_with_defau
169170
group_by_regex("Memory manipulation", mod, %regex~(intptr|memcmp|variant_index|set_variant_index|hash|memcpy|lock_data|map_to_array|map_to_ro_array)$%%),
170171
group_by_regex("Binary serializer", mod, %regex~(binary_load|binary_save)$%%),
171172
group_by_regex("Path and command line", mod, %regex~(get_command_line_arguments|with_argv)$%%),
172-
group_by_regex("Time and date", mod, %regex~(get_time_usec|ref_time_ticks|get_clock|get_time_nsec|mktime)$%%),
173+
group_by_regex("Time and date", mod, %regex~(get_time_usec|ref_time_ticks|get_clock|get_time_nsec|mktime|iso8601_now)$%%),
173174
group_by_regex("Platform queries", mod, %regex~(get_context_share_counter|das_is_dll_build|is_standalone_exe|get_platform_name|get_architecture_name)$%%),
174175
group_by_regex("String formatting", mod, %regex~(fmt)$%%),
175176
group_by_regex("Argument consumption", mod, %regex~(consume_argument)$%%),
@@ -196,7 +197,7 @@ def document_module_fio(root : string) {
196197
group_by_regex("Directory manipulation", mod, %regex~(dir|dir_rec|mkdir|mkdir_rec|mkdir_result|rmdir|rmdir_rec|rmdir_rec_result|rmdir_result|chdir|getcwd)$%%),
197198
group_by_regex("Glob and pattern matching", mod, %regex~(match_glob|glob|glob_filtered|is_glob_pattern|expand_glob|parse_file_list)$%%),
198199
group_by_regex("Filesystem queries", mod, %regex~(temp_directory|temp_directory_result|create_temp_file|create_temp_file_result|create_temp_directory|create_temp_directory_result|disk_space)$%%),
199-
group_by_regex("OS specific routines", mod, %regex~(sleep|exit|system|popen|popen_binary|popen_timeout|popen_argv|popen_timed_out|run_and_capture|get_env_variable|sanitize_command_line|has_env_variable)$%%),
200+
group_by_regex("OS specific routines", mod, %regex~(sleep|exit|system|popen|popen_binary|popen_timeout|popen_argv|popen_argv_pipe|popen_timed_out|run_and_capture|get_env_variable|sanitize_command_line|has_env_variable)$%%),
200201
group_by_regex("Dynamic modules", mod, %regex~(register_dynamic_module|register_native_path)$%%)
201202
)
202203
documents("File input output library", mod, "fio.rst", groups)
@@ -1173,6 +1174,18 @@ def document_module_toml(root : string) {
11731174
document("TOML 1.0 parser", mod, "toml.rst", groups)
11741175
}
11751176

1177+
def document_module_logger(root : string) {
1178+
var mod = find_module("logger")
1179+
var groups <- array<DocGroup>(
1180+
group_by_regex("Setup", mod, %regex~.*(logger_init|logger_set_name|logger_set_path|logger_get_path|logger_get_name)$%%),
1181+
group_by_regex("Level filters", mod, %regex~.*(logger_set_min_level|logger_get_min_level|logger_set_category_level|logger_clear_category_levels)$%%),
1182+
group_by_regex("Output control", mod, %regex~.*(logger_set_stderr_fallback|logger_flush|logger_close)$%%),
1183+
group_by_regex("Log calls", mod, %regex~.*(logger_log|logger_trace|logger_debug|logger_info|logger_warning|logger_error|logger_critical)$%%),
1184+
group_by_regex("Stdout hook", mod, %regex~.*(logger_install_hook)$%%)
1185+
)
1186+
document("Structured JSON Lines logger", mod, "logger.rst", groups)
1187+
}
1188+
11761189
def document_module_lint_everything(root : string) {
11771190
var mod = find_module("lint_everything")
11781191
var groups <- array<DocGroup>(
@@ -1694,6 +1707,7 @@ def main {
16941707
document_module_lint(root)
16951708
document_module_lint_config(root)
16961709
document_module_toml(root)
1710+
document_module_logger(root)
16971711
// document_module_lint_everything(root) // global_lint_macro causes lint errors in other modules
16981712
// document_module_live(root) // uses multiple_contexts
16991713
document_module_lpipe(root)
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Returns the current UTC wallclock time as an ISO 8601 string with millisecond precision: ``"YYYY-MM-DDTHH:MM:SS.mmmZ"`` (24 characters, trailing ``Z`` for UTC). Used by ``daslib/logger`` for log-record timestamps; useful anywhere a sortable, parseable, timezone-explicit timestamp is needed.

0 commit comments

Comments
 (0)