utils/jobque-timeline: the per-lane jobque trace viewer, and its story - #3553
Merged
Conversation
Productizes the browser-artifact lane-timeline: opens jobque trace JSON (self-describing jobque_profile format, legacy files fall back) in dockable windows with the artifact's rendering — per-lane rows, category colors with stage shading, publish ticks, barrier waits, marker/unit row, overview strip. - tl_loader: rigid-format line parser (no JSON DOM for events; exact int-ns time parse), chain resolution, per-lane sort, jobqueProfile header - tl_lod: per-lane bucket mipmaps; viewer switches to run-merged bucket rendering above 8k visible events, and replays CACHED draw runs until the view changes (106k-event decode trace: 12.5 -> 68 fps interpreted) - tl_view: wheel zoom-at-cursor, drag pan, dblclick reset, overview scrub, shift+drag range selection, hover tooltips; custom canvas via the sanctioned ItemSize/ItemAdd/ButtonBehavior rail (all lint-allowed calls) - tl_stats: computed chips — idle %, parallelism, publishes, busy by category — for the view and the selection, cached per range+generation - multi-file compare: stacked dock layout, share zoom + share timer ON by default (one-way intent propagation, per-file edge clamping) - tl_launch: launch rail — pick an app (dasStdDlg dialog), args, run via popen_argv in a background job with output tail + exit code; mtime/size debounced auto-refresh reloads changed traces (manual reload buttons too) - timeline_* live commands (files/open/get+set view/focus_unit/selection/ range_stats/describe/launch/refresh) — selections and stats come back as structured JSON, no screenshots needed - co-located dastest files (loader round-trip incl. old-format fallback, stats math, argv split + file watch) Setup: daspkg install --root utils/jobque-timeline (fetches + builds dasImgui against the local daslang). Run: daslang -project_root utils/jobque-timeline utils/jobque-timeline/main.das -- trace.json Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dasStdDlg's native dialog silently no-ops here — and the right rail is dasImgui's own imgui_file_dialog anyway (pure boost widgets, so it's snapshot-visible and playwright-drivable). One modal serves two consumers via g_dlg_purpose: a new "open trace..." button (json filter, multi-select up to MAX_DOCS) and the launch-app "browse..." (exe filter). Dialog draws at update() top level per its registry-path contract. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…anel - per-file rows: refresh + close icon buttons (imgui_icons) ahead of the label; trace dock windows get tab close buttons too (closable = true). Closes are deferred to end of frame (close_file compacts the parallel arrays, retitles, resets the positional dock slots, reseeds the layout) - separator between the sync checkboxes and the file section - Launch moves to its own dock window at the bottom of the left column (folder icon + app name, args, play button, output tail under a separator) - open-trace/browse/launch buttons are icon_buttons with tooltips - same_line in the per-file loop uses indexed EmptyMarkerState idents (single-global widgets must render once per frame) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Files with unequal spans no longer pin at their edges under share-timer: sync adoption is UNCLAMPED (the shorter file draws honest empty space when the shared time range is past its data), so 117ms decode vs 1508ms prefill track each other exactly in both directions. Direct interaction still clamps — perfetto-style, at most one window-width past either end — so a single view can't get lost in the void. Negative grid times format sanely; the overview viewport rect clips to the strip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- a .das app runs as `daslang -jit <script> -- <args>` via the tool's own binary (get_das_exe) — benchmarks are scripts, picking them directly beats spelling the interpreter invocation; browse filter gains *.das - args: the cramped single-line input is gone — the panel shows the args as a label with an edit icon opening a popup_modal multi-line editor (seeded through the InputTextState pending rail; newlines count as whitespace in split_args, so one-arg-per-line needs no quoting) - timeline_launch gains start=false (pre-fill the panel without running) - fio: new set_env_variable builtin (process-wide, inherited by popen children) — groundwork for the launch rail owning DASLLAMA_*_TRACE env vars instead of requiring them on the tool's own environment (wiring lands after the next daslang rebuild) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Launch-output lines are plain ImGui text (not OS-selectable), so lines that name a trace file (e.g. "jobque trace saved: <path>") now get two small icon buttons ahead of the text: open the trace (open_or_reload — an already-open path reloads instead of duplicating; the dialog multi-select uses the same guard) and copy the path to the OS clipboard (via the sanctioned log_to_clipboard/log_text/log_finish rail). Path detection = first whitespace token ending .json (extract_json_path, tested). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…esults
- fio set_env_variable finished (stub + registration + AOT decl) — children
of popen inherit process env
- launch_start owns the dasLLAMA env-armed trace rails: DASLLAMA_PREFILL_TRACE
and DASLLAMA_BATCH_TRACE point at <temp>/jobque-timeline/, so env-only
benchmarks (prefill_perf, batch_decode_perf) trace without any setup outside
the tool; DAS_JOBQUE_AFFINITY follows the new Launch-panel affinity checkbox
(ON by default — bench discipline)
- on a rc=0 finish, every trace the run reported saving ("jobque trace
saved: <path>" lines) auto-opens in the viewer
- output tail keeps 60 lines (profile dumps pushed the saved-line out of 30)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DAS_JOBQUE_THREADS for the launched child as an input next to affinity; 0 = inherit. Seeded from the tool's own environment at init so the box config (e.g. DAS_JOBQUE_THREADS=16) shows up as the default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tutorial-00 story, and the trailer reel. Lands with the jobque timeline viewer per its own claim. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017vC4MHGzEXZtj4N7MaXxdE
Cross-context captures in launch_start now clone_string explicitly; byte scans compare int chars directly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017vC4MHGzEXZtj4N7MaXxdE
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an in-repo ImGui-based “jobque lane timeline” viewer under utils/jobque-timeline/, with loaders/LOD rendering/stats/live-commands and a subprocess “launch rail”, plus a small fio runtime addition to support environment-variable control for launched children. Also includes a blog post announcing the tool.
Changes:
- Introduces
utils/jobque-timeline/(trace model + fast loader + mip/LOD rendering + stats + launch/file-watch rail + main UI + docs). - Adds unit tests for loader/stats/launch helpers under the tool directory.
- Adds
set_env_variablefio builtin (C++ + AOT header) used by the launch rail; adds the accompanying blog post.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| utils/jobque-timeline/tl_view.das | ImGui canvas timeline rendering with cached draw runs + overview strip + input handling. |
| utils/jobque-timeline/tl_stats.das | Computes cached “range stats” (busy/idle/parallelism + per-category shares). |
| utils/jobque-timeline/tl_model.das | Defines trace document schema (events/lanes/categories/markers + meta packing helpers). |
| utils/jobque-timeline/tl_lod.das | Builds per-lane mip bucket levels and selects LOD level per zoom. |
| utils/jobque-timeline/tl_loader.das | Fast rigid-format parser for traceSave JSON + optional jobqueProfile trailer parsing. |
| utils/jobque-timeline/tl_launch.das | Launch rail for subprocess execution + argv parsing + debounced file watch. |
| utils/jobque-timeline/test_tl_stats.das | Tests for range-stat math, edge clipping, and stats cache freshness. |
| utils/jobque-timeline/test_tl_loader.das | Tests loader behavior (new/old format, sorting, chain resolution, markers). |
| utils/jobque-timeline/test_tl_launch.das | Tests argv splitting, JSON-path extraction, and file-watch debounce logic. |
| utils/jobque-timeline/README.md | Setup/run instructions and interaction/live-command documentation. |
| utils/jobque-timeline/main.das | Viewer app: dock layout, multi-file windows, live commands, launch panel, auto-refresh. |
| utils/jobque-timeline/.das_package | DASPkg manifest for the tool (depends on dasImgui). |
| src/builtin/module_builtin_fio.cpp | Adds set_env_variable builtin implementation and binding. |
| include/daScript/simulate/aot_builtin_fio.h | Exposes set_env_variable for AOT builds. |
| site/blog/_posts/stories.md | Blog post announcing/mentioning the tool. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| let lvl & = unsafe(lane.mips[lvl_i]) | ||
| let bw = span / double(lvl.nb) | ||
| let b0 = clamp(int(cache.t0 / bw), 0, lvl.nb - 1) | ||
| let b1 = clamp(int(cache.t1 / bw) + 1, 0, lvl.nb - 1) |
| } | ||
| visible++ | ||
| let kind = ev_kind(e.meta) | ||
| if (kind == EV_CHUNK || kind == EV_FIFO) { |
| name : string // e.g. "token", "layer" | ||
| ts : array<double> // sorted µs; unit i spans [ts[i], ts[i+1]) | ||
| args : array<int> | ||
| lane : array<int8> |
| let mi = marker_kind_index(doc, mname) | ||
| doc.marker_kinds[mi].ts |> push(ts) | ||
| doc.marker_kinds[mi].args |> push(arg) | ||
| doc.marker_kinds[mi].lane |> push(int8(tid)) |
| Tell your own stories. I'll tell mine. | ||
|
|
||
| Here is the fun part. I'm not a doctor, Carl. I'm in the business of making programming language. Just one too (hi Andrei). | ||
| Nothing tests programming languages like scientific research on frontier performance critical applications in heterogenic environment, a mouthful. |
|
|
||
| Then there was a jobque. Did u know that jobque bites? I thought I had it good. I fixed it. I thought I had it decent. I fixed it. I thought it might do. I had to add team mode, spin-to-win, limits. It hardly holds now. I might have a few more rounds in me. There is now a tool - u can see what went in when - PS2 A-Probe style, GPAD style, beyond debugging... nah, that last one is too awkward unless u were on ice during the PS3 launch. Same flashback. Same bar graphs. Same job manager struggles. Way better latency way back then. | ||
|
|
||
| Of cause, there is also first part of this story. Which I skipped. For now...(strudel, evil chords, empire strikes back). Not today, really, not today. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The jobque lane-timeline viewer as an in-repo ImGui tool, plus the blog post that announces it.
utils/jobque-timeline
Per-lane jobque trace viewer — the in-repo successor to the hand-spliced lane-timeline browser artifact. Opens
jobque_trace_save/daslib/jobque_profileJSON (perfetto-compatible) in dockable ImGui windows: one row per lane, events colored by category with per-stage shading, unit markers (token/layer/step), computed stats (idle %, parallelism, busy-by-category), multi-file compare with shared zoom/timer, and a launch rail that reruns the trace-producing app and auto-refreshes on save. Needs dasImgui viadaspkg install --root utils/jobque-timeline; runs standalone or underdaslang-live.Small runtime addition: one fio builtin (+15 lines) the launch rail uses.
Blog
site/blog/_posts/stories.md— "Tell me a story." Lands with the tool per the post's own claim ("There is now a tool - u can see what went in when").🤖 Generated with Claude Code
https://claude.ai/code/session_017vC4MHGzEXZtj4N7MaXxdE