fix(test-classifier): --submit prompt dropped a typed answer (type-ahead) - #66
Merged
Conversation
…ead)
Symptom: after a long OBSERVED run, answering 'y' at 'Was the classification
helpful?' still printed 'no answer — skipping the metrics row'. The comment
posted but the Testing Events row was dropped.
Cause: the prompt used a bare `read -r answer` on fd 0. During the multi-second
streamed suite run, stray newlines/keystrokes (and the agent's own Bash tool
calls perturbing the TTY) queue in the terminal input buffer. `read` consumed
that buffered line — usually empty → the `*)` 'no answer' branch — while the
'y' typed AT the prompt was handed back to the shell after the script exited
(hence 'y' echoed but never seen).
Fix:
• open the controlling terminal ONCE on fd 3 and read from it throughout, so
successive reads (answer, then the 👎 reason) advance the same stream rather
than re-opening /dev/tty and re-grabbing the first line;
• flush pending type-ahead (read -t 0) before prompting, so only a fresh
keypress is read;
• re-prompt up to 3× on unrecognized input instead of silently skipping — a
stray char shouldn't discard the row after a full suite run; an explicit
empty Enter still skips.
Tested the loop logic (single-fd harness): plain y → 👍; stray char then y →
re-prompts and recovers; n + reason → 👎 with the full reason string (previously
captured 'n'); empty Enter → skip.
fg-nava
added a commit
that referenced
this pull request
Jun 24, 2026
…y, not fd 0 (#67) Follow-up to #66. The helpfulness prompt reads from /dev/tty, but the early 'non-interactive' guard still tested fd 0 ([[ ! -t 0 ]]). During an OBSERVED run the agent's Bash tool subprocesses (pnpm install, vitest, …) inherit the dispatcher's fd 0 as their stdin and can leave it non-TTY / consumed-to-EOF by the time --submit runs. The guard then reports 'non-interactive', returns early, and the prompt is skipped silently — the comment posts but no 'Was this helpful?' question appears and no Testing Events row is written. (Observed: a --submit run ended at 'Comment posted … Awaiting the developer's reaction' with no prompt.) Fix: gate on the SAME resource the prompt uses — whether /dev/tty can be opened — instead of fd 0. CI / real non-TTY runs have no controlling terminal so they still skip correctly. The probe ': <>/dev/tty' is set -e-safe inside the if.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Symptom
In your live run: a full OBSERVED classification posted the comment to
navapbc/ai-chatbot#266correctly (the #64 fix works ✓), then asked 'Was the classification helpful?', you typedy, and it printed:The 👍 was lost; no Testing Events row.
Cause — type-ahead buffering
The prompt used a bare
read -r answeron fd 0. During the multi-second streamed suite run, any stray newline/keystroke (and the agent's ownBashtool calls touching the TTY) queues in the terminal input buffer.readconsumed that buffered line — usually empty, hitting the*)'no answer' branch — while theyyou typed at the prompt was handed back to the shell after the script exited. That's whyyechoed but the script never saw it.Fix
/dev/ttyper read restarts it and re-grabs the first line — that also fixes a latent bug where the 👎 reason capturedninstead of the typed reason).read -t 0) before prompting, so only a fresh keypress is read.Tested
Single-fd harness across input cases: plain
y→ 👍; stray char theny→ re-prompts and recovers to 👍;n+ reason → 👎 with the full reason string (previouslyn); empty Enter → skip.bash -nclean.Note: the type-ahead flush can only be fully exercised on a real TTY (a piped file can't replicate terminal buffer timing), so the flush behavior is verified by construction, not by the harness.
Independent of the smolvm PR (#65); branches off main.