Skip to content

Commit 55c1e4b

Browse files
authored
fix(test-classifier): --submit prompt skipped a typed answer (type-ahead) (#66)
Symptom: after a long OBSERVED run, answering 'y' at 'Was the classification helpful?' still printed 'no answer — skipping the metrics row'. The comment posted but the Testing Events row was dropped. Cause: the prompt used a bare `read -r answer` on fd 0. During the multi-second streamed suite run, stray newlines/keystrokes (and the agent's own Bash tool calls perturbing the TTY) queue in the terminal input buffer. `read` consumed that buffered line — usually empty → the `*)` 'no answer' branch — while the 'y' typed AT the prompt was handed back to the shell after the script exited (hence 'y' echoed but never seen). Fix: • open the controlling terminal ONCE on fd 3 and read from it throughout, so successive reads (answer, then the 👎 reason) advance the same stream rather than re-opening /dev/tty and re-grabbing the first line; • flush pending type-ahead (read -t 0) before prompting, so only a fresh keypress is read; • re-prompt up to 3× on unrecognized input instead of silently skipping — a stray char shouldn't discard the row after a full suite run; an explicit empty Enter still skips. Tested the loop logic (single-fd harness): plain y → 👍; stray char then y → re-prompts and recovers; n + reason → 👎 with the full reason string (previously captured 'n'); empty Enter → skip.
1 parent 5c6512e commit 55c1e4b

1 file changed

Lines changed: 57 additions & 15 deletions

File tree

testing/classifier/.skills/test-classifier/scripts/test-classifier-dispatcher.sh

Lines changed: 57 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -755,22 +755,64 @@ submit_metrics_row() {
755755
return 0
756756
fi
757757

758-
# Prompt for the tuning signal. Empty/invalid → skip (don't guess a verdict).
758+
# Prompt for the tuning signal, reading from the controlling terminal.
759+
#
760+
# We MUST read from /dev/tty, not fd 0. During the long OBSERVED run the agent
761+
# spawns Bash tool calls and the stream scrolls for many seconds; any keystroke
762+
# or stray newline that arrives in that window sits in the TTY's input buffer.
763+
# A bare `read -r answer` would consume that buffered line (often empty) instead
764+
# of the answer typed AT this prompt — the user types `y`, but `read` already
765+
# grabbed a queued blank line and skipped. So:
766+
# 1. flush pending type-ahead before prompting (so we only see a fresh keypress);
767+
# 2. read from /dev/tty explicitly;
768+
# 3. re-prompt on empty/unrecognized input rather than silently skipping — a
769+
# stray keystroke shouldn't discard the row after a full suite run. Enter
770+
# on an empty line (after the flush) is the explicit skip.
759771
local answer reason="" thumbs_up=0 thumbs_down=0
760-
printf '%s' " Was the classification helpful? [y/n] (enter to skip): " >&2
761-
read -r answer
762-
case "${answer}" in
763-
y|Y|yes|YES) thumbs_up=1 ;;
764-
n|N|no|NO)
765-
thumbs_down=1
766-
printf '%s' " Optional one-line reason (enter to skip): " >&2
767-
read -r reason
768-
;;
769-
*)
770-
ai_review::info "--submit: no answer — skipping the metrics row (comment still posted)." >&2
771-
return 0
772-
;;
773-
esac
772+
# Open the controlling terminal ONCE on fd 3 and read from that fd throughout,
773+
# so successive reads advance through the SAME stream (re-opening /dev/tty per
774+
# read would restart it and re-grab the first line). Closed on return.
775+
if ! exec 3<>/dev/tty 2>/dev/null; then
776+
ai_review::info "--submit: no controlling terminal — skipping the helpfulness prompt + row (comment still posted)." >&2
777+
return 0
778+
fi
779+
780+
# Flush any buffered type-ahead so the prompt sees only a fresh keypress, not a
781+
# stray newline/char queued during the long suite run (which would otherwise be
782+
# consumed as the "answer" and skip the row). read -t 0 succeeds iff input waits.
783+
while read -r -t 0 -u 3 _flush 2>/dev/null; do :; done
784+
785+
local attempt
786+
for attempt in 1 2 3; do
787+
printf '%s' " Was the classification helpful? [y/n] (enter to skip): " >&2
788+
if ! read -r -u 3 answer; then
789+
ai_review::info "--submit: input closed — skipping the metrics row (comment still posted)." >&2
790+
exec 3<&- 3>&-; return 0
791+
fi
792+
case "${answer}" in
793+
y|Y|yes|YES) thumbs_up=1; break ;;
794+
n|N|no|NO)
795+
thumbs_down=1
796+
printf '%s' " Optional one-line reason (enter to skip): " >&2
797+
read -r -u 3 reason || reason=""
798+
break
799+
;;
800+
"")
801+
# Deliberate empty Enter = skip.
802+
ai_review::info "--submit: skipped (no answer) — comment still posted." >&2
803+
exec 3<&- 3>&-; return 0
804+
;;
805+
*)
806+
# Unrecognized (e.g. a stray char) — re-prompt instead of discarding.
807+
ai_review::warn "--submit: please answer y or n (or press enter to skip)." >&2
808+
;;
809+
esac
810+
done
811+
exec 3<&- 3>&- # close the tty fd
812+
if (( thumbs_up == 0 && thumbs_down == 0 )); then
813+
ai_review::info "--submit: no valid answer after 3 tries — skipping the row (comment still posted)." >&2
814+
return 0
815+
fi
774816

775817
local webhook_url="${METRICSAI_WEBHOOK_URL:-}"
776818
local webhook_key="${METRICSAI_WEBHOOK_KEY:-}"

0 commit comments

Comments
 (0)