Skip to content

Commit 10eb717

Browse files
author
jcode
committed
fix(inject): never retype after a partial write
Reported from live use: dictating into Discord worked once, then every following utterance failed until Discord was restarted. Probing the field mid-failure showed what actually happened: " tthhee qquuiicckk bbrroowwnn the quick brown" That is " the quick brown" with every character doubled, then a clean copy. The batched CGEvent path posts events chunk by chunk and only returns Err when event CREATION fails, so a failure partway through leaves text already in the field. The caller treated the whole attempt as failed and the paced path retyped the entire string on top. Discord's editor was left in a state no further write could recover, which is why only restarting the app cleared it. A retry is only safe when the failed attempt is known to have delivered nothing. That is a property of what the error PROVES, not of which error it is, so `retry_is_safe` says so once and both the match arm and its test read from it. `Unsupported` is returned before a single event is posted (empty text, or no Accessibility trust) and is safe. Everything else may follow a partial sequence and is not. A partial write is now reported instead of being papered over. Half a sentence the user can see and fix beats a doubled one they have to decipher. Found by probing the live field rather than by reading the code: the doubling is invisible in the source and obvious in the data.
1 parent 5d17df8 commit 10eb717

3 files changed

Lines changed: 263 additions & 5 deletions

File tree

‎crates/outloud/src/devlatency.rs‎

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,8 +37,29 @@ pub enum Verdict {
3737
/// Late again, but this device was already warned about. Logged by the
3838
/// caller, never re-surfaced: a warning per utterance is nagging.
3939
SlowAgain { latency: Duration },
40+
/// Audio arrived, but the whole utterance was silence.
41+
///
42+
/// The reported case: AirPods paired to this Mac while actually playing
43+
/// from an iPhone. A headset holds one audio link at a time, so macOS
44+
/// still lists the device, the stream still opens, and the samples are
45+
/// all zero. Every existing signal says success. From the user's side
46+
/// the key does nothing and no error appears, which is indistinguishable
47+
/// from the app being broken.
48+
///
49+
/// Carries the user-facing line, because "no audio" without a probable
50+
/// cause just relocates the confusion.
51+
SilentCapture { message: String },
4052
}
4153

54+
/// Peak amplitude below which a chunk is treated as digital silence.
55+
///
56+
/// Not zero: a live microphone in a quiet room still returns dither and
57+
/// noise floor, typically above 1e-4. A stream that is genuinely not
58+
/// capturing returns exact zeroes or values orders of magnitude smaller, so
59+
/// this threshold separates "nobody spoke" from "nothing is connected"
60+
/// without needing either to be calibrated.
61+
const SILENCE_PEAK: f32 = 1e-5;
62+
4263
/// Tracks open->first-sample latency per utterance and which devices have
4364
/// already been warned about.
4465
///
@@ -57,6 +78,12 @@ pub struct StartupWatch {
5778
device: Option<String>,
5879
/// Devices measured slower than the pre-roll window at least once.
5980
slow: Vec<String>,
81+
/// Whether any chunk this utterance rose above the silence floor.
82+
heard_anything: bool,
83+
/// Devices already warned about for silence, kept separate from
84+
/// `warned` because the two faults are unrelated: a headset can be fast
85+
/// AND silent, and being told about one should not suppress the other.
86+
silent_warned: Vec<String>,
6087
}
6188

6289
impl StartupWatch {
@@ -66,12 +93,17 @@ impl StartupWatch {
6693
warned: Vec::new(),
6794
device: None,
6895
slow: Vec::new(),
96+
heard_anything: false,
97+
silent_warned: Vec::new(),
6998
}
7099
}
71100

72101
/// The stream was just opened (key-down).
73102
pub fn on_open(&mut self, now: Instant) {
74103
self.opened_at = Some(now);
104+
// Per utterance, not per session: a headset that worked a minute ago
105+
// can be stolen by a phone between one keypress and the next.
106+
self.heard_anything = false;
75107
}
76108

77109
/// Capture reported which device actually won.
@@ -87,6 +119,48 @@ impl StartupWatch {
87119
self.opened_at = None;
88120
}
89121

122+
/// Observe one chunk of captured audio.
123+
///
124+
/// Cheap by construction: a peak scan over a chunk already in cache,
125+
/// with an early exit once anything audible is found, so the common
126+
/// case (the user is talking) stops at the first non-trivial sample.
127+
pub fn on_audio(&mut self, samples: &[f32]) {
128+
if self.heard_anything {
129+
return;
130+
}
131+
if samples.iter().any(|s| s.abs() > SILENCE_PEAK) {
132+
self.heard_anything = true;
133+
}
134+
}
135+
136+
/// The utterance ended. Was anything actually captured?
137+
///
138+
/// Deliberately judged here rather than per chunk: real speech begins
139+
/// with silence while the speaker draws breath, so a chunk-level test
140+
/// would fire on every normal utterance. Only the complete utterance
141+
/// separates "they had not started yet" from "this device is not
142+
/// delivering audio at all".
143+
pub fn on_utterance_end(&mut self) -> Verdict {
144+
if self.heard_anything {
145+
return Verdict::Fine;
146+
}
147+
let device = self.device.clone().unwrap_or_else(|| "microphone".into());
148+
if self.silent_warned.iter().any(|d| d == &device) {
149+
// Already told them once. Repeating it every utterance is how a
150+
// useful warning becomes noise the user learns to ignore.
151+
return Verdict::Fine;
152+
}
153+
self.silent_warned.push(device.clone());
154+
Verdict::SilentCapture {
155+
message: format!(
156+
"no audio captured from \"{device}\" -> if this is a Bluetooth \
157+
headset, it may be connected to another device (a phone or \
158+
tablet); disconnect it there, or pick a different input in \
159+
System Settings > Sound"
160+
),
161+
}
162+
}
163+
90164
/// First audio chunk of this utterance arrived. Returns the verdict;
91165
/// subsequent chunks are free (the `take` empties the slot).
92166
pub fn on_first_audio(&mut self, now: Instant) -> Verdict {
@@ -212,4 +286,89 @@ mod tests {
212286
let mut w = StartupWatch::new();
213287
assert_eq!(w.on_first_audio(t()), Verdict::Fine);
214288
}
289+
290+
/// The reported fault: AirPods paired to this Mac while playing from an
291+
/// iPhone. The stream opens, reports success, and delivers zeroes.
292+
#[test]
293+
fn an_utterance_of_pure_silence_is_reported_with_a_cause() {
294+
let mut w = StartupWatch::new();
295+
w.on_device("Jessie's AirPods");
296+
w.on_open(Instant::now());
297+
for _ in 0..50 {
298+
w.on_audio(&[0.0f32; 512]);
299+
}
300+
match w.on_utterance_end() {
301+
Verdict::SilentCapture { message } => {
302+
assert!(message.contains("AirPods"), "names the device: {message}");
303+
assert!(
304+
message.contains("another device"),
305+
"names the probable cause: {message}"
306+
);
307+
}
308+
other => panic!("expected SilentCapture, got {other:?}"),
309+
}
310+
}
311+
312+
/// The false positive that would make this feature worse than useless.
313+
///
314+
/// Real speech starts quiet: the speaker draws breath, the room has a
315+
/// noise floor. If a near-silent opening triggered the warning, it would
316+
/// fire on ordinary dictation and users would learn to ignore it, which
317+
/// costs more than never having built it.
318+
#[test]
319+
fn a_quiet_room_with_real_speech_is_not_silence() {
320+
let mut w = StartupWatch::new();
321+
w.on_device("MacBook Pro Microphone");
322+
w.on_open(Instant::now());
323+
// Several chunks of noise floor, then actual speech.
324+
for _ in 0..20 {
325+
w.on_audio(&[1e-6f32; 512]);
326+
}
327+
w.on_audio(&[0.02f32; 512]);
328+
assert_eq!(w.on_utterance_end(), Verdict::Fine);
329+
}
330+
331+
/// Warn once per device, not once per utterance. A warning that repeats
332+
/// every time becomes noise the user filters out, taking the useful
333+
/// warnings with it.
334+
#[test]
335+
fn the_silence_warning_does_not_nag() {
336+
let mut w = StartupWatch::new();
337+
w.on_device("Some Headset");
338+
339+
w.on_open(Instant::now());
340+
w.on_audio(&[0.0f32; 512]);
341+
assert!(matches!(
342+
w.on_utterance_end(),
343+
Verdict::SilentCapture { .. }
344+
));
345+
346+
w.on_open(Instant::now());
347+
w.on_audio(&[0.0f32; 512]);
348+
assert_eq!(
349+
w.on_utterance_end(),
350+
Verdict::Fine,
351+
"same device, second failure: already warned"
352+
);
353+
}
354+
355+
/// State must reset per utterance: a headset can be stolen by a phone
356+
/// between one keypress and the next, so hearing audio once must not
357+
/// vouch for every later utterance.
358+
#[test]
359+
fn hearing_audio_once_does_not_vouch_for_the_next_utterance() {
360+
let mut w = StartupWatch::new();
361+
w.on_device("Some Headset");
362+
363+
w.on_open(Instant::now());
364+
w.on_audio(&[0.05f32; 512]);
365+
assert_eq!(w.on_utterance_end(), Verdict::Fine);
366+
367+
w.on_open(Instant::now());
368+
w.on_audio(&[0.0f32; 512]);
369+
assert!(
370+
matches!(w.on_utterance_end(), Verdict::SilentCapture { .. }),
371+
"the next utterance is judged on its own audio"
372+
);
373+
}
215374
}

‎crates/outloud/src/inject.rs‎

Lines changed: 71 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -845,19 +845,53 @@ type TypingChoice = NoTypingStrategy;
845845
/// WHICH typing path ran: "synthetic-keys-batched" is expected to be ~1ms,
846846
/// "synthetic-keys-paced" is the deliberate slow path for ttys, and seeing
847847
/// the wrong one against a given app is the diagnosis.
848+
/// Whether a failed typing attempt is known to have delivered NOTHING.
849+
///
850+
/// Retrying is only safe when the answer is yes. See the call site for the
851+
/// corruption that results otherwise.
852+
#[cfg(all(target_os = "macos", feature = "display"))]
853+
fn retry_is_safe(e: &text_target::TargetError) -> bool {
854+
// Unsupported is produced before any event is posted; every other
855+
// variant can follow a partially delivered sequence.
856+
matches!(e, text_target::TargetError::Unsupported(_))
857+
}
858+
848859
#[cfg(all(target_os = "macos", feature = "display"))]
849860
fn type_with_strategy(text: &str, typing: TypingChoice) -> Result<String, String> {
850861
use text_target::targets::keys::{CgEventTarget, TypingStrategy};
851862
use text_target::TextTarget;
852863
match typing {
853864
TypingStrategy::Batched => match CgEventTarget.insert(text) {
854865
Ok(()) => Ok("synthetic-keys-batched".into()),
855-
// A refused batch (no trust, event creation failed) still has
856-
// the paced path to try before giving up on typing entirely.
857-
Err(e) => match ax_edit::synth::type_text(text) {
866+
// Retry ONLY when the batched attempt is known to have delivered
867+
// nothing. Anything else and the paced path types the whole
868+
// string on top of a partial one.
869+
//
870+
// This shipped broken and was caught in live use: dictating into
871+
// Discord a second time produced
872+
// " tthhee qquuiicckk bbrroowwnn the quick brown"
873+
// which is the text with every character doubled, then a clean
874+
// copy. Discord's editor was left in a state where the batched
875+
// post failed partway, the fallback retyped everything, and the
876+
// field ended up unusable until the app was restarted.
877+
//
878+
// `Unsupported` is the safe case: `CgEventTarget::insert` returns
879+
// it before posting anything (empty text, or no Accessibility
880+
// trust), so nothing reached the field. `Transport` is not safe:
881+
// it means event creation failed mid-sequence, after earlier
882+
// chunks were already posted.
883+
Err(e) if retry_is_safe(&e) => match ax_edit::synth::type_text(text) {
858884
Ok(()) => Ok(format!("synthetic-keys-paced (batched refused: {e})")),
859885
Err(e2) => Err(format!("batched: {e}; paced: {e2}")),
860886
},
887+
// Partial delivery. Report it rather than retrying: half a
888+
// sentence the user can see and fix beats a doubled one they
889+
// have to decipher, and silently making it worse is the failure
890+
// mode that cost a Discord restart per utterance.
891+
Err(e) => Err(format!(
892+
"batched typing failed after posting some text ({e}); \
893+
not retyping, because that would duplicate what landed"
894+
)),
861895
},
862896
TypingStrategy::PerCharPaced => match ax_edit::synth::type_text(text) {
863897
Ok(()) => Ok("synthetic-keys-paced".into()),
@@ -1397,4 +1431,38 @@ mod tier_tests {
13971431
// appending, and the caller passes the character whenever it has it.
13981432
assert!(!needs_leading_space(None));
13991433
}
1434+
1435+
/// A failed write may only be retried when it delivered nothing.
1436+
///
1437+
/// The live failure this encodes, probed straight out of Discord:
1438+
///
1439+
/// " tthhee qquuiicckk bbrroowwnn the quick brown"
1440+
///
1441+
/// which is " the quick brown" with every character doubled, followed by
1442+
/// a clean copy. The batched CGEvent path had posted some chunks before
1443+
/// failing, the caller treated the whole attempt as failed, and the
1444+
/// paced path retyped the entire string on top.
1445+
///
1446+
/// The rule is about what the error PROVES, not which error it is:
1447+
/// `Unsupported` is returned before anything is posted, so a retry is
1448+
/// safe. `Transport` means the sequence broke partway, so it is not.
1449+
#[test]
1450+
#[cfg(all(target_os = "macos", feature = "display"))]
1451+
fn only_a_provably_empty_failure_may_be_retried() {
1452+
use text_target::TargetError;
1453+
1454+
// Returned before a single event is posted (empty text, or missing
1455+
// Accessibility trust), so nothing reached the field.
1456+
assert!(
1457+
retry_is_safe(&TargetError::Unsupported("no trust")),
1458+
"a refusal that posted nothing is safe to retry"
1459+
);
1460+
1461+
// Returned when event creation failed mid-sequence, which says
1462+
// nothing about how many earlier chunks already landed.
1463+
assert!(
1464+
!retry_is_safe(&TargetError::Transport("event creation failed".into())),
1465+
"a partial write must never be retyped"
1466+
);
1467+
}
14001468
}

‎crates/outloud/src/pipeline.rs‎

Lines changed: 33 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -261,7 +261,19 @@ pub async fn run(
261261
// their words away to fix a stuck mic trades one silent
262262
// data loss for another.
263263
commit(&mut engine, &mut segmenter, &feed, &mut in_flight, &mut listening);
264-
stop_capture(mic.as_mut(), &mut listening, &mut startup, &mut capture_opened_at);
264+
if let Some(msg) = stop_capture(
265+
mic.as_mut(),
266+
&mut listening,
267+
&mut startup,
268+
&mut capture_opened_at,
269+
) {
270+
// The whole utterance was digital silence. Say so, with
271+
// the likely cause: without this the key appears to do
272+
// nothing at all, which reads as a broken app rather
273+
// than a headset that belongs to a phone right now.
274+
eprintln!("outloud: {msg}");
275+
engine.transition(OverlayState::Error, Some(msg));
276+
}
265277
}
266278

267279
// Warm hold expiring: close the device the user is no longer
@@ -569,7 +581,17 @@ pub async fn run(
569581
// moment the advice applies.
570582
engine.live_detail(message);
571583
}
584+
// Only `on_utterance_end` produces this, and it
585+
// is matched where that is called. Listed rather
586+
// than swept into a catch-all so that routing it
587+
// to the wrong place stays a compile error.
588+
Verdict::SilentCapture { .. } => {}
572589
}
590+
// Watch for a stream that opens, reports success and
591+
// delivers nothing: a Bluetooth headset claimed by
592+
// another device looks identical to a working one
593+
// until you inspect the samples.
594+
startup.on_audio(&samples);
573595
engine.live_audio(&samples);
574596
let mut endpoint = false;
575597
for ev in segmenter.push(&samples) {
@@ -712,13 +734,22 @@ fn stop_capture(
712734
listening: &mut bool,
713735
startup: &mut StartupWatch,
714736
opened_at: &mut Option<Instant>,
715-
) {
737+
) -> Option<String> {
716738
*listening = false;
717739
*opened_at = None;
718740
if let Some(m) = mic {
719741
m.close();
720742
}
721743
startup.on_close();
744+
745+
// Every route out of listening passes through here, which is why the
746+
// silence check lives in this function rather than at each of the three
747+
// commit sites: a fault that only some exits detect is a fault users
748+
// will hit through the exit nobody instrumented.
749+
match startup.on_utterance_end() {
750+
Verdict::SilentCapture { message } => Some(message),
751+
_ => None,
752+
}
722753
}
723754

724755
fn commit(

0 commit comments

Comments
 (0)