Skip to content

Add Jarvis mode: Sonnet 4.6 + double-clap status briefing#5

Open
arinunezm wants to merge 1 commit into
mediar-ai:mainfrom
arinunezm:feature/jarvis-clap-sonnet
Open

Add Jarvis mode: Sonnet 4.6 + double-clap status briefing#5
arinunezm wants to merge 1 commit into
mediar-ai:mainfrom
arinunezm:feature/jarvis-clap-sonnet

Conversation

@arinunezm

Copy link
Copy Markdown

Summary

  • Switch to claude-sonnet-4-6 — all three warmup sessions (main, floating, observer) moved from Opus 4.6, reducing token costs while keeping full capability
  • ClapDetector.swift — new background service that listens to the mic at near-zero CPU cost (RMS energy only, no transcription). Detects two sharp transients within 600ms as a double-clap
  • Status briefing flow — double-clap → ChatProvider.sendStatusBriefing() → Claude (floating session) generates a concise spoken-style status update → speak_response tool reads it aloud via DeepGram TTS
  • PushToTalkManager integration — ClapDetector starts automatically on app launch alongside the existing PTT listener; only fires when PTT is idle

How it works

User claps twice → ClapDetector detects RMS spike pattern
  → sendStatusBriefing() sends query to floating Claude session
  → Claude responds with ≤5-sentence spoken summary
  → speak_response tool reads it via DeepGram Aura TTS

Detection algorithm

For each ~100ms audio chunk (16kHz PCM):
  1. Compute RMS
  2. onset = RMS > 0.12 AND previousRMS < 0.03 (quiet→loud transient)
  3. Two onsets within 600ms → DOUBLE CLAP
  4. 2.5s cooldown prevents re-trigger

Test plan

  • Build with ./run.sh — confirm logs show claude-sonnet-4-6 at warmup
  • Confirm ClapDetector: background mic listening started in /private/tmp/fazm-dev.log
  • Clap twice near the mic — confirm DOUBLE CLAP detected! in logs
  • Confirm floating bar opens and TTS reads a status briefing
  • Verify PTT (Option key) still works normally after ClapDetector starts
  • Confirm ClapDetector does NOT fire while PTT is actively listening

🤖 Generated with Claude Code

- Switch all three Claude sessions (main, floating, observer) from
  claude-opus-4-6 to claude-sonnet-4-6 for cost efficiency
- Add ClapDetector.swift: background mic listener that detects two
  sharp transients (RMS > 0.12 from quiet < 0.03) within 600ms
- Add ChatPrompts.statusBriefing: concise spoken-style status prompt
  that instructs Claude to use speak_response tool automatically
- Add ChatProvider.sendStatusBriefing(): sends the briefing query to
  the floating session with the status prompt as a suffix
- Wire ClapDetector into PushToTalkManager.setup() so it starts on
  app launch; on double-clap fires sendStatusBriefing() when idle

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@m13v m13v left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @arinunezm, thanks for the contribution! Cool concept. A few notes:

Model switch (Opus → Sonnet): We actually already made this change across the codebase (Apr 5-13) in ShortcutSettings, acp-bridge, ChatProvider, and DetachedChatWindow. So this part of the PR is redundant and would conflict with current main.

Clap detector feedback:

The idea is fun, but the current implementation has some significant reliability concerns:

  1. No spectral analysis. The detector uses pure RMS energy detection, which means it can't distinguish a clap from typing, coughing, a door closing, snare hits in music, or plosive consonants in speech. Claps have a specific broadband frequency signature that could be detected with even a basic FFT, but raw energy alone will produce a lot of false positives in real environments.

  2. Hardcoded thresholds. The 0.12 clap / 0.03 quiet thresholds are fixed with no calibration. Different microphones have different gain levels, and a quiet home office vs. a coffee shop will have completely different RMS baselines. There's no adaptive noise floor or startup calibration.

  3. Always-on by default. The detector starts listening on app launch with no user opt-in. For a feature that uses the microphone continuously, users should explicitly enable it.

  4. Environment dependency. Without adaptive thresholding, this will work great in one specific room and fail everywhere else.

What would help move this forward:

Before we can evaluate merging this, it'd be great to hear more from you on:

  • What are the concrete use cases you're solving? When would someone want a status briefing via double-clap vs. just asking Fazm directly?
  • How do you envision handling the false positive problem in practice?
  • Would you be open to making this an opt-in feature with a calibration step?

The core infrastructure (mic access, PTT integration) is solid, but the detection algorithm needs more work to be production-ready. Happy to discuss further!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants