[RFC] Handy UI Overhaul

[cjpais]

As Handy has grown, more and more features have been added. Especially to the debug menu, and some are mature enough to bring in more generally. But doing this impacts the rest of the UI and I think it will start to feel cluttered or it just needs some reorganization to best fit the app.

Features I'm considering moving out of debug based on user feedback:

• Sound Theme
• Mute While Recording
• Append Trailing Space
• History Limit/Recording Retention (I also think we need to clean up these settings first before doing a move on them)
• Cancel

In addition I think there are some other pain points in the UI which should be addressed in some way.

Models
I think largely we have outgrown the pop-up model selector in the bottom left corner. I like the UI a lot in some ways, but there is some awkwardness. There's a few issues

1. People don't know if the model they are using will transcribe their language. There is no indication of what languages are supported. I'm not sure the best way of displaying this information.
2. Different models have different parameters. The main example is that Parakeet does not allow you to select the transcription language. Similarly translation is also only supported by Whisper. This really needs to be more obvious I think.

What is the best solution? I'm not sure. Would love any feedback and discussion on this point.

Onboarding

I think the onboarding right now is simple enough, but it might be helpful to add a few things while onboarding.

• Selecting the UI language
• Guiding the person through their first transcription with Handy.
• Setting up the most basic settings (ptt, audio devices, and comments around recommendations and limitations)
  • potentially history retention periods as well?

Internationalization
Internationalization is another one where we could improve the UI for selecting the language (see #437)

Maybe it makes sense to have a globe icon > English dropdown or similar somewhere. Potentially replacing the updates available bit? Or maybe it's in a consistent spot in the UI? I'm not really sure but it could use some help.

^ I would love any feedback on how we can do this and have a really beautiful consistent UI

Design System

I think the design system worked quite well when Handy was small, but could use a bit of overhaul to clean it up and make it feel more polished. I think there are some inconsistencies in the UI that would be best to make nicer. Open to feedback here. Would love any screenshots/figma files from designers who are willing to help

Custom Shortcuts

It's obvious many people want custom keyboard shortcuts. We should support this at some point in time, but we need GREAT UI and UX to bring it in. I would love discussion to keep this moving forward.

History

History is an interesting one. There are a few things that would be nice to do here.

• Maybe colocating the settings for history in this pane?
• Considering if transcribing an audio file directly should happen in this pane?
  • or is this it's own side menu item? I really want this feature for myself, and actually with post-processing as well. I just don't really know the best UI for this. Especially as it starts to become like a 'workflow', which is out of scope for Handy at the moment. I mean maybe it's something we support in the future but I am quite skeptical of doing this directly in Handy. I think it would be best to extract the libraries and patterns out of Handy to enable another app to do these things better. But at the bare minimum I think we should have transcribing files in the app, even if it's a slim and limited implementation
• History also has the potential to contribute to building open datasets to improve STT models generally. I think it would be really cool if as a community we could build datasets together to improve these models, and make them more ubiquitous and accurate across all devices with small resource constraints.

Menu Reorganization

@dannysmith suggested a re-organization of the sidebar menu. I also think this needs to happen. He suggested

Transcription (incl some of the advanced stuff)
Post-Processing
Interface (incl overlay position, UI language)
History
Debug
About

I've also considered something like

Keybindings
Audio
History
Models
Advanced
Debug
About

I really don't know the best organization for this, so any feedback is appreciated. Especially considering how the app might grow.

Wrapping Up
I guess this turned more into open questions for Handy and maybe roadmap to some extent. Perhaps we use this thread as a jumping off point into other discussions. @dannysmith if you have any feedback, or ways to organize this, would love to know

[VirenMohindra]

thanks for putting this together! here are my unsolicited thoughts on each section :)

debug menu to main UI promotion

agree that most of the current debug options are actually user-facing and shouldn't be hidden. things like auto-update checks, sound theme, word correction threshold, and history limit aren't developer-specific at all. the only truly dev-relevant item is logs, and even that needs to remain easily accessible for users opening issues. maybe we keep a minimal "debug" or "advanced" section just for logs and any future dev tooling

model selection

i think there's a better way to surface models than a dropdown. remember the onboarding experience where we showed models with their names, descriptions, and what they excelled at? that was actually a really intuitive approach. could we possibly reuse that existing page / component as the model selection interface?

additionally, similar to how we now filter languages based on OS locale for i18n, we could recommend or highlight models based on the user's system locale. this would help address the "will this model transcribe my language" confusion

onboarding

hot take but i actually like the current onboarding - it's quick, painless, and simple. if we add ui language selection, let's stick with sane defaults (detect from system locale, let user override)

the guided first transcription idea is neat. one small addition: maybe a text field during onboarding, where users need to use Handy to proceed might help alleviate some of the friction

basic settings

sane defaults everywhere. for any advanced configuration, i'd lean toward the Apple-style approach: show the recommended defaults prominently, with an "advanced settings" toggle / section for power users who want to tweak things

internationalization

no major concerns here. the dropdown approach should work fine, and the recent tray refresh changes should automatically apply locale updates. again, sane defaults based on system locale

design system

i have a lot of thoughts here and think there's significant opportunity to leverage existing typescript frontend patterns to make this more polished and accessible. happy to scope out my initial ideas in a separate POC or discussion thread rather than cluttering this one

custom shortcuts / history

no immediate thoughts on these specifically, but +1 on the menu reorganization being necessary

sample menu structure

i like the "advanced" menu concept. keeps the main interface clean while still providing access to power-user options

[dannysmith]

OK quick thoughts on this from me...

1. debug menu to main UI promotion – I agree. I feel like the debug section should basically have two groups of stuff:

• Developer - actual debug/developer stuff
• Experimental Features - stuff we're trying which should eventually be promoted into somewhere else in the settings. The advantage to keeping this here rather than Advanced is that it won't be visible to users who haven't switched on debug: true.

2. Model Selection - Fully agree.

3. Onboarding - I also like the current onboarding flow (over-the-top onboarding is frustrating for many "power users" and Handy is not conceptually complex for non-"power users"). I'd always favour good documentation on the website (maybe with a few short videos) over adding code to walk users through stuff. At least until there's really clear data proving the docs aren't enough.

4. basic settings - Agree

5. i18n - Agree

6. design system - Agree. We should 💯 have a decent, reusable system of design tokens and reusable UI components, while keeping it as light as possible (because Handy's UI is minimal).

[cjpais]

from another thread for debug menu

Ooh: also regarding the debug menu, I'm wondering at what point there is a 'power-user' toggle or similar which enables an 'experimental' tab instead of debug. Where debug really gets left to debug features. And experimental more cleanly is a collection of random stuff. I'm almost okay having experimental be enabled by default as well, however I know some apps like LMStudio let you choose your level (User, Power User, Developer) which might be appropriate here as well.

3. Cool this is nice to hear

[VirenMohindra]

following up on my promise to scope out design system thoughts. did a deep dive into the fe architecture and had a couple thoughts

the foundation is solid. we have~

• 16 reusable ui primitives in components/ui/
• no heavy ui framework dependencies
• type-safe bindings from rust via tauri-specta

where we can improve

1. settings container complexity
   SettingContainer is 199 lines handling 4 layout modes (stacked, horizontal, tooltip, inline). this makes it hard to extend and easy to break and the proposal here is to decompose into smaller primitives instead of one god component

2. model registry
   model ids are hardcoded across multiple files. we should have a centralized model registry somewhere in lib/constants/models.ts with metadata (supported languages, recommended use cases, VRAM requirements). this would also enable the locale-based model recommendations i mentioned earlier

3. store architecture

settingsStore.ts is 500 lines handling 50+ settings, audio devices, post-process caching, and loading states. this will get worse as we add features like custom keyboard shortcuts and we should split into domain stores~

• useAudioStore - devices, mute state
• useShortcutsStore - bindings, conflict detection
• useSettingsStore - general settings (like a slimmed down version)

4. validation layer

we have zod but its underutilized. user input goes straight to rust without any fe validation. nitty but we should add validation schemas, especially important for custom shortcuts like conflict detection, reserved keys

5. event listener cleanup

useModels hook has 5 chained .then() calls for cleanup. we should have a unified useTauriEvents hook~

  useTauriEvents({
    'download-progress': handleProgress,
    'model-loaded': handleLoaded,
  });

6. other misc lower priority stuff

• react error boundaries (one uncaught error can crash the whole ui)
• component documentation/storybook
• test coverage for ui components

happy to put together a more detailed implementation plan or start with a POC on any of these. the model registry extraction is probably the easiest win and would directly support the locale-based model recommendations we discussed

[dannysmith]

Thanks for doing this thinking @VirenMohindra ❤️

component documentation/storybook

Storybook feels like overkill until we have more complex reusable UI components, but agree with docs of some sort (either inline or as MD)

[dannysmith]

I agree with everything else here tho 🙏

[dannysmith]

@VirenMohindra I'm keen to help with this (even if just feedback or bouncing ideas) - tag me in any issues/PRs related to this if ya want?

[cjpais]

@VirenMohindra this is a great summary of the frontend and captures a lot of the issues. I think you pretty much nailed it, and I think these are all fixes and refactors I look forward to. Would love any help in them.

The model registry is somewhere I've looked at in particular as well. Curious to see any refactor of it. I attempted a json based refactor of the model registry system a while back and scrapped it due to it feeling a little overly complex with little gain. The main benefit of it would be that someone could add their own models outside of the UI. But I'm not super concerned with that at the moment, especially if the UI for the model selector is better, and maybe gives power users more options.

@dannysmith also very interested to hear your opinion how to organize this on GitHub if possible to keep everything transparent and open. It feels like maybe these need to be issues/discussions/or some other way of tracking? I'm okay right now just using this thread, but I feel given the changes it might be good to have a solid dedicated place too

[dannysmith]

Part of this might be easier to do after I've got around to parts of #409 , and other parts of 409 will be easier with this work implemented. So we should probably try do do these in a sensible order (easier said than done! 😂)

[dannysmith]

@cjpais Perhaps we should start a GH Project to plan work on this and #409 (and any other architecture-ish work which cuts across multiple files and would benefit from being done/merged in some intentional order)?

I'd be happy to manage that – I think you can grant permissions to Projects without giving me anything else.

Or is that overkill? Thoughts @VirenMohindra ?

[cjpais]

Added you to the repo, I think I can give permissions after you accept

Sidenote: If you want I can probably start tackling the MacOS rewrite bit of the keyboard handling fairly soon I think.

[dannysmith]

Also: when it comes to layout and the CSS for UI primitives and layout, we should probably take this opportunity to ensure everything is robust enough to gracefully handle i18n-related stuff like unexpectedly long/short translations, rtl languages and the like.

[cjpais]

I think this would be good to do as well. In German for example post-processing runs off the sidebar which is not great.

[dannysmith]

Lol, I almost wrote "unexpectedly long (ie. German 😂") in that message.