Provide feedback for the arrow keys in the Virtual MIDI Keyboard.

[jcsteh]

Fixes #1214.

@ScottChesworth, please take this for a spin and let me know what you think.

[github-actions]

Build 50 succeeded!

• Download for Windows
• Download for Mac

[ScottChesworth]

Trying this build on Windows with NVDA to start with, I don't hear anything reported when hitting arrows in the VKB yet, same with or without Shortcut Help. What did I miss?

[jcsteh]

I... have no idea. It should just work. Does the vkb open as soon as you press alt+b? Does NVDA+tab say "Virtual keyboard keys grouping"?

[ScottChesworth]

Yep to both questions. I've made sure it's not docked and isn't obscured by any other windows or floating thingies, still nada. Weird!

[jcsteh]

Let me know if the next build improves things.

[github-actions]

Build 51 succeeded!

• Download for Windows
• Download for Mac

[ScottChesworth]

That's got it going yep. Lovely stuff! @jennykbrennan can you check this out on Mac and report back?

[jennykbrennan]

Right. Where should I start.
So no. this is all sorts of messed up. Hotkey help gives me the tcp help. Also,up and down arrow throws me out of the vkb. Further, with hotkey help off, octave change doesn't work. The thing is though, that with the latest official release grabbed today, it's equally not functioning.
Clearly I'm not using the vkb enough so that was surprise to me and I have no idea when that would have broken.
Nor have i heard mac folks report anything being messed up this bad. We need additional mac testers on this. Is my outdated MacOs a variable at all? I'm still on a Monterey machine here.

I gave it a spin on windows and all works a charm.
So, let's take a look today, @ScottChesworth to see if you can make more sense out of this than I.
but bottom line is, no go.

[jcsteh]

Damn.

The thing is though, that with the latest official release grabbed today, it's equally not functioning.

Can you clarify which bits are regressed specifically by this PR? That is, in what ways (if any) does this PR make things worse than the release snapshots? I'm wondering whether we can merge this for the benefit of Windows users, given that vkb is clearly very broken on mac already anyway.

[ScottChesworth]

Hmm sounds like REAPER's not supporting arrows the same on Mac doesn't it. Hang on for a while before merging, I'll investigate and ask Cockos to do a fixy if needed. Would be nicer to say "requires version whatever on Mac" than "uh, sorry Mac peeps".

[jennykbrennan]

I'll give it a slightly more scientific test and report back in a few with more detail. An yeah, this might be one of those, sorry Mac folks moments. For a while anyways.

[jcsteh]

And here I was happy that I managed to come up with a way to (supposedly) do this cross-platform without too many horrible hacks. 😢

[jennykbrennan]

Oh, how we love Apple.

[jennykbrennan]

In this PR
Arrow keys left and right moves cursor by pixel and reports that.
Up and down arrows throws me out of vkb into TCP.
With hotkey help on.
reporting as if in tcp. View move cursor, track go to next track and so on.
But notes speak as they should. C4, D4 etc.

With osara_2025.3.14.1846
Behaves in the same way as above, But notes do not report in hotkey help
Whaaaa?

Right, so with my system and how it is, merging would make it plenty better for Windows users, and a bit better for mac users than it currently is.
However. i'd like to know what Reaper says as well as other mac users.
I'm sure speaking notes did work, as we always check these changes on mac. So either I'm going insane or my Mac is gaslighting all of us.

[ScottChesworth]

Are you sure the track was armed every time you tested Shortcut Help in the VKB? That could explain why speaking note names wasn't happening.

[jennykbrennan]

yeah, I did insert virtual instrument on new track, ran reasynth. Definitely armed as I checked again now just in case.

[pitermach]

I'm not seeing what Jenny is experiencing here (Reaper 7.34 on Mac OS 15.4 Beta). With the keyboard help on if I press the arrow keys I hear the announcements about channels and octaves, at least when the keyboard itself has focus. A separate issue which is on Reaper is the keyboard itself is one of those controls that, like a lot of sliders, doesn't send a focus notification when you tab through the dialog instead of using the VO cursor, but the window opens with the keyboard in focus and OSARA is doing what it should at least here.

[jennykbrennan]

Yeah, interesting. What is happening here is very strange and I'm not beyond believing that it's my machine that is acting up. Very strange symptoms though and I can't say what would make it do this. But I'll do a proper check and a restart to see if I can get it to behave. Mostly for my own curiosity now. I really don't think I can trust my machine in this case. It's been overall rather unreliable lately.
I suspect this build is all good to go. and even if it's an OS version problem, that's really a different issue.

[jcsteh]

@ScottChesworth, thoughts on merging?

[ScottChesworth]

Figured out what was up with Jen's settings, can be avoided in future by adding three things to optimal config. I'll send a PR over the weekend. However, there are still inconsistencies comparing Windows and Mac described in nexdt comment.

[ScottChesworth]

Some gotchas/differences I noticed comparing Windows and Mac:

1. On Mac, arrows getting intercepted by Shortcut Help in the default focus position when the VKB opens depends on the VKB being undocked and arrowen=1 being present in the [vkb] section of reaper.ini. However, as soon as we move with tab or VO, arrows unexpectedly start passing through Shortcut Help for the remainder of the time the VKB window is open. Even when focus is moved back to where VO said it was when the window first opened (on the "Send all input to VKB" check box, the arrows will keep passing through Shortcut Help until the window is reloaded fresh.
2. On Mac (at least on Jen's one), there was no reporting when changing octaves or MIDI channels with arrows and Shortcut Help off yet.
3. On Windows, reporting doesn't happen if focus is moved anywhere else other than the "Virtual keyboard keys" group control. Arrows still work when the "Send all keyboard input to VKB" and "Arrow keys change octave/channel" check boxes have focus, so can reporting continue to work in those positions of the tab order too?

[jcsteh]

2. On Mac (at least on Jen's one), there was no reporting when changing octaves or MIDI channels with arrows and Shortcut Help off yet.

Right, so this isn't going to be a cross-platform thing. I've written this code about as cross-platform as I can, so if it isn't working, REAPER must be doing something different on Mac. I can't fathom what.

3. On Windows, reporting doesn't happen if focus is moved anywhere else other than the "Virtual keyboard keys" group control. Arrows still work when the "Send all keyboard input to VKB"

I don't even see a way to check that box without using screen reader specific commands.

and "Arrow keys change octave/channel"

Wow. Is that new? I haven't updated REAPER for a bit, but I don't have that option here.

check boxes have focus, so can reporting continue to work in those positions of the tab order too?

That is... a little tricky. I'll see what I can do.

[jcsteh]

The upcoming build fixes hooking of the arrow keys when the check boxes have focus on Windows. It's slightly possible this might help things on Mac too, but I doubt it.

[github-actions]

Build 52 succeeded!

• Download for Windows
• Download for Mac

[ScottChesworth]

It's very excellent on Windows! One last check nothing got hosed on Mac then merge I reckon.

[jennykbrennan]

When opening vkb, doing nothing else, arrows change octave/channel but doesn't report.
With shortcut help on, they still change octave/channel but doesn't report. (Other than the notes.)
Then after VO away and back to that checkbox, everything works as it should.
Damn, why does Mac has to be so weird.

[github-actions]

Build 54 succeeded!

• Download for Windows
• Download for Mac

[jcsteh]

I managed to drum up the fortitude to do some debugging on Mac. It turns out that when the vkb window first gets focus, despite what VoiceOver reports, the check box isn't actually the thing that has focus. Instead, it's the control for the keys themselves, which (as @pitermach points out) doesn't report anything with VO. However, on Windows, that control is a child of the vkb dialog and has a window class name. On Mac, there is no child window - it just reports as the dialog itself - and the dialog has no window class name. I guess there's something quirky going on with the SWELL emulation layer here. Anyway, it turns out that hacking around that wasn't too hard; it was just painful to debug, especially as I hardly know how to use a Mac.

[ScottChesworth]

Nice! I couldn't tell there was an equivalent control on Mac here. Would love to hear about how you debugged this if it's easy to explain?

[jcsteh]

Figuring out that there was a specific problematic control was mostly a process of deduction:

1. The fact that things work when you focus the check box manually made me suspicious that VoiceOver was lying about the focus when the vkb opened.
2. I noticed that if I tabbed after opening the vkb, the first two tab key presses reported nothing.
3. I noticed that if I tabbed all the way around, there were two silent spots before those check boxes. I still haven't worked out what the second one is, but the first one is where the hooking code failed and thus presumably the one that gets focus when the vkb opens.
4. My assertion that it is equivalent to the keys control on Windows is an educated guess; I actually have no real evidence of that, since it doesn't have a proper HWND.

Figuring out the fix involved shoving a lot of debug console messages into the code. I established that the problematic HWND had no class name and that its parent wasn't the vkb window. I nearly gave up at that point, since I figured I had nothing to work with to detect it. But on a final hunch, I started printing the pointer addresses for the HWND and its parent, then looked at the output for both the working and non-working cases. I was able to spot then that the HWND in the non-working case had the same memory address as the parent HWND (the vkb container window) in the working case. From there, the fix was simple: compare both the HWND itself, as well as its parent, against the vkb window.

[jcsteh]

One bug I'll flag here (because it took me a while to figure out wtf was going on) is that if you start REAPER with the vkb already open, this new OSARA reporting won't work at all until you restart REAPER. This will happen if you exit REAPER with the vkb open; that window state gets saved. That's fairly difficult to do on Windows because alt+f4 (the most natural way to close an application) closes the vkb. However, on Mac, command+q will quit REAPER with the vkb still open.

I think the reason is that REAPER registers its own keyboard hook for the vkb the first time the vkb opens in a given session. If REAPER registers its hook before OSARA can, since hooks are processed in the order they're installed, OSARA's hook never gets called. To make matters worse, it seems that REAPER keeps its own vkb hook registered even when you close the vkb; I have to keep OSARA's hook registered forever to work around that. That's why closing and reopening the vkb doesn't fix this situation and you have to restart REAPER.

[ScottChesworth]

Thanks heaps for explaining. We tried hitting tab when we were testing on Jen's Mac, but when we heard nothing, I assumed too soon that tabbing wasn't gonna do anything useful in that window. Duh! I've seen blank spots in tab orders plenty of times, never occured the same might happen in a SWELL GUI.

[ScottChesworth]

So if I ask Justin to tidy up the labelling to be more like the Windows VKB, will it break what you've done here?

[jcsteh]

I honestly don't know. Also, I might have a fix for that bug I mentioned above.

[jennykbrennan]

Ooh, Excellent! I hope you treated yourself with a nice cold happy brew of some sort after that, @jcsteh What a nightmare.