Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Accessibility] Reading is no longer possible when hovering the mouse over items in the tab and explorer areas of the editor. #224704

Open
ryusei-48 opened this issue Aug 3, 2024 · 16 comments
Assignees
Labels
accessibility Keyboard, mouse, ARIA, vision, screen readers (non-specific) issues accessibility-sla Accessibility issue which have to be fixed or lowered severity based on process bug Issue identified by VS Code Team member as probable bug electron Issues and items related to Electron regression Something that used to work is now broken

Comments

@ryusei-48
Copy link

ryusei-48 commented Aug 3, 2024

Type: Bug

I use a screen reader. I often use the mouse cursor to read the text below it. However, when I upgraded to version 1.92, I encountered the title problem.
NVDA's mouse cursor reading function is used.
I hope it will be corrected.

バージョン: 1.92.0 (user setup)
コミット: b1c0a14
日付: 2024-07-31T23:26:45.634Z
Electron: 30.1.2
ElectronBuildId: 9870757
Chromium: 124.0.6367.243
Node.js: 20.14.0
V8: 12.4.254.20-electron.0
OS: Windows_NT x64 10.0.22631

VS Code version: Code 1.92.0 (b1c0a14, 2024-07-31T23:26:45.634Z)
OS version: Windows_NT x64 10.0.22631
Modes:

System Info
Item Value
CPUs AMD Ryzen 9 5900X 12-Core Processor (24 x 3700)
GPU Status 2d_canvas: enabled
canvas_oop_rasterization: enabled_on
direct_rendering_display_compositor: disabled_off_ok
gpu_compositing: enabled
multiple_raster_threads: enabled_on
opengl: enabled_on
rasterization: enabled
raw_draw: disabled_off_ok
skia_graphite: disabled_off
video_decode: enabled
video_encode: enabled
vulkan: disabled_off
webgl: enabled
webgl2: enabled
webgpu: enabled
webnn: disabled_off
Load (avg) undefined
Memory (System) 31.91GB (16.77GB free)
Process Argv --crash-reporter-id 819c0f14-6763-4f86-8b72-bd2914389149
Screen Reader yes
VM 40%
Extensions (54)
Extension Author (truncated) Version
rust-bundle 1Yi 1.0.0
Bookmarks ale 13.5.0
vscode-sqlite ale 0.14.1
vscode-django bat 1.15.0
vscode-intelephense-client bme 1.10.4
htmltagwrap bra 1.0.0
vscode-tailwindcss bra 0.12.5
vscode-copy-filename bra 0.1.1
python-environment-manager don 1.2.4
python-extension-pack don 1.7.0
rust-syntax dus 0.6.1
LogFileHighlighter emi 3.3.0
auto-rename-tag for 0.1.10
vscode-edit-csv jan 0.9.2
vsc-python-indent Kev 1.18.0
iconbuddy-vs-code-plugin mdd 0.0.3
zenkaku mos 0.0.3
vscode-language-pack-ja MS- 1.92.2024073109
csharp ms- 2.39.29
vscode-dotnet-runtime ms- 2.1.1
debugpy ms- 2024.10.0
python ms- 2024.12.1
vscode-pylance ms- 2024.7.1
remote-wsl ms- 0.88.2
cmake-tools ms- 1.18.44
cpptools ms- 1.21.5
cpptools-extension-pack ms- 1.3.0
copy-file-name nem 1.2.0
innerhtml nic 0.2.0
autodocstring njp 0.6.1
indent-rainbow ode 8.3.1
laravel-blade one 1.36.1
laravel5-snippets one 1.18.0
phind phi 0.25.3
material-icon-theme PKi 5.8.0
vscode-template-literal-editor pli 0.10.0
vscode-css-peek pra 4.4.1
prisma Pri 5.17.0
inline-sql-syntax quf 2.16.0
vscode-xml red 0.27.1
rust-analyzer rus 0.3.2053
crates ser 0.6.7
trailing-spaces sha 0.4.1
vscode-scss-formatter sib 3.0.0
code-clip-ring Sir 0.4.0
rust-pack Swe 0.3.38
even-better-toml tam 0.19.2
tauri-vscode tau 0.2.6
cmake twx 0.0.17
intellicode-api-usage-examples Vis 0.2.8
vscodeintellicode Vis 1.3.1
jinja who 0.0.8
sysinfo-vscode wra 2.6.0
scss-to-css yut 3.2.3

(2 theme extensions excluded)

A/B Experiments
vsliv368:30146709
vspor879:30202332
vspor708:30202333
vspor363:30204092
vswsl492:30256859
vscorecescf:30445987
vscod805cf:30301675
binariesv615:30325510
vsaa593cf:30376535
py29gd2263:31024239
c4g48928:30535728
azure-dev_surveyone:30548225
a9j8j154:30646983
962ge761:30959799
pythongtdpath:30769146
pythonnoceb:30805159
asynctok:30898717
pythonregdiag2:30936856
pythonmypyd1:30879173
2e7ec940:31000449
pythontbext0:30879054
dsvsc016:30899300
dsvsc017:30899301
dsvsc018:30899302
cppperfnew:31000557
dsvsc020:30976470
pythonait:31006305
dsvsc021:30996838
da93g388:31013173
pythoncenvpt:31062603
a69g1124:31058053
dvdeprecation:31068756
dwnewjupyter:31046869
impr_priority:31102340
nativerepl2:31104044
refactort:31108082
ccplc:31103425
pythonrstrctxt:31103193
wkspc-onlycs-c:31106320
wkspc-ranged-t:31107835

@ryusei-48 ryusei-48 changed the title Reading is no longer possible when hovering the mouse over items in the tab and explorer areas of the editor. [Accessibility] Reading is no longer possible when hovering the mouse over items in the tab and explorer areas of the editor. Aug 3, 2024
@meganrogge
Copy link
Contributor

I am not familiar with using the mouse cursor functionality. Could you pls provide exact steps and what happened before vs what's happening now (or not)?

cc @rperez030

Thank you

@meganrogge meganrogge added the info-needed Issue requires more information from poster label Aug 5, 2024
@rperez030
Copy link
Contributor

NVDA has a feature to read the object under the cursor. That is also part of magnification tools such as zoomtext. I understand they rely on the UIA_BoundingRectangle property to determine the object under the mouse, which I've heard is not very reliable in browsers. I'm personally not very familiar with it since it is not something i would use myself.

@meganrogge meganrogge added accessibility Keyboard, mouse, ARIA, vision, screen readers (non-specific) issues confirmation-pending accessibility-sla Accessibility issue which have to be fixed or lowered severity based on process and removed info-needed Issue requires more information from poster labels Aug 6, 2024
@ryusei-48
Copy link
Author

I use the Japanese version of NVDA, so I don't know exactly how to enable that feature in the English version, but it is probably almost the same.

To enable it, open the menu by pressing "NVDA Ki~ + N", select "Settings", and check "Mouse" under the category "Mouse" -> "Report text at mouse cursor position".

Then, in various applications and browsers, when you move the mouse over the text, it will be read out loud.

Because I have low vision, I work with the mouse cursor there and have the text read to me, while somehow perceiving the position of the text.

In one previous version of VS Code (1.92 or earlier), I was able to read out the text without any problem.

Is there some change in the structure of the DOM or is it a problem with the version of the Chrome Driver that is installed?

I have tried various DOM change operations on my end using VS Code's developer tools, but so far it seems to have no effect.

If that didn't help, you could listen for f-focus events on the JavaScript side, write the target text to the element with the "aria-live" attribute, and notify the NVDA side of the text.

@ryusei-48
Copy link
Author

ryusei-48 commented Aug 7, 2024

There is one sure cause for not reading them.
We need to be sure that this is the case, but we would like to share our findings.

If you want NVDA to read text under the mouse cursor, the object must be directly touching the mouse cursor.

For example, as an example of how to make it unreadable, if you place a transparent DOM over the object for some purpose (using position:absolute, etc.) or create an object using CSS pseudo-classes (before, after, etc.), NVDA will not read NVDA will not read it.

Also, I just downgraded the VS Code version to 1.91. I am reading the text under the mouse cursor with no problem.

@meganrogge
Copy link
Contributor

@deepak1556 know of any change that could've caused this?

@deepak1556
Copy link
Collaborator

@ryusei-48 given this seems like a possible regression, it would help to know which change caused it. We maintain a node.js based CLI tool vscode-bisect that automatically downloads and runs previous VS Code insiders and asks for each build if the issue reproduces. It takes up to 8 steps to find exactly the build that caused the regression.

Steps:

  • install node.js (if not yet done)
  • npx --yes @vscode/vscode-bisect@latest
  • follow the instructions until you found the offending build (its fine to leave the commit empty)
  • report back the commit range in this issue

Thanks!

@ryusei-48
Copy link
Author

ryusei-48 commented Aug 7, 2024

The range of problematic commits is now as follows

git bisect start && git bisect bad 2fe05272052cf1f0ab2133f7f4874e9a748e8e74 && git bisect good b23e791eb5afbd95f05aa24da7693ce89344a079

Since the problem was with a release build, we ran the command as follows

npx --yes --releasedOnly @vscode/vscode-bisect@latest

The GitHub page summarizing the scope of the problematic commits is below.
b23e791...2fe0527

Thanks!

@ryusei-48
Copy link
Author

We first reported that the text in the Explorer area and editor tabs was no longer being read out loud, but after much verification, we found that the text under the mouse cursor was no longer being read out loud over a fairly wide area.

For example, the popup that appears in the lower right corner of the screen when there is some notification, the description popup that appears when the mouse cursor is over a TypeScript keyword or type, and all items on the settings page.

It seems to me that something is wrong with the entire VS Code screen.

@hwf1324
Copy link

hwf1324 commented Aug 7, 2024

I'm having the same issue.

I am using the https://github.com/hwf1324/objectViewer NVDA add-on I developed to view the layout of the NVDAObject.

Here is an example:

屏幕截图 2024-08-07 162921

The Report Issues window for the latest preview version

屏幕截图 2024-08-07 163245

Report issue window of an earlier preview version

The difference between them can be observed with a new NVDAObject.
Perhaps as mentioned above there is something blocking the mouse from finding the object below?

@hwf1324
Copy link

hwf1324 commented Aug 7, 2024

On a side note, could this be related to Electron? Because one of the apps I use had a similar problem after one of the recent updates. It uses Electron as well.

Of course I haven't investigated or tested it, just making a hypothesis.

@deepak1556
Copy link
Collaborator

Thanks @ryusei-48 for the bisect results, the commit does reference an Electron update we had in 1.92 it could be very well be a regression in Chromium.

I don't have the setup to confirm this issue yet, If anyone is interested in the meantime to investigate further please try the following next steps to bisect chromium builds,

python tools/bisect-builds.py -a win -g 122.0.6261.156 -b 124.0.6367.243 --use-local-cache --verify-range -- --no-first-run https://vscode.dev

@ryusei-48
Copy link
Author

Thank you for your research.
I just did a bisect of cromium.

As a result, it seems that there is a problem with cromium as you mentioned.

The problem is happening in the following commit ranges
f777d3b07c236181ee6f8899523171e871d7abe9 -> 28ea064cc7c474a6e850b53e35487ad6bdab58d0

CHANGELOG URL:
https://chromium.googlesource.com/chromium/src/+log/f777d3b07c236181ee6f8899523171e871d7abe9..28ea064cc7c474a6e850b53e35487ad6bdab58d0

@deepak1556
Copy link
Collaborator

Thanks for the quick turnaround, but looking at the commit history in that range only one commit is relevant to a11y but that is not bundled as part of our application. It could be that the bisect range is incorrect, can you try the chromium bisect again to confirm.

@ryusei-48
Copy link
Author

ryusei-48 commented Aug 7, 2024

OK, I tried again.
The results were a little different.

PS> python bisect-builds.py -a win -g 122.0.6261.156 -b 124.0.6367.243 --use-local-cache --verify-range -- --no-first-run https://vscode.dev
Downloading list of known revisions. If the range is large, this can take several minutes...

Loaded revisions 1625-1338356 from C:\Users\yamaz\Downloads\test\.bisect-builds-cache.json
Downloading revision 1250580...
Received 209320771 of 209320771 bytes, 100.00%
Trying revision 1250580...
Revision 1250580 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g
Downloading revision 1274506...
Trying revision 1274506...
Revision 1274506 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: b
Downloading revision 1262540...
You have about 10 more steps left.
Bisecting range [1250580 (good), 1274506 (bad)].
Trying revision 1262540...
Revision 1262540 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g
Downloading revision 1268390...
You have about 9 more steps left.
Bisecting range [1262540 (good), 1274506 (bad)].
Trying revision 1268390...
Revision 1268390 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g
Downloading revision 1271471...
You have about 8 more steps left.
Bisecting range [1268390 (good), 1274506 (bad)].
Trying revision 1271471...
Revision 1271471 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: b
Downloading revision 1270530...
You have about 7 more steps left.
Bisecting range [1268390 (good), 1271471 (bad)].
Trying revision 1270530...
Revision 1270530 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g
Downloading revision 1270722...
You have about 6 more steps left.
Bisecting range [1270530 (good), 1271471 (bad)].
Trying revision 1270722...
Revision 1270722 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g
Downloading revision 1270902...
You have about 5 more steps left.
Bisecting range [1270722 (good), 1271471 (bad)].
Trying revision 1270902...
Revision 1270902 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g
Downloading revision 1271272...
You have about 4 more steps left.
Bisecting range [1270902 (good), 1271471 (bad)].
Trying revision 1271272...
Revision 1271272 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g
Downloading revision 1271386...
You have about 3 more steps left.
Bisecting range [1271272 (good), 1271471 (bad)].
Trying revision 1271386...
Revision 1271386 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g
Downloading revision 1271438...
You have about 2 more steps left.
Bisecting range [1271386 (good), 1271471 (bad)].
Trying revision 1271438...
Revision 1271438 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g
Downloading revision 1271458...
You have about 1 more steps left.
Bisecting range [1271438 (good), 1271471 (bad)].
Trying revision 1271458...
Revision 1271458 is [(g)ood/(b)ad/(r)etry/(u)nknown/(s)tdout/(q)uit]: g
You are probably looking for a change made after 1271458 (known good), but no later than 1271471 (first known bad).
CHANGELOG URL:
  https://chromium.googlesource.com/chromium/src/+log/c84e8a928888a874d39e0f3ef70b13721b8bdbdf..ab56da8dd8d41d786caf4954fd8e9aea55beebb0

@meganrogge meganrogge added electron Issues and items related to Electron regression Something that used to work is now broken and removed confirmation-pending labels Aug 7, 2024
@hwf1324
Copy link

hwf1324 commented Aug 9, 2024

Related: electron/electron#42945

@hwf1324
Copy link

hwf1324 commented Sep 9, 2024

I created an NVDA add-on to temporarily fix this.

https://github.com/hwf1324/mouseEnhancements

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
accessibility Keyboard, mouse, ARIA, vision, screen readers (non-specific) issues accessibility-sla Accessibility issue which have to be fixed or lowered severity based on process bug Issue identified by VS Code Team member as probable bug electron Issues and items related to Electron regression Something that used to work is now broken
Projects
None yet
Development

No branches or pull requests

5 participants