Extend Accessibility Features#6295
Open
natefinch wants to merge 16 commits into
Open
Conversation
Expand the accessibility public API to support the breadth of widgets the Fyne toolkit ships: - Add 15 new AccessibleRole constants (Checkbox, TextField, Tab, TabList, List, ListItem, Tree, TreeItem, Slider, ProgressBar, Image, Heading, Separator, Radio, Table) alongside the existing Button/Container/ Link/Text. - Introduce typed AccessibleAction (press, increment, decrement, select, showMenu, setValue) and AccessibleState (checked, disabled, expanded, focused, invalid, required, selected) string constants. - Add five optional interfaces (AccessibleChildren, AccessibleValue, AccessibleValueSetter, AccessibleActions, AccessibleStates) so widgets can opt into richer behaviour without breaking existing implementors of Accessible. All new symbols are tagged Since: 2.8 to match the existing scaffold. The accompanying tests pin the constant values and verify a stub implementor satisfies every interface.
The accessibility walker on every platform only recurses through *fyne.Container.Objects today, which means widgets that compose their children outside of a Container (tabbed containers, trees, lists, forms) are invisible to assistive technologies even when their children would otherwise be reachable. Introduce a shared common.AccessibilityChildren helper that returns the canvas objects to descend into, preferring fyne.AccessibleChildren when the widget implements it and falling back to *fyne.Container otherwise. The four platform walkers (darwin/windows/ios/android) all delegate child enumeration to this helper. While here, fix a Darwin walker bug that pruned the entire subtree when the current object did not itself implement fyne.Accessible: non-accessible wrapper objects (decorative containers, custom layouts) are now traversed transparently.
AppTabs and DocTabs now implement AccessibleChildren by returning their renderer's objects so the tab bar (and through it each tab button) and the active tab's content are reachable from the accessibility walker. The internal tabButton widget reports the AccessibleRoleTab role, labels itself with its tab text (or icon name when only an icon is set), exposes Disabled and Selected states, and handles AccessibleActionPress and AccessibleActionSelect to switch tabs from assistive technologies. Split exposes itself as an AccessibleRoleContainer with the leading and trailing children, skipping nil siblings. Scroll exposes itself as an AccessibleRoleContainer with its wrapped Content as the sole child so the walker can keep descending into the scrollable subtree. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Button gains AccessibilityStates (Disabled), AccessibilityActions (Press), and AccessibilityPerformAction so assistive technologies can describe a button's enabled state and trigger its tap handler. Hyperlink gains the same press action so links can be followed without a pointer. Check, the radio item used inside RadioGroup, and Slider all expose their role, label, supported actions, and per-state flags (Checked/Selected/Disabled/Focused/Required/Invalid where applicable) and execute Press/Select/Increment/Decrement/SetValue actions through their existing handlers. Slider additionally implements AccessibleValueSetter so AT clients can write a numeric value directly. Entry exposes its placeholder as the accessibility label, its text as the accessibility value, and accepts AccessibilitySetValue to drive typing from AT. State flags reflect Disabled/Focused as well as Required (when a Validator is set) and Invalid (when validation has failed). Select reports as a button whose value is the currently selected option, with Press and ShowMenu actions both opening the option list. ProgressBar reports its percentage (or TextFormatter result) as the value, while ProgressBarInfinite reports "indeterminate". RadioGroup, CheckGroup, and Form expose AccessibleChildren so the AX walker can descend into their rendered items and form rows. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
List, GridWrap, Tree, and Table each implement AccessibleChildren so the accessibility walker can descend into the rows or cells that are currently rendered, rather than treating these virtualized widgets as opaque leaves. Each container reports its appropriate role (List/List/Tree/Table) and an empty default label that callers can override on the embedding BaseWidget. Visible list and grid items expose role ListItem, defer their label to the wrapped child when it implements fyne.Accessible, report the Selected state, and accept Press and Select actions through their existing onTapped handler. Tree nodes expose role TreeItem with the Expanded state for open branches and the Selected state for the current selection; Press toggles a branch and selects a leaf, while Select always selects the node. Table exposes the visible cell content (including header rows and columns) as accessible children so screen readers can read across the rendered viewport. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Card, Toolbar, Accordion, Menu, and PopUp implement AccessibleChildren so the accessibility walker can descend through their content. Card labels itself with its title (or subtitle if no title is set). Toolbar filters out the layout spacer items but exposes actions and separators. Accordion exposes section header buttons and, when a section is open, its detail content. Menu items expose role Button, the disabled and checked states, and a press action that triggers their existing tap handler. Icon and FileIcon report role Image with a label derived from the resource or URI name respectively. Separator reports role Separator with no label. Activity reports role ProgressBar with an "active"/"idle" value reflecting its animation state. RichText and TextGrid report role Text with the plain-text content as their label so screen readers can announce them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Expand the macOS NSAccessibility bridge to cover the full Fyne accessibility surface introduced in earlier commits: 19 roles, six actions (press/increment/decrement/showMenu/select/setValue), and the seven accessibility states. Cgo Handles are used to round-trip widgets through Obj-C; the handle is freed in -dealloc via a small destroy callback so AppKit can outlive a Go-side rebuild without dangling. Notifications (value changed, selected children changed, layout changed, focused element changed) are posted on dynamic updates so AT clients pick up changes live. Tab uses the radio-button role with a tab subrole; Heading is exposed as static text with a 'heading' role description, matching the NSAccessibility role catalogue. List/Tree/Table remain accessibility elements so screen readers can navigate into them — only Group/Container nodes with children continue to delegate to their descendants. Also fixes a nil-icon panic in widget.Button.AccessibilityLabel that surfaced once the bridge started exercising every walked widget. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Map all 19 Fyne AccessibleRole values to the closest UIA control type (CheckBox, RadioButton, Slider, ProgressBar, Edit, List/ListItem, Tree/TreeItem, DataGrid, Tab/TabItem, Image, Header, Separator) so NVDA and other UIA clients see meaningful semantics. Action and value patterns remain to be wired up in a follow-up commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Map the full Fyne AccessibleRole set through to the iOS and Android bridges. iOS surfaces traits for Button/Checkbox/Radio/Tab (button), Heading (header), Image, Link, Slider (adjustable), ProgressBar (updatesFrequently), and StaticText. Android forwards the full role integer to the Java accessibility delegate so the Kotlin side can attach the appropriate setClassName mapping. Action and value patterns remain to be wired up in a follow-up commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Author
|
oy, I think this got munged.. will try to repair the diff later. |
macOS bridge — fix reference-counting and cleanup of AccessibleElement: - Remove the extra [elem retain] in AccessibilityElementCreate; the +1 from alloc/init is already the create-ownership reference. Previously every element was over-retained by one. - In AccessibilityElementAddChild, release the child after adding it to the parent children array (which retains it). This transfers create-ownership to the parent so the child will dealloc when the parent releases it. - In AccessibilityElementRemoveChild, nil the parentElement pointer before removing from the children array so the child does not reference a stale parent during its potential dealloc. - In AccessibilityElementDestroy, remove the element from globalAccessibilityElements so stale roots do not remain visible to VoiceOver after a rebuild. If the element is still parented under another AccessibleElement, detach it from that parent (which triggers the final release via the children array). - Autorelease the NSArray returned by -accessibilityChildren and customAccessibilityChildren; [copy] returns a +1 object that was previously leaked on every query. - Use [[NSMutableArray alloc] init] instead of [NSMutableArray array] for globalAccessibilityElements so the global array is properly owned (not autoreleased out from under us). Android bridge — sync Java role constants with Go enum: The Go side (accessibility_android.go) sends role integers 0–18 matching the androidRole* iota, but the Java side only understood the old four-value set (Button=1, Text=2, Link=3, Container=4). This caused most roles to be misreported to TalkBack — e.g. Checkbox (2) was announced as "Text", Heading (3) as "Link". Update GoNativeActivity.java to: - Define all 19 role constants matching the Go iota (0=Container through 18=TreeItem). - Map each role to the closest Android widget className and appropriate AccessibilityNodeInfo properties (checkable, clickable, heading, etc.). Doc fix — clarify AccessibleChildren fallback behaviour: The AccessibleChildren doc comment previously said "the walker will fall back to recursing through any Container.Objects" when nil is returned. In reality the fallback to Container.Objects only applies to objects that do not implement AccessibleChildren at all. Reword to match the actual behaviour of common.AccessibilityChildren(). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The macOS accessibility bridge had a critical bug that made Fyne windows invisible to System Events (AppleScript) and other external accessibility clients. The root cause was that AccessibilityAttachToWindow used method_setImplementation to override accessibilityChildren on the GLFWContentView class, but this actually modified the method on the parent class (NSView/NSResponder), breaking accessibilityChildren for ALL views — including NSWindow — which caused the window to disappear from the accessibility hierarchy entirely. Fixes in accessibility_darwin.m: - Replace method_setImplementation with class_addMethod for the content view's accessibilityChildren override, ensuring only GLFWContentView is affected without modifying parent classes - Add class_addMethod overrides on GLFWContentView to fix its default AXUnknown role (now AXGroup), isAccessibilityElement (now YES), and accessibilityIsIgnored (now NO) - Add class_addMethod override on GLFWWindow for accessibilityChildren so the window exposes its content view to the accessibility tree - Add accessibilityWindow and accessibilityTopLevelUIElement to AccessibleElement so assistive tech can navigate back to the window - Add legacy API support (accessibilityIsIgnored, accessibilityAttributeNames, accessibilityAttributeValue:) to AccessibleElement, since external AX clients like System Events and VoiceOver query via the deprecated legacy path rather than the modern NSAccessibility protocol New documentation: - ACCESSIBILITY.md: Comprehensive guide covering the public API, widget implementation patterns, platform bridge architecture, macOS debugging techniques, known pitfalls, and file reference - .agents/skills/macos-accessibility/: AI agent skill for diagnosing and fixing macOS accessibility issues in GLFW/Fyne apps Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document that Xcode's Accessibility Inspector requires a .app bundle and provide a minimal bundle creation example. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
natefinch
force-pushed
the
nf/accessibility
branch
from
823345b to
ab6963e
Compare
|
coverage: 60.45% (+0.3%) from 60.171% — natefinch:nf/accessibility into fyne-io:develop |
natefinch
commented
May 8, 2026
natefinch
commented
May 8, 2026
Co-authored-by: Nate Finch <natefinch@github.com>
Author
|
My main impetus for this was to enable computer use with LLMs. In working on this it eventually became clear that there are some interesting requirements for at least some computer use systems on MacOS - the binary has to be bundled into a .app file, or the computer use code can't access it. I added the LLM's accessibility research as a document in th PR, because that can help future devs continuing the work. Most of the additions are pretty simple, especially for non-Mac OSes, since I didn't have a way to test those (though now that I think of it, I can grab a Linux machine to test on). For MacOS, the changes are more extensive, since I can test thoroughly there. I was able to test not only via copilot-cli's computer-use mcp but also using XCode's accessibility testing tool (both require building into a .app to function). The accessibility tool shows the correct name and type etc.. I'll get some screenshots tonight. |
Member
Thanks for all the hard work here, were you still planning to share the screenshots? Also, please remove the CHANGELOG updates - we create that during the release process. |
Member
|
@natefinch are you here to help us get this over the line? There are some CI failures and a couple of small requests. Should not take long but we can't land it as-is. |
Author
|
Yep sorry... I'll get screenshots hopefully tonight. I don't want this to rot. |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Author
|
Went to get screenshots last night and my app kept hard-crashing when the accessibility inspector examined it, which is obviously not a good sign. This wasn't happening the last time I was working on this. I'll try to figure out what's going on and fix it. |
Member
Any luck? We're a day or so away from code freeze and this is at risk of missing :( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds a complete, opt-in accessibility implementation for Fyne (issue #1285), exposed under the
accessibilitybuild tag. It builds on the scaffolding already ondevelopand turns it into a usable surface for assistive technologies (VoiceOver, NVDA, Narrator, TalkBack, VoiceOver iOS) and for tooling that drives apps via the OS accessibility APIs.The change is organised into ten focused commits so each layer can be reviewed independently. The work was driven by the need to let computer-use / RPA-style tooling read and manipulate the UI through real platform AT APIs rather than synthetic input — so the macOS bridge in particular implements the full action/value/state surface end-to-end, while Windows / iOS / Android currently get the role mapping with action wiring marked as follow-up.
Why these specific design choices
Accessible; richer ones layer onAccessibleValue,AccessibleActions,AccessibleStates, etc.internal/driver/commonso all four platform drivers see the same tree, and a renderer-awareAccessibleChildrenso collection widgets can expose only their visible items without forcing every container to implement the interface.cgo.Handlelifetime owned by Obj-C-dealloc, with a destroy-callback handed to the C bridge. This is the only safe way to let AppKit retain elements past a Go-side rebuild without dangling handles.NSWindowalready is the AX window; inserting another would have broken focus/Z-order semantics.AccessibilitySetValueso VoiceOver value adjustments behave the same as keyboard interaction.Fixes #1285.
What changed, commit by commit
All commit links target this PR's commit view.
1. Add accessibility roles, actions, states and optional interfaces
Defines the public API in
accessibility.go:AccessibleRoleconstants (Button, Checkbox, Container, Heading, Image, Link, List, ListItem, ProgressBar, Radio, Separator, Slider, Tab, TabList, Table, Text, TextField, Tree, TreeItem).AccessibleActionconstants (Press, Increment, Decrement, ShowMenu, Select, SetValue).AccessibleStateconstants (Checked, Disabled, Expanded, Focused, Invalid, Required, Selected).Accessible,AccessibleChildren,AccessibleValue,AccessibleValueSetter,AccessibleActions,AccessibleStates) — each with aSince: 2.8doc tag.2. Walk into AccessibleChildren and stop dropping non-accessible subtrees
Adds
common.AccessibilityChildren(): the single place that decides "what are this object's accessible children?" — preferringAccessibleChildrenwhen implemented, falling back to*fyne.Container.Objects, otherwise returning nil. All four platform walkers (accessibility_darwin.go,accessibility_windows.go,accessibility_ios.go,accessibility_android.go) now go through this helper so they cannot drift apart, and so a non-Accessible wrapper (e.g. a styled rectangle) no longer hides the accessible widgets nested inside it.3. Expose accessibility on tabbed and split containers
Adds
Accessibility*methods toAppTabs,DocTabs,Splitand the internalScroll. Tabs report roleTabListwith each header reporting roleTabplus theSelectedstate;SplitreportsContainerwith both panes as accessible children.4. Expose accessibility on form and input widgets
Wires up the full action/value/state surface for
Button,Hyperlink,Check,RadioGroup/radioItem,CheckGroup,Slider,Entry,Select,ProgressBar,ProgressBarInfinite, andForm. Highlights:Slider.AccessibilitySetValueparses, clamps and snaps to step using the existingclampValueToRangemath so AT-driven changes match keyboard behaviour.Slider.AccessibilityActionsadvertises Increment/Decrement/SetValue;AccessibilityPerformActiondelegates to existing internal helpers.Entry.AccessibilitySetValueroutes throughSetTextso validators, change callbacks and undo all behave correctly.Check,RadioGroup,CheckGroupreportChecked/Selectedstates.5. Expose accessibility on collection widgets
List,GridWrap,Tree, andTableeach gain anAccessibilityChildrenthat returns only their currently visible items, by reaching through the cached renderer (cache.CachedRenderer) to find the livelistItem/gridWrapItem/treeNode/ table cell widgets. Per-item AX is implemented on those internal types — for example:listItem.AccessibilityStateschecks the parentList.selectedslice.treeNode.AccessibilityActionsexposes Press (which toggles or selects) and Select.Tree.AccessibilityChildrenwalksscroller.Content → treeContent → treeContentRenderer.{branches,leaves}keyed by the visible-node order.This avoids materialising AX nodes for thousands of off-screen rows in a long list while still letting AT clients tab through every visible item.
6. Expose accessibility on remaining standard widgets
Smaller widgets get straightforward implementations:
Card(Container, label = title or subtitle),Toolbar(Container, children = eachToolbarObject()excluding spacers),Accordion(List, children = headers + open Detail),Icon/FileIcon(Image),Separator,Activity(ProgressBar, value="active"/"idle"),RichText(Text, label =String()),TextGrid,Menu,menuItem(Button + Disabled/Checked, Press →Tapped),PopUp.7. Wire accessibility actions, states, and roles on macOS ⭐
This is the biggest commit and the one that makes the rest visible to assistive tech on macOS. Touches the three
accessibility_darwin.{go,h,m}files plus a new test.-accessibilityRole, with Tab subrole and Heading role description implemented separately to match NSAccessibility conventions.accessibilityPerformPress/Increment/Decrement/ShowMenu, all returningBOOLbased on the Go callback's int result.-setAccessibilityValue:at accessibility_darwin.m#L216 only mutates state when the Go side accepts the new value, then postsNSAccessibilityValueChangedNotification.stateMaskFor/actionMaskFor, Obj-C consumes them inAccessibilityElementSetStates/AccessibilityElementSetSupportedActions. This avoids 13 separate setters and keeps the C surface tiny.cgo.Handlelifetime owned by Obj-C-dealloc: Go creates a handle for each widget, passes it through avoid*context, and the CdestroyTrampolinecalls back into Go to delete the handle when the Obj-C element is finally released. This is essential because AppKit can keep elements retained past a Go-side rebuild.NSAccessibilityValueChangedNotificationon value updates,NSAccessibilitySelectedChildrenChangedNotificationon selection state changes,NSAccessibilityRowExpandedNotificationon expand/collapse,NSAccessibilityLayoutChangedNotificationon child add/remove, andNSAccessibilityFocusedUIElementChangedNotificationon focus.-isAccessibilityElementkeeps the existing Group-with-children behaviour but deliberately does not hide List/Tree/Table — those need to remain elements so AT can navigate into them.accessibility_darwin_test.go: role-distinctness, mask round-trips, state/action bitmask uniqueness, andactionFromCround-trips.Also fixes a pre-existing nil-icon panic in
Button.AccessibilityLabelthat surfaced once the bridge started walking every Accessible widget.8. Expand Windows UIA role coverage to all accessibility roles
Maps all 19 Fyne roles to the closest UIA control type in
roleToUIA— CheckBox, RadioButton, Slider, ProgressBar, Edit, List/ListItem, Tree/TreeItem, DataGrid, Tab/TabItem, Image, Header, Separator, etc. NVDA / Narrator now report the right control type for each widget. Action and value patterns (IInvokeProvider,IValueProvider, …) are deliberately deferred to a follow-up PR.9. Expand iOS and Android accessibility role coverage
iOS gets the full role enum forwarded to Obj-C, with
UIAccessibilityTraitsmapped for Button/Checkbox/Radio/Tab (button), Heading (header), Image, Link, ProgressBar (updatesFrequently), Slider (adjustable), and StaticText. Android forwards the full role integer to the Java accessibility delegate so the Kotlin side can attach the appropriatesetClassNamemapping. Action/value pattern wiring on these two platforms is deferred.10. Document accessibility support in CHANGELOG
Adds a 2.8.0 — Unreleased entry summarising the new public surface and per-platform coverage.
How to try it
go build -tags accessibility ./... go test -tags accessibility ./...On macOS, run any sample app under VoiceOver or use the macOS Accessibility Inspector — buttons, tabs, sliders, lists, trees and form inputs all surface with their semantics, can be activated, and announce value changes live.
Tests
accessibility_test.go(public-API round-trips)internal/driver/common/accessibility_test.go(walker semantics)widget/accessibility_test.go(~20+ widget AX tests covering Button / Hyperlink / Check / Entry / Select / Slider / ProgressBar* / RadioGroup / CheckGroup / Form / List / GridWrap / Tree / Table / Card / Toolbar / Accordion / Icon / Separator / Activity / RichText / TextGrid)widget/radio_item_test.go(radio item Accessibility)internal/driver/glfw/accessibility_darwin_test.go(role/state/action bitmask mapping)apptabs_internal_test.go,doctabs_internal_test.go,split_test.go,internal/widget/scroller_test.go.go test -tags accessibility ./....Checklist
goimports,go vet).Where applicable
Since: 2.8line.accessibilitybuild tag for the native bridges.runtime/cgo(already used in the toolchain) andinternal/cachefor collection widgets.Follow-ups (not in this PR)
IInvokeProvider,IValueProvider,IRangeValueProvider,IToggleProvider,ISelectionItemProvider,IExpandCollapseProvider) on Windows.AccessibilityNodeProvider.