Commit 1f96838
Fix non-finite bounds producing UB via static_cast in XAML/WinUI3 elements (#7)
* Initial plan
* Fix element bounds validation and safe double-to-int conversion
- TAP DLL: validate _wtof/ParseOffset results are finite (reject NaN/Inf)
- graft_json_node (xaml_diag_common, plugin_loader, wpf_inject): add
safe_double_to_int helper that clamps to INT_MIN..INT_MAX range
- screenshot.cpp: use long long arithmetic to prevent int overflow in
bounds computations during annotation
- Add unit tests for bounds edge cases (zero size, large offsets, normal)
Co-authored-by: asklar <22989529+asklar@users.noreply.github.com>
* Extract safe_double_to_int to shared bounds_util.h and fix screenshot overflow
- Move safe_double_to_int to src/bounds_util.h to eliminate code duplication
- Cast winRect fields to long long before subtraction in screenshot annotation
Co-authored-by: asklar <22989529+asklar@users.noreply.github.com>
* Extract safe_double_to_int to shared bounds_util.h and fix screenshot overflow
Co-authored-by: asklar <22989529+asklar@users.noreply.github.com>
* Remove CodeQL artifact and add to gitignore
Co-authored-by: asklar <22989529+asklar@users.noreply.github.com>
* Return std::optional from safe_double_to_int, add annotations test hook and integration tests
- bounds_util.h: safe_double_to_int now returns std::optional<int>, nullopt for non-finite
- Call sites updated to use if(sx && sy && sw && sh) pattern
- Added --annotations-json CLI flag for structured annotation output
- Added collect_annotations() for computing which elements get annotated
- Integration tests: Win32 bounds validation, annotation verification,
framework-conditional WinUI3/XAML bounds checks
Co-authored-by: asklar <22989529+asklar@users.noreply.github.com>
* Use named constants for bounds thresholds, match annotate_pixels clipping in collect_annotations
Co-authored-by: asklar <22989529+asklar@users.noreply.github.com>
* Guard --annotations-json flag with #ifndef NDEBUG for debug-only builds
Co-authored-by: asklar <22989529+asklar@users.noreply.github.com>
* Add controlled Win32 window integration tests for tree structure, bounds, and annotations
Co-authored-by: asklar <22989529+asklar@users.noreply.github.com>
* Fix integration test file locking: close ifstream before fs::remove
On Windows, fs::remove throws when the file is still open by an ifstream
in the same process. Close the ifstream explicitly before removing the
annotations JSON temp files in the three affected tests.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix WinUI3 element bounds: layout offset computation and bridge matching
WinUI3's XAML diagnostics API serializes Vector3 ActualOffset as '0'
instead of proper vector components, causing all XAML elements to stack
at the bridge origin with no position differentiation.
TAP DLL (lvt_tap.cpp):
- Collect Orientation, Spacing, Margin from property chain for layout
- ComputeLayoutOffsets: compute child offsets by accumulating sibling
widths (horizontal) or heights (vertical) along StackPanel stacking axis
- Use first Orientation value in property chain (most specific override)
Bridge-to-XAML matching (xaml_diag_common.cpp):
- Replace order-based matching with size-based best-fit matching
- Recursively search XAML root descendants for content dimensions
- Skip contentless XAML roots (w=0, h=0) to prevent bridge consumption
- Match each XAML island to the bridge with closest width+height
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove unreliable layout offset heuristic, add DPI scaling
The ComputeLayoutOffsets heuristic produced wrong positions for elements
in Grid, Canvas, and other non-StackPanel containers, causing annotation
rectangles to be misaligned with actual UI elements.
Changes:
- Remove ComputeLayoutOffsets and layout field collection from TAP DLL
- Add ApplyDpiScaling: scale XAML logical pixel dimensions by the window
DPI factor (dpi/96) so they match the physical pixel space used by
GetWindowRect and the screenshot capture bitmap
- Remove unused IVisualTreeService2 member (GetProperty approach didn't
work for Vector3 value types)
- Keep the improved size-based bridge-to-XAML matching
Without per-element offsets from ActualOffset (WinUI3 serializes Vector3
as '0'), elements within a XAML island share the bridge origin position.
This is a known WinUI3 diagnostics limitation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix annotation bounds, exclude own terminal from --title, add build script
Annotation bounds fix:
- TAP DLL: only include offsetX/offsetY in JSON when non-zero, since
WinUI3 serializes ActualOffset (Vector3) as '0' making zero offsets
indistinguishable from missing data
- graft_json_node: only set positioned screen bounds on XAML elements
when explicit offset data is present in the JSON; elements without
offsets still appear in the tree (type/name/framework/width/height)
but won't produce garbage overlapping annotation rectangles
- Remove ApplyDpiScaling (no longer needed without positioned bounds)
Terminal self-match fix:
- Exclude windows belonging to our own process or console host from
--name/--title enumeration, preventing the terminal window (whose
title contains the lvt command line) from matching
Build script:
- Add build.cmd for building from a VS Developer Command Prompt
- Builds .NET dependencies (WPF TAP, Avalonia walker) before CMake
- Supports 'build.cmd clean' to reconfigure from scratch
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix XAML element positions via TransformToVisual and DPI scaling
TAP DLL — get real element positions:
- Use IXamlDiagnostics::GetIInspectableFromHandle to get the IInspectable
for each XAML element, QI for IUIElement (IID from WinUI3 winmd), and
call TransformToVisual(nullptr).TransformPoint({0,0}) via raw vtable
calls to get each element's position relative to the XAML island root
- This bypasses the broken ActualOffset property serialization (WinUI3
serializes Vector3 as '0' for readonly computed properties)
- CollectPositionsViaTransformToVisual runs on the UI thread via
SendMessage dispatch, collecting positions for 284/289 elements
TAP DLL — serialize offsets correctly:
- Only include offsetX/offsetY in JSON when non-zero, so graft_json_node
can distinguish 'no offset data' from 'offset is (0,0)'
graft_json_node — use absolute offsets:
- TransformToVisual offsets are absolute within the XAML island (not
cumulative parent-relative), so pass bridge base to all children
instead of accumulating offsets
- Only set bounds when explicit offset data is present in JSON
screenshot.cpp — fix DPI scaling:
- Use GetWindowRect (logical pixels, same space as element bounds)
instead of DwmGetWindowAttribute (physical pixels) for the annotation
origin, then scale by bmpWidth/winW to convert to bitmap coordinates
- This fixes annotation alignment on high-DPI displays (e.g. 150%)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Extract important XAML properties into tree dump
TAP DLL: capture key properties from GetPropertyValuesChain during
bounds collection. Properties extracted:
- Text content: Text, Content, Header, PlaceholderText, Description,
Title, Glyph (filtered: skip numeric handle references >10 digits)
- Accessibility: AutomationProperties.Name, AutomationProperties.AutomationId,
AutomationProperties.HelpText
- State: IsEnabled, Visibility, IsChecked, IsSelected, IsOn, Orientation
- Other: Source, Tag
Properties are serialized as a JSON object in the TAP pipe output and
copied into Element.properties by graft_json_node. Text/Content/Header
values also populate Element.text for display.
The Win32 provider already populates properties (hwnd, style, visible,
enabled). XAML elements now get similar treatment, enabling AI consumers
to identify controls by their accessible names (e.g. 'Bold (Ctrl+B)').
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use C++/WinRT projected types for XAML element inspection
Replace raw vtable calls with proper C++/WinRT projected types from
generated WinUI3 headers (Microsoft.UI.Xaml.winmd → cppwinrt.exe).
TAP DLL changes:
- #include winrt/Microsoft.UI.Xaml.h, .Media.h, .Controls.h for
UIElement, GeneralTransform, TextBlock projected types
- CollectPositionsAndText: use inspectable.try_as<UIElement>() and
call TransformToVisual(nullptr).TransformPoint({0,0}) for position
- Read TextBlock.Text() directly via C++/WinRT (no more numeric handle
references from the property chain)
- Add WINRT_LEAN_AND_MEAN to avoid windowsapp.lib dependency; provide
stub for WINRT_IMPL_RoOriginateLanguageException
graft_json_node (xaml_diag_common.cpp):
- Map XAML x:Name to properties['name'] instead of Element.text
- Set Element.text from Text > Content > Header > AutomationProperties.Name
Build:
- Generate C++/WinRT headers from WinUI3 winmd at build setup time
- Add src/tap/winui3/ to TAP DLL include path; gitignored (generated)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix text property: don't use AutomationProperties.Name as text, remove duplicates
- Only set Element.text from actual text-content properties (Text,
Content, Header) — not AutomationProperties.Name
- AutomationProperties.Name stays as a property attribute in the output
- When Text/Content/Header populates Element.text, remove it from
properties to avoid outputting both text='File' and Text='File'
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reduce annotation clutter: skip layout containers, place labels inside boxes
- Skip pure layout types (Grid, StackPanel, Border, ContentPresenter,
ScrollViewer, Window, etc.) from screenshot annotations unless they
have text content or an AutomationProperties.Name
- Place labels inside the bounding box when they fit, falling back to
above or below the box when they don't
- Keeps interactive controls (Button, ToggleButton, TextBlock, etc.)
and elements with meaningful text/a11y names
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix CI: generate WinUI3 C++/WinRT headers at configure time
The TAP DLL's WinUI3 C++/WinRT projected headers are generated from
Microsoft.UI.Xaml.winmd using cppwinrt.exe and are gitignored. CI needs
to generate them at build time.
CMakeLists.txt:
- Add configure-time step to find cppwinrt.exe (from Windows SDK),
locate the Windows App SDK winmd files (NuGet global cache or local
packages/ dir), and run cppwinrt to generate headers
- Search both NuGet global cache and CI packages/ directory
- Skip generation if headers already exist (incremental builds)
CI (ci.yml):
- Add step to install Windows App SDK NuGet package before configure
TAP DLL (lvt_tap.cpp):
- Make WinUI3 C++/WinRT include optional via __has_include guard
- All TransformToVisual/TextBlock.Text code is #if LVT_HAS_WINUI3_PROJECTION
- Without the headers, TAP DLL compiles and works for property chain
bounds but without position resolution or text reading
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Support UWP/system XAML (Windows.UI.Xaml) for position and text reading
The TAP DLL now tries both WinUI3 (Microsoft.UI.Xaml) and system XAML
(Windows.UI.Xaml) interfaces when reading element positions and text.
This enables proper annotation of UWP apps like Clock, Calculator, etc.
- System XAML headers (Windows.UI.Xaml) are always available from the
Windows SDK — no generation needed
- TransformToVisual: tries WinUI3 UIElement first, falls back to
system XAML UIElement
- TextBlock.Text: tries WinUI3 TextBlock first, falls back to system
- Guard changed from LVT_HAS_WINUI3_PROJECTION to LVT_HAS_XAML_PROJECTION
(always true since system headers are always present)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: asklar <22989529+asklar@users.noreply.github.com>
Co-authored-by: Alexander Sklar <AskLar@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>1 parent 25e3ef9 commit 1f96838
15 files changed
Lines changed: 1143 additions & 56 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
26 | 29 | | |
27 | 30 | | |
28 | 31 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
53 | 53 | | |
54 | 54 | | |
55 | 55 | | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
56 | 128 | | |
57 | 129 | | |
58 | 130 | | |
59 | 131 | | |
60 | | - | |
61 | | - | |
| 132 | + | |
| 133 | + | |
62 | 134 | | |
63 | 135 | | |
64 | 136 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
| 14 | + | |
14 | 15 | | |
15 | 16 | | |
16 | 17 | | |
| |||
30 | 31 | | |
31 | 32 | | |
32 | 33 | | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
33 | 37 | | |
34 | 38 | | |
35 | 39 | | |
| |||
47 | 51 | | |
48 | 52 | | |
49 | 53 | | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
50 | 57 | | |
51 | 58 | | |
52 | 59 | | |
| |||
75 | 82 | | |
76 | 83 | | |
77 | 84 | | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
78 | 89 | | |
79 | 90 | | |
80 | 91 | | |
| |||
265 | 276 | | |
266 | 277 | | |
267 | 278 | | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
268 | 298 | | |
269 | 299 | | |
270 | 300 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
| 3 | + | |
3 | 4 | | |
4 | 5 | | |
5 | 6 | | |
| 7 | + | |
6 | 8 | | |
7 | 9 | | |
8 | 10 | | |
| |||
143 | 145 | | |
144 | 146 | | |
145 | 147 | | |
146 | | - | |
147 | | - | |
| 148 | + | |
| 149 | + | |
148 | 150 | | |
149 | | - | |
150 | | - | |
151 | | - | |
152 | | - | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
153 | 161 | | |
154 | 162 | | |
155 | 163 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| 9 | + | |
9 | 10 | | |
10 | 11 | | |
11 | 12 | | |
| |||
15 | 16 | | |
16 | 17 | | |
17 | 18 | | |
| 19 | + | |
18 | 20 | | |
19 | 21 | | |
20 | 22 | | |
| |||
71 | 73 | | |
72 | 74 | | |
73 | 75 | | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
78 | 86 | | |
79 | 87 | | |
80 | 88 | | |
| |||
0 commit comments