Commit 6abc147
[Shift-Left] Auto-detect many typedefs and manual remaps by walking the AST before PInvokeGenerator runs (#2240)
* Add Win32MetadataScraper: single-pass auto-remap discovery
Hosts ClangSharp.PInvokeGenerator v17.0.1 as a library (NuGet package
references). Parses the translation unit, walks the AST to discover
typedef-tag remappings, merges them with manual --remap entries, then
runs PInvokeGenerator.GenerateBindings on the same TranslationUnit in
a single parse pass.
- Win32MetadataScraper: new console app tool
- remapExclusions.rsp: exclusion list for unsafe auto-remaps
- ScrapeHeaders.cs: launches scraper, collects .remaps, writes auto RSP
- sdk.targets: includes auto-remaps RSP in @(ScraperRsp)
- Graceful fallback to stock ClangSharpPInvokeGenerator if scraper not found
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Enhance auto-remap heuristics: remove 12,197 manual remaps and 160 fn ptr fixups
- Rewrite AST walker to collect ALL typedefs per tag, disambiguate
multi-typedef tags automatically (no exclusion list needed)
- Add UnwrapType helper for ElaboratedType/AttributedType/ParenType
sugar types that were hiding typedef-tag and fn-ptr relationships
- Auto-discover function pointer typedef pairs from AST: detect
FunctionProtoType typedefs and their PointerType aliases, generate
--remap, --exclude, and --reducePointerLevel entries automatically
- Remove 12,197 of 12,705 manual --remap entries from scraper.settings.rsp
(508 genuinely manual entries remain: semantic primitives, pointer
remaps, nested qualifieds, uppercase renames)
- Remove 160 of 184 entries from functionPointerFixups.json
(24 remain: 6 alreadyPointer, 13 name-only, 5 edge cases)
- Delete remapExclusions.rsp (no longer needed)
- Zero conflicts: auto-discovery never picks wrong name
- Winmd output identical to baseline (24,328,704 bytes)
- All 36 tests pass
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Refactor discovery logic into RemapDiscovery class with unit tests
Extract WalkTranslationUnit, ResolveTagRemaps, FilterTagRemaps,
ResolveFunctionPointerFixups, and UnwrapType from Program.cs into a
public RemapDiscovery class that can be tested independently.
Add Win32MetadataScraperTests project with 22 xUnit tests covering:
- Tag-typedef discovery: _Uppercase, _lowercase, tag prefix, lowercase
tag, trailing underscore, enum typedef patterns
- Multi-typedef disambiguation: stripped prefix match, ambiguous skip,
manual hint resolution
- Semantic override protection: manual remap takes priority
- Built-in type filtering: _GUID excluded
- Identity remap filtering: same tag/typedef skipped
- Function pointer discovery: two-step LP/P/PFN patterns, direct fn
ptr, calling convention (AttributedType unwrap), non-standard naming
skip, typedef alias pattern, ambiguous pointer targets skip
- UnwrapType: ElaboratedType and AttributedType sugar peeling
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix heuristics: case-insensitive disambiguator, suffix filter, fn ptr direction, simplified consistency check
Heuristic fixes:
- Add case-insensitive match to multi-typedef disambiguator (fixes in6_addr→IN6_ADDR
when competing with IPv6Addr typedef)
- Fix fn ptr alreadyPointer direction: when proto has P/LP prefix and alias doesn't,
remap alias→proto (e.g., EXCEPTION_ROUTINE→PEXCEPTION_ROUTINE, not reverse)
- Filter suffix-adding remaps (e.g., GLUnurbs→GLUnurbsObj) since tag is canonical
- Skip C++ namespace-qualified tags (Gdiplus::Status, ABI::*)
Cross-partition consistency:
- Simplified CheckCrossPartitionRemapConsistency: scans declared type names in
generated .cs files and checks against discovered remap tags. No regex needed.
- Warning-only (PInvokeGenerator handles these internally)
- Added 7 partition #include fixes for cross-partition typedef visibility
- 5 irreducible manual remaps for types where AST walker can't discover the typedef
in certain partition configurations
Tests: 30 scraper tests (7 new), all 60 pass. Winmd 24,328,704 bytes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Copilot instructions
* Code review fixes: remove dead code, tighten heuristics, add plan docs
- Remove unused FilterRemaps() method from Program.cs (duplicated RemapDiscovery.FilterTagRemaps)
- Tighten HasPointerPrefix to require uppercase after single P prefix
- Change CheckCrossPartitionRemapConsistency to void (always returned true)
- Fix trailing space in LSA_GET_EXTENDED_CALL_FLAGS in functionPointerFixups.json
- Add plan docs from win32metadata repo (auto-type-remappings, shift-left, annotations)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix metadata remap regressions
Restore key manual remaps, tighten auto-discovery heuristics, and capture the remaining rename-drift findings.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix remap drift handling
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Allow rsp comments
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Restore main remaps
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add auto-remap compatibility pins
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix WinmdGenerator packaging
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix CI test failure: make auto fn-ptr rsp arch-specific
The auto-generated emitter.autoFnPtr.generated.rsp was written to a
non-arch-specific path. When the three CI scrape jobs (x64, x86, arm64)
upload artifacts and the build-test job merges them, only one arch's
version survives. The x86 version is missing entries for x64/arm64-only
APIs (e.g. PGET_RUNTIME_FUNCTION_CALLBACK for RtlInstallFunctionTableCallback),
causing the NoInvalidPointersToDelegates test to fail.
Fix: include the ScanArch in the rsp filename and use conditional includes
in sdk.targets - the specific arch file for local builds, a wildcard glob
for CI merged-artifact builds (SkipScraping=true).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Improve fn-ptr heuristic: split alias tracking, handle already-pointer types
Three improvements to RemapDiscovery.ResolveFunctionPointerFixups:
1. Split FnProtoToPointerTypedefs into pointer-adding aliases (typedef FOO *PFOO)
and same-level aliases (typedef FOO BAR) via new FnSameLevelAliases field.
This fixes LSA_AP_CALL_PACKAGE which had 1 pointer alias + 1 same-level alias
incorrectly counted as 2 aliases (ambiguous).
2. Filter configuredExcludes from alias candidates before the ambiguity check.
This fixes DRVCALLBACK which has LPDRVCALLBACK + PDRVCALLBACK, but PDRVCALLBACK
is already excluded in scraper.settings.rsp.
3. Handle already-pointer typedefs with pointer-adding aliases (new pass 2).
Types like ACMDRIVERPROC (already *) with LPACMDRIVERPROC (adds another *)
now auto-generate remap + exclude + reducePointerLevel.
Also removes 5 entries from functionPointerFixups.json (now auto-discovered):
- FN_CUSTOM_HELP (stale, not in SDK 10.0.26100.0)
- DRVCALLBACK, LSA_AP_CALL_PACKAGE, ACMDRIVERPROC, INSTALLUI_HANDLER_RECORD
Auto-discovered reducePointerLevel entries: 152 -> 156.
Manual entries: 32 -> 27.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add plan doc for remaining functionPointerFixups.json entries
Documents all 27 remaining entries grouped by category, explains why
each can't be auto-discovered yet, and provides concrete steps for
a future agent to address them.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update plan: headers as source of truth, annotation fallbacks
Rewrites the remaining-function-pointer-fixups plan to align with the
goal of eliminating functionPointerFixups.json entirely. For each
category: try a simple heuristic first, fall back to w32m: annotations
in headers for cases the heuristic can't handle.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>1 parent 278d01c commit 6abc147
27 files changed
Lines changed: 4255 additions & 12880 deletions
File tree
- .github
- BuildTools
- docs/copilot/plans
- generation/WinSDK
- Partitions
- IpHlp
- NetMgmt
- RRas
- Security.Cryptography.Sip
- Security.WinTrust
- TermServ
- Wsw
- scripts
- sources
- GeneratorSdk
- MetadataTasks
- nuget
- sdk
- Win32MetadataScraper
- tests/Win32MetadataScraperTests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| |||
0 commit comments