Skip to content

Commit 6abc147

Browse files
jevansaksCopilot
andauthored
[Shift-Left] Auto-detect many typedefs and manual remaps by walking the AST before PInvokeGenerator runs (#2240)
* Add Win32MetadataScraper: single-pass auto-remap discovery Hosts ClangSharp.PInvokeGenerator v17.0.1 as a library (NuGet package references). Parses the translation unit, walks the AST to discover typedef-tag remappings, merges them with manual --remap entries, then runs PInvokeGenerator.GenerateBindings on the same TranslationUnit in a single parse pass. - Win32MetadataScraper: new console app tool - remapExclusions.rsp: exclusion list for unsafe auto-remaps - ScrapeHeaders.cs: launches scraper, collects .remaps, writes auto RSP - sdk.targets: includes auto-remaps RSP in @(ScraperRsp) - Graceful fallback to stock ClangSharpPInvokeGenerator if scraper not found Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Enhance auto-remap heuristics: remove 12,197 manual remaps and 160 fn ptr fixups - Rewrite AST walker to collect ALL typedefs per tag, disambiguate multi-typedef tags automatically (no exclusion list needed) - Add UnwrapType helper for ElaboratedType/AttributedType/ParenType sugar types that were hiding typedef-tag and fn-ptr relationships - Auto-discover function pointer typedef pairs from AST: detect FunctionProtoType typedefs and their PointerType aliases, generate --remap, --exclude, and --reducePointerLevel entries automatically - Remove 12,197 of 12,705 manual --remap entries from scraper.settings.rsp (508 genuinely manual entries remain: semantic primitives, pointer remaps, nested qualifieds, uppercase renames) - Remove 160 of 184 entries from functionPointerFixups.json (24 remain: 6 alreadyPointer, 13 name-only, 5 edge cases) - Delete remapExclusions.rsp (no longer needed) - Zero conflicts: auto-discovery never picks wrong name - Winmd output identical to baseline (24,328,704 bytes) - All 36 tests pass Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refactor discovery logic into RemapDiscovery class with unit tests Extract WalkTranslationUnit, ResolveTagRemaps, FilterTagRemaps, ResolveFunctionPointerFixups, and UnwrapType from Program.cs into a public RemapDiscovery class that can be tested independently. Add Win32MetadataScraperTests project with 22 xUnit tests covering: - Tag-typedef discovery: _Uppercase, _lowercase, tag prefix, lowercase tag, trailing underscore, enum typedef patterns - Multi-typedef disambiguation: stripped prefix match, ambiguous skip, manual hint resolution - Semantic override protection: manual remap takes priority - Built-in type filtering: _GUID excluded - Identity remap filtering: same tag/typedef skipped - Function pointer discovery: two-step LP/P/PFN patterns, direct fn ptr, calling convention (AttributedType unwrap), non-standard naming skip, typedef alias pattern, ambiguous pointer targets skip - UnwrapType: ElaboratedType and AttributedType sugar peeling Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix heuristics: case-insensitive disambiguator, suffix filter, fn ptr direction, simplified consistency check Heuristic fixes: - Add case-insensitive match to multi-typedef disambiguator (fixes in6_addr→IN6_ADDR when competing with IPv6Addr typedef) - Fix fn ptr alreadyPointer direction: when proto has P/LP prefix and alias doesn't, remap alias→proto (e.g., EXCEPTION_ROUTINE→PEXCEPTION_ROUTINE, not reverse) - Filter suffix-adding remaps (e.g., GLUnurbs→GLUnurbsObj) since tag is canonical - Skip C++ namespace-qualified tags (Gdiplus::Status, ABI::*) Cross-partition consistency: - Simplified CheckCrossPartitionRemapConsistency: scans declared type names in generated .cs files and checks against discovered remap tags. No regex needed. - Warning-only (PInvokeGenerator handles these internally) - Added 7 partition #include fixes for cross-partition typedef visibility - 5 irreducible manual remaps for types where AST walker can't discover the typedef in certain partition configurations Tests: 30 scraper tests (7 new), all 60 pass. Winmd 24,328,704 bytes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Copilot instructions * Code review fixes: remove dead code, tighten heuristics, add plan docs - Remove unused FilterRemaps() method from Program.cs (duplicated RemapDiscovery.FilterTagRemaps) - Tighten HasPointerPrefix to require uppercase after single P prefix - Change CheckCrossPartitionRemapConsistency to void (always returned true) - Fix trailing space in LSA_GET_EXTENDED_CALL_FLAGS in functionPointerFixups.json - Add plan docs from win32metadata repo (auto-type-remappings, shift-left, annotations) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix metadata remap regressions Restore key manual remaps, tighten auto-discovery heuristics, and capture the remaining rename-drift findings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix remap drift handling Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Allow rsp comments Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore main remaps Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add auto-remap compatibility pins Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix WinmdGenerator packaging Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix CI test failure: make auto fn-ptr rsp arch-specific The auto-generated emitter.autoFnPtr.generated.rsp was written to a non-arch-specific path. When the three CI scrape jobs (x64, x86, arm64) upload artifacts and the build-test job merges them, only one arch's version survives. The x86 version is missing entries for x64/arm64-only APIs (e.g. PGET_RUNTIME_FUNCTION_CALLBACK for RtlInstallFunctionTableCallback), causing the NoInvalidPointersToDelegates test to fail. Fix: include the ScanArch in the rsp filename and use conditional includes in sdk.targets - the specific arch file for local builds, a wildcard glob for CI merged-artifact builds (SkipScraping=true). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve fn-ptr heuristic: split alias tracking, handle already-pointer types Three improvements to RemapDiscovery.ResolveFunctionPointerFixups: 1. Split FnProtoToPointerTypedefs into pointer-adding aliases (typedef FOO *PFOO) and same-level aliases (typedef FOO BAR) via new FnSameLevelAliases field. This fixes LSA_AP_CALL_PACKAGE which had 1 pointer alias + 1 same-level alias incorrectly counted as 2 aliases (ambiguous). 2. Filter configuredExcludes from alias candidates before the ambiguity check. This fixes DRVCALLBACK which has LPDRVCALLBACK + PDRVCALLBACK, but PDRVCALLBACK is already excluded in scraper.settings.rsp. 3. Handle already-pointer typedefs with pointer-adding aliases (new pass 2). Types like ACMDRIVERPROC (already *) with LPACMDRIVERPROC (adds another *) now auto-generate remap + exclude + reducePointerLevel. Also removes 5 entries from functionPointerFixups.json (now auto-discovered): - FN_CUSTOM_HELP (stale, not in SDK 10.0.26100.0) - DRVCALLBACK, LSA_AP_CALL_PACKAGE, ACMDRIVERPROC, INSTALLUI_HANDLER_RECORD Auto-discovered reducePointerLevel entries: 152 -> 156. Manual entries: 32 -> 27. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add plan doc for remaining functionPointerFixups.json entries Documents all 27 remaining entries grouped by category, explains why each can't be auto-discovered yet, and provides concrete steps for a future agent to address them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plan: headers as source of truth, annotation fallbacks Rewrites the remaining-function-pointer-fixups plan to align with the goal of eliminating functionPointerFixups.json entirely. For each category: try a simple heuristic first, fall back to w32m: annotations in headers for cases the heuristic can't handle. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 278d01c commit 6abc147

27 files changed

Lines changed: 4255 additions & 12880 deletions

.github/copilot-instructions.md

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# Copilot Instructions for win32metadata
2+
3+
## Build Commands
4+
5+
```powershell
6+
# Full build (25-30 min): build tools, scrape headers, emit winmd, package, test
7+
./DoAll.ps1 -Clean
8+
9+
# Incremental build (faster, skips packages/samples)
10+
./DoAll.ps1 -ExcludePackages -ExcludeSamples
11+
12+
# Build just the build tools (MetadataTasks, ConstantsScraper, ClangSharpSourceToWinmd, etc.)
13+
dotnet build BuildTools -c Release
14+
15+
# Build the winmd only (requires tools built first)
16+
dotnet build generation/WinSDK -c Release -t:EmitWinmd
17+
18+
# Scrape a single partition (useful for debugging)
19+
dotnet build generation/WinSDK -c Release -p:ScanArch=crossarch -t:ScrapeHeaders -p:PartitionFilter=Foundation
20+
```
21+
22+
## Test Commands
23+
24+
```powershell
25+
# Run all tests
26+
./scripts/DoTests.ps1
27+
28+
# Individual test suites
29+
dotnet test tests/MetadataUtils.Tests -c Release
30+
dotnet test tests/Windows.Win32.Tests -c Release # winmd integrity tests
31+
dotnet test tests/Win32MetadataScraperTests -c Release # scraper heuristic tests
32+
dotnet test tests/ClangSharpSourceToWinmdTests -c Release
33+
34+
# Single test
35+
dotnet test tests/Win32MetadataScraperTests -c Release --filter "TransitiveInclude"
36+
```
37+
38+
## Architecture
39+
40+
The toolchain converts Windows SDK C/C++ headers into a .NET metadata assembly (`Windows.Win32.winmd`).
41+
42+
**Pipeline stages** (in order, defined in `sources/GeneratorSdk/sdk/sdk.targets`):
43+
1. **ScrapeHeaders**`Win32MetadataScraper` hosts ClangSharp PInvokeGenerator as a library, walks the AST to auto-discover typedef-tag remappings, then generates C# bindings per partition
44+
2. **ScrapeConstants**`ConstantsScraper` extracts `#define` constants via regex matching on headers
45+
3. **EmitWinmd**`ClangSharpSourceToWinmd` compiles the generated C# into the final winmd
46+
47+
**Key projects:**
48+
- `Win32MetadataScraper` — Single-pass scraper: parses the translation unit, discovers remaps from the AST, merges with manual config, calls PInvokeGenerator.GenerateBindings on the same TU
49+
- `ClangSharpSourceToWinmd` — Reads generated C# sources, applies `MetadataSyntaxTreeCleaner` transformations (memberRemap, enum additions), emits the winmd via System.Reflection.Metadata
50+
- `MetadataTasks` — MSBuild tasks (`ScrapeHeaders`, `EmitWinmd`, etc.) that orchestrate the pipeline
51+
- `WinmdUtils` — CLI for winmd inspection (duplicate detection, suggested remappings, etc.)
52+
53+
**Partition system:** Each Win32 API area (Foundation, Kernel, DirectX, etc.) is a "partition" in `generation/WinSDK/Partitions/`. Each has a `main.cpp` (header includes) and optional `settings.rsp` (per-partition config). Partitions are scraped in parallel.
54+
55+
**Response files (.rsp):** Configuration is heavily RSP-based. Key files in `generation/WinSDK/`:
56+
- `scraper.settings.rsp` — Type remaps (`--remap`), exclusions (`--exclude`), type overrides (`--with-type`)
57+
- `emitter.settings.rsp` — Member remaps (`--memberRemap`), enum additions
58+
- `autoTypes.json` — Handle/typedef definitions (RAII close functions, invalid values)
59+
- `functionPointerFixups.json` — Function pointer prototype→alias pairs
60+
61+
**CI pipeline** (`azure-pipelines.yml` / `.github/workflows/pr-validation.yml`):
62+
Three parallel scrape jobs (x64, x86, arm64) → artifacts merged → one build/test/package job.
63+
Each arch job scrapes ALL partitions for that architecture.
64+
65+
## Key Conventions
66+
67+
**NuGet feed:** The repo uses a private ADO Artifact Feed (`Win32Metadata-Dependencies`) as its sole package source. New packages from nuget.org must be saved to this feed first.
68+
69+
**Winmd equivalence:** After any change, the winmd must be compared against the baseline. The `NoSuggestedRemappings` test in `Windows.Win32.Tests` validates that no new unhandled remappings were introduced. Changes to the winmd are tracked in `scripts/ChangesSinceLastRelease.txt`.
70+
71+
**Cross-partition remaps:** Remaps in `scraper.settings.rsp` are global — they apply to all partitions. Auto-discovered remaps (from `Win32MetadataScraper`) are per-partition. If a tag name is used across partitions but the typedef is only in one partition's headers, the remap must be global (in `scraper.settings.rsp`) OR the referencing partition must `#include` the header with the typedef.
72+
73+
**C++ namespace types:** Types inside C++ namespaces (`Gdiplus::Status`, `ABI::Windows::*`) are handled by PInvokeGenerator internally. The auto-remap discovery skips namespace-qualified tags.
74+
75+
**ClangSharp version pinning:** The repo pins ClangSharp v17.0.1. `Win32MetadataScraper` references ClangSharp NuGet packages directly; native libs (`libclang.dll`, `libClangSharp.dll`) are resolved at runtime via NuGet's `runtime.json` mechanism.

BuildTools/BuildTools.proj

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
<ProjectReference Include="..\sources\ConstantsScraper\ConstantsScraper.csproj"/>
55
<ProjectReference Include="..\sources\ClangSharpSourceToWinmd\ClangSharpSourceToWinmd.csproj"/>
66
<ProjectReference Include="..\sources\WinmdUtils\WinmdUtils.csproj"/>
7+
<ProjectReference Include="..\sources\Win32MetadataScraper\Win32MetadataScraper.csproj"/>
78
</ItemGroup>
89

910
<ItemGroup>

0 commit comments

Comments
 (0)