[Shift-Left] Auto-detect many typedefs and manual remaps by walking the AST before PInvokeGenerator runs

[jevansaks]

Summary

This PR introduces automatic discovery of typedef→tag remaps and function pointer fixups by walking the Clang AST before PInvokeGenerator runs. This eliminates 12,176 manual --remap entries from scraper.settings.rsp and 152 manual entries from functionPointerFixups.json, replacing them with entries that are auto-discovered at scrape time.

The net winmd output is identical — the same types, the same metadata, the same binary. What changes is how the toolchain arrives at the correct configuration: instead of a 12,700-line hand-maintained remap list, the scraper discovers most remaps from the headers themselves.

Motivation

The manual remap list in scraper.settings.rsp was a growing maintenance burden:

• Every new Windows SDK release could introduce new typedef→tag pairs requiring manual additions
• Mistakes were hard to detect — a missing remap silently produces a less-useful type name in the winmd
• The list had grown to 12,718 entries with no automated way to validate completeness

How it works

New tool: Win32MetadataScraper — a .NET 8 console app that hosts ClangSharp's PInvokeGenerator as a library. For each partition:

1. Parses the translation unit using ClangSharp's CXIndex
2. RemapDiscovery walks the AST to find:
   • Typedef→tag remaps: typedef struct _FOO { ... } FOO; → --remap _FOO=FOO
   • Function pointer fixups: typedef void (*PFOO_CALLBACK)(...) where FOO_CALLBACK is a bare function typedef → --exclude PFOO_CALLBACK + --reducePointerLevel PFOO_CALLBACK
3. Applies heuristic filters (disambiguation for 1:N tag→typedef, suffix-based selection, case-insensitive matching)
4. Merges auto-discovered remaps with manual --remap entries (manual always wins on conflict)
5. Passes the merged config to PInvokeGenerator.GenerateBindings on the same TranslationUnit — single parse, no re-parsing

ScrapeHeaders MSBuild task collects the per-partition discoveries and:

• Writes scraper.autoRemaps.generated.rsp (fed back to the scraper for cross-partition consistency)
• Writes emitter.autoFnPtr.{arch}.generated.rsp (consumed by the emitter for --reducePointerLevel)
• Deduplicates discovered entries against manual fixup RSPs to avoid conflicts
• Validates cross-partition remap consistency (detects when a tag is remapped in one partition but not another)

What remained manual (and why)

Category	Count	Reason
--remap entries	542	Non-typedef remaps (e.g., long=int, semantic overrides, COM-specific), remaps for types in C++ namespaces, or cases where the auto-heuristic can't disambiguate
functionPointerFixups.json entries	32	Irregular naming patterns (e.g., DRVCALLBACK→LPDRVCALLBACK), alreadyPointer entries, callback types without a P-prefixed pointer typedef, or functions not declared via standard typedef patterns

Partition header adjustments

Seven partitions needed additional #include directives so that typedef declarations are visible during AST walking. Without these, the auto-discovery can't find the typedef→tag pair because the typedef is declared in a header not included by that partition. These are additive-only changes to main.cpp files.

CI pipeline fix

The auto-generated emitter.autoFnPtr.generated.rsp was originally written to a non-arch-specific path. In the CI pipeline (3 parallel scrape jobs for x64/x86/arm64), the artifact merge would overwrite the file — and the x86 version was missing x64/arm64-only entries (e.g., PGET_RUNTIME_FUNCTION_CALLBACK for RtlInstallFunctionTableCallback). Fixed by making the filename arch-specific and using conditional includes in sdk.targets.

Other changes

• ChangesSinceLastRelease.txt: Empty! (no net winmd changes)
• Test suite: New Win32MetadataScraperTests project with 41 tests covering RemapDiscovery heuristics, RSP file parsing, and header snippet parsing
• DoTests.ps1: Updated to run the new scraper test suite
• BuildSdk.nuspec: Includes Win32MetadataScraper in the SDK package
• Copilot instructions: Added .github/copilot-instructions.md for repo-specific context
• Planning docs: docs/copilot/plans/ contains research and implementation plan documents

Verification

Check	Result
Winmd binary	Identical output (24,328,704 bytes)
ChangesSinceLastRelease.txt	Empty (no net API changes)
MetadataUtils.Tests	17/17 ✅
Windows.Win32.Tests	13/13 ✅
Win32MetadataScraperTests	41/41 ✅
ClangSharpSourceToWinmdTests	6/6 ✅
Auto-discovered remaps	~12,176 (replacing manual entries)
Auto-discovered fn-ptr fixups	152 reducePointerLevel entries

Files changed (32 files, +4,033 / −14,104)

Area	Files	Summary
New tool	sources/Win32MetadataScraper/ (3 files)	Program.cs (scraper host), RemapDiscovery.cs (AST walking + heuristics), .csproj
New tests	tests/Win32MetadataScraperTests/ (4 files)	41 tests for discovery heuristics, RSP parsing, header snippet parsing
Build orchestration	ScrapeHeaders.cs, TaskUtils.cs, sdk.targets	Auto-remap collection, deduplication, cross-partition consistency checks, arch-specific RSP generation
Configuration	scraper.settings.rsp, functionPointerFixups.json	−12,176 manual remaps, −152 manual fn-ptr fixups (replaced by auto-discovery)
Partitions	7 main.cpp files	Additional #include directives for typedef visibility
Packaging	BuildSdk.nuspec, BuildTools.proj, version.json	Include new tool in SDK, version bump to 71.0
CI/infra	DoTests.ps1, .config/dotnet-tools.json, Directory.Build.props	Run new tests, tool manifest, build config
Docs	docs/copilot/plans/ (4 files), .github/copilot-instructions.md	Planning documents, repo instructions

[jonwis]

What is this trying to do? Looks like this isn't legal - https://godbolt.org/z/TEzKEKrWG

[jevansaks]

The headers have _LARGE_INTEGER and the existing scraper.settings.rsp remaps _LARGE_INTEGER to "long" in the metadata. "It was like that when I got here", so I'm not questioning why it was done that way.

The new AST walker logic would cause _LARGE_INTEGER usage in the ASTs to show up in the public metadata as "LARGE_INTEGER", but that would be a breaking change.

So we keep the manual _LARGE_INTEGER=long mapping in the .rsp file and then this test validates that .rsp mapping overrides any AST discovered mappings.

[jevansaks]

So, to clarify, it's not trying to inject a typedef to long in the .h to be parsed, but rather this is telling the tool that scrapes the AST that LARGE_INTEGER should be output into the winmd as "long".