Commit 174e481
authored
[TrimmableTypeMap] Improve trimmable typemap build incrementality (#11764)
## Summary
Makes the **trimmable** typemap build pipeline (`_AndroidTypeMapImplementation=trimmable`, used by CoreCLR and NativeAOT) fully incremental, skipped in design‑time builds, and self‑consistent on every rebuild. Extracted as a focused, well‑tested slice from the larger #11617, and implemented on top of `main`'s current generator/task API rather than the runtime rewrite in that PR.
The bulk of this description is a deep dive into **how incrementality is achieved**, because the pipeline has several non‑obvious moving parts (two generator passes, content‑based writes, dynamic outputs vs. `IncrementalClean`, and a post‑link Java regeneration step).
---
## Background: what the trimmable typemap generator produces
The legacy typemap implementations (`llvm-ir`, `managed`) embed the managed ↔ Java type mapping into native binaries. The **trimmable** implementation instead generates, from a scan of the app's assemblies:
- a set of small managed **TypeMap assemblies** — one `_<Assembly>.TypeMap.dll` per input assembly plus a `_Microsoft.Android.TypeMaps.dll` root — that are trimmer‑friendly (unused entries are removed by the IL linker), and
- the **Java Callable Wrapper** (`*.java`, "JCW") sources that Java needs to call back into managed code, plus `acw-map.txt` and a merged `AndroidManifest.xml`.
This is driven by the `GenerateTrimmableTypeMap` MSBuild task. The active runtime is selected by `$(_AndroidRuntime)` (`CoreCLR` or `NativeAOT`); runtime‑specific targets extend the shared pipeline (`…TypeMap.Trimmable.CoreCLR.targets`, `…TypeMap.Trimmable.NativeAOT.targets`).
---
## The pipeline (target graph)
### Shared / Debug CoreCLR & NativeAOT (no trimming)
```
CoreCompile
└─► _GenerateTrimmableTypeMap (AfterTargets="CoreCompile")
• scans @(ReferencePath) + framework/SDK assemblies + the app .dll
• writes typemap/_*.TypeMap.dll + _Microsoft.Android.TypeMaps.dll
• writes typemap/java/**/*.java (JCWs), acw-map.txt, merged AndroidManifest.xml
• prunes stale JCWs, then touches typemap/_GenerateTrimmableTypeMap.stamp
…
_ReadGeneratedTrimmableTypeMapAssemblies (reads typemap-assemblies.txt → items)
_PrepareTrimmableNativeConfigAssemblies (feeds _GeneratePackageManagerJava)
_PrepareTrimmableTypeMapAssemblies (feeds packaging / assembly store)
_GenerateJavaStubs (copies JCWs → android/src, manifest, acw-map)
└─► _CompileJava ─► _CompileToDalvik ─► packaging
```
`_GenerateJavaStubs` **overrides** the legacy target of the same name from `BuildOrder.targets`; on the trimmable path the JCWs already exist, so it only copies them into `$(IntermediateOutputPath)android/src` and wires up the manifest/`acw-map.txt`/native config.
### Release CoreCLR (`PublishTrimmed=true`) — two generator passes
```
CoreCompile ─► _GenerateTrimmableTypeMap (pre-trim: full assembly set)
…
ILLink (trim) ─► CrossGen/R2R
└─► _ComputePostTrimTrimmableTypeMapInputs (collect existing linked .dll)
└─► _GeneratePostTrimTrimmableTypeMapJavaSources (post-trim)
• regenerates JCWs from the *linked* assemblies only
• writes typemap/linked-java/**/*.java (GenerateTypeMapAssemblies=false)
• touches stamp/_GeneratePostTrimTrimmableTypeMapJavaSources.stamp
_GenerateJavaStubs (copies linked-java → android/src)
```
The post‑trim pass exists because, after trimming, the set of types that still need JCWs is a **subset** of the pre‑trim set. `_GenerateJavaStubs` sources its JCWs from `$(_TypeMapJavaStubsSourceDirectory)`, which is `typemap/linked-java` for CoreCLR + `PublishTrimmed` and `typemap/java` otherwise.
NativeAOT (`PublishTrimmed=true` as well) does **not** use the post‑trim Java pass — it feeds the pre‑trim JCWs to ILC and its own native steps — so for NativeAOT the sources stay `typemap/java`.
---
## How incrementality is achieved (deep dive)
The pipeline follows the repo's [MSBuild best practices](https://github.com/dotnet/android/blob/main/Documentation/guides/MSBuildBestPractices.md): every expensive target declares `Inputs`/`Outputs`, re‑emits its dynamic `FileWrites`, and uses stamp files where a real output can't serve as a reliable timestamp sentinel. There are six principles at work.
### 1. Stamp files are the incremental sentinels — because the real outputs are content‑addressed
`GenerateTrimmableTypeMap` writes every output with `Files.CopyIfStreamChanged` (and the per‑assembly `WriteAssembliesToDisk` additionally compares timestamps): **an output whose content did not change keeps its old timestamp**. That's exactly what you want to avoid churning downstream Java/native compilation — but it makes those files unusable as MSBuild `Outputs`, because their timestamps can be *older* than the `Inputs` even on a successful run, which makes MSBuild consider the target perpetually out‑of‑date and re‑run it every build.
So `_GenerateTrimmableTypeMap` declares a **dedicated stamp** as its sole output and touches only that stamp:
```xml
Inputs="@(ReferencePath);@(PrivateSdkAssemblies);@(FrameworkAssemblies);$(IntermediateOutputPath)$(TargetFileName);$(_AndroidManifestAbs);$(_AndroidBuildPropertiesCache)"
Outputs="$(_TrimmableTypeMapOutputStamp)"
…
<Touch Files="$(_TrimmableTypeMapOutputStamp)" AlwaysCreate="true" />
```
The stamp is always touched on a run and never touched on a skip, so the target is correctly skipped iff none of its inputs changed. The generated DLLs and `typemap-assemblies.txt` are **not** touched and are **not** declared as outputs — they are content‑addressed, and no other target consumes them as timestamp `Inputs` (verified), so touching them would be pure churn. *(This is the `@jonathanpeppers` review point.)*
> The `Inputs` deliberately include `@(PrivateSdkAssemblies)` and `@(FrameworkAssemblies)`: they can contribute managed ↔ Java mappings, so a change in them must re‑run generation.
`_GenerateJavaStubs` uses the analogous idea, but its sentinel must reflect **whichever** pass produced the JCWs it copies:
```xml
Inputs="$(_TrimmableJavaSourceStamp);@(_EnvironmentFiles)"
```
where `_TrimmableJavaSourceStamp` resolves to the **post‑trim** stamp for CoreCLR + `PublishTrimmed`, and the **pre‑trim generator** stamp otherwise. This is what lets `_GenerateJavaStubs` react to post‑trim JCW regeneration while staying incremental on no‑op builds. *(This is the `@Copilot` review point; the non‑trim default was previously the TypeMap DLL, whose timestamp is unreliable for the reason above.)*
### 2. `Inputs` must reference files that actually exist
MSBuild treats a **non‑existent `Inputs` file as "out of date"** and runs the target. The post‑trim pass originally declared `Inputs="@(ResolvedFileToPublish)"`, which also lists non‑assembly publish outputs — e.g. `…/bin/Release/<rid>/UnnamedProject.runtimeconfig.json` — whose paths do not exist when the target runs. That single phantom input made `_GeneratePostTrimTrimmableTypeMapJavaSources` run on **every** build, wiping and regenerating `linked-java` each time.
The fix factors input collection into `_ComputePostTrimTrimmableTypeMapInputs`, which keeps only `.dll` items that exist on disk:
```xml
<_PostTrimTrimmableTypeMapInputAssemblies Include="@(ResolvedFileToPublish)"
Condition=" '%(Extension)' == '.dll' and Exists('%(FullPath)') and (…RID filter…) " />
```
so the post‑trim pass is now genuinely incremental and its stamp is a trustworthy sentinel for `_GenerateJavaStubs`.
### 3. Dynamic outputs must be re‑published to `@(FileWrites)` on skipped builds
The set of generated assemblies and JCWs is data‑dependent (it comes from scanning), so the `ItemGroup`s that register those files into `@(FileWrites)` live **inside** the generating targets — and therefore don't execute on a no‑op build where those targets are skipped. If nothing re‑declares them, MSBuild's `IncrementalClean` sees the previously‑tracked files as orphaned and **deletes them** before packaging reads its inputs.
`_RecordTrimmableTypeMapFileWrites` runs unconditionally (gated only on the assemblies‑list file existing) and re‑emits the previous dynamic outputs back into `@(FileWrites)` before `IncrementalClean`:
- the generated TypeMap assemblies (read back from `typemap-assemblies.txt`),
- the pre‑trim JCWs (`typemap/java`) **and** their `android/src` copies,
- the post‑trim JCWs (`typemap/linked-java`) **and** their `android/src` copies, plus the post‑trim stamp, and
- the merged manifest, `acw-map.txt`, `ApplicationRegistration.java`, and the generator stamp.
The post‑trim entries are new in this PR: once `_GeneratePostTrimTrimmableTypeMapJavaSources` became skippable (principle 2), its `linked-java` outputs would otherwise be deleted by `IncrementalClean` on the builds where it's skipped. The globs are empty for configurations that have no post‑trim pass, so this is a no‑op there.
> Two more targets — `_PrepareTrimmableNativeConfigAssemblies` and `_PrepareTrimmableTypeMapAssemblies` — exist for the same "skipped target leaves item groups empty" reason: they re‑populate the TypeMap‑assembly item groups (used by packaging, the assembly store, and native config) from the on‑disk list via `BeforeTargets`, so incremental builds that skip generation still package correctly.
### 4. Content‑based writes + explicit stale pruning keep `android/src` exact
Because generation is content‑addressed (`CopyIfStreamChanged`) and the `android/src` copy uses `SkipUnchangedFiles="true"`, an unchanged JCW never updates a timestamp and never triggers Java recompilation.
The flip side is removal: when a JCW disappears between builds — a type deleted from source, or **trimmed away** on the `PublishTrimmed` path — its `android/src` copy must not linger, or it would be compiled and packaged. Both generator passes report the JCWs they no longer produce as `DeletedJavaFiles`; the owning target mirrors each deletion into the `android/src` copy and, if anything was deleted, deletes `$(_AndroidCompileJavaStampFile)` so `_CompileJava` re‑runs and drops the stale `.class`:
```xml
<Delete Files="@(_DeletedCopiedJavaFiles)" />
<Delete Files="$(_AndroidCompileJavaStampFile)" Condition=" '@(_DeletedCopiedJavaFiles->Count())' != '0' " />
```
The two passes compute the deleted set differently because of how each manages its output directory:
- **Pre‑trim** (`_GenerateTrimmableTypeMap`, writing `typemap/java`): the task scans the output dir and deletes any `*.java` the current pass didn't produce.
- **Post‑trim** (`_GeneratePostTrimTrimmableTypeMapJavaSources`, writing `typemap/linked-java` with `CleanJavaSourceOutputDirectory=true`): the dir is wiped before regeneration, so the task snapshots the previous `*.java` set *before* the wipe and reports `previous − regenerated`. This keeps the deletion precise — only files the generator itself previously produced are ever removed from `android/src`, never unrelated sources like `ApplicationRegistration.java`.
The net invariant is two‑directional: **`android/src` contains exactly the JCWs the active pass produces** (`typemap/linked-java` for `PublishTrimmed`, `typemap/java` otherwise) — no missing files (copied by `_GenerateJavaStubs`) and no stale files (pruned via `DeletedJavaFiles`). Exercised directly by the `…KeepsAndroidSrcConsistentWithLinkedJava` and `…DeletesStaleAndroidSrcWhenLinkedJavaShrinks` tests.
### 5. The generator is skipped in design‑time builds and runs exactly once
`_GenerateTrimmableTypeMap` is gated on `'$(DesignTimeBuild)' != 'true'`: in a design‑time build, project references may resolve to target paths that aren't produced when `SkipCompilerExecution=true`, and the output isn't needed for IDE information. Combined with the existing `'$(_OuterIntermediateOutputPath)' == ''` guard (which skips inner per‑RID builds), the generator runs exactly once per outer build.
### 6. Two‑pass consistency under trimming
For CoreCLR + `PublishTrimmed`, correctness depends on the pre‑trim and post‑trim sentinels composing cleanly:
- **No‑op rebuild** — both passes are skipped (their inputs are unchanged); `_GenerateJavaStubs` keys off the stable post‑trim stamp and is also skipped. *(Newly true; previously the post‑trim pass ran every build.)*
- **Managed source change** — the pre‑trim generator re‑runs (its inputs changed) and re‑linking changes the linked assemblies, so the post‑trim pass re‑runs too; `_GenerateJavaStubs` re‑copies.
- **Trim‑only change** (e.g. a change that alters the linked output without touching the pre‑trim inputs) — the linked `.dll` change re‑runs the post‑trim pass, which touches the post‑trim stamp, so `_GenerateJavaStubs` re‑runs even though the pre‑trim generator did not. This is precisely the staleness the `@Copilot` review flagged, and is why `_GenerateJavaStubs` must key off `_TrimmableJavaSourceStamp` rather than the generator stamp alone.
---
## What changed in this PR
| Area | Change |
| ---- | ------ |
| `…TypeMap.Trimmable.targets` | Stamp sentinel for `_GenerateTrimmableTypeMap` (sole `Output`, sole `Touch`); expanded `Inputs` (`PrivateSdkAssemblies`/`FrameworkAssemblies`); design‑time‑build skip; `SkipUnchangedFiles` JCW copy; stale‑JCW deletion wiring; `_GenerateJavaStubs` keyed off `_TrimmableJavaSourceStamp`; post‑trim outputs re‑emitted in `_RecordTrimmableTypeMapFileWrites`. |
| `…TypeMap.Trimmable.CoreCLR.targets` | New `_ComputePostTrimTrimmableTypeMapInputs` (existing‑`.dll` inputs) so `_GeneratePostTrimTrimmableTypeMapJavaSources` is incremental; post‑trim pass now mirrors `DeletedJavaFiles` into `android/src` + busts the Java compile stamp. |
| `Tasks/GenerateTrimmableTypeMap.cs` | New `DeletedJavaFiles` output + `DeleteStaleJavaSources()`; in the `CleanJavaSourceOutputDirectory` (post‑trim) case it snapshots the prior JCW set before the wipe and reports `previous − regenerated`. |
| `Microsoft.Android.Sdk.TrimmableTypeMap/README.md` | Documents the pipeline and its incrementality design. |
| `TrimmableTypeMapBuildTests.cs` | Tests for stale‑JCW pruning (both directions), updated‑JCW copying, the `android/src` ↔ `linked-java` invariant, post‑trim no‑op incrementality, and stale `android/src` removal when `linked-java` shrinks. |
### Review feedback addressed
- **@Copilot** (×2): `_GenerateJavaStubs` now keys off `$(_TrimmableJavaSourceStamp)` (post‑trim stamp for `PublishTrimmed`, generator stamp otherwise), and the non‑trim default of `_TrimmableJavaSourceStamp` was changed from the TypeMap DLL to the generator stamp. The underlying reason the bare generator stamp was insufficient — a non‑incremental post‑trim pass — is fixed too.
- **@jonathanpeppers**: `_GenerateTrimmableTypeMap` declares only the stamp as its `Outputs` and touches only the stamp; the DLLs/list are content‑addressed and not consumed as timestamp inputs elsewhere.
---
## Scope notes (and a little beyond)
- All changes are gated on `_AndroidTypeMapImplementation == 'trimmable'`; non‑trimmable (`llvm-ir`/`managed`) builds are unaffected.
- Unlike #11617, this PR **keeps** `main`'s CoreCLR post‑trim `linked-java` machinery intact and instead makes it incremental.
- **Beyond this PR**, the post‑trim pass still regenerates `linked-java` wholesale (`CleanJavaSourceOutputDirectory=true`) when it runs; it now also reports the JCWs it dropped so `android/src` stays exact. A further optimization would replace the wholesale clean with content‑based writes + in‑place stale pruning (as the pre‑trim pass does), letting unchanged `linked-java` JCWs keep stable timestamps across a post‑trim run. Left as a follow‑up.
---
## Testing
Verified locally against a Debug local SDK.
**Host (`Xamarin.Android.Build.Tests`, full `TrimmableTypeMapBuildTests`): 24 passed, 2 skipped, 0 failed**, including:
- `…IncrementalBuild` (Debug CoreCLR, Release CoreCLR, Release NativeAOT)
- `…DeletesStaleGeneratedJavaSourcesAndCopies`, `…CopiesUpdatedGeneratedJavaSources`
- `…PublishTrimmed_KeepsAndroidSrcConsistentWithLinkedJava` (the `android/src` ↔ `linked-java` invariant)
- `…PublishTrimmed_DeletesStaleAndroidSrcWhenLinkedJavaShrinks` (a JCW dropped from `linked-java` is removed from `android/src`)
- `…PublishTrimmed_PostTrimJavaGenerationIsIncremental` (no‑op rebuild skips post‑trim and `_GenerateJavaStubs`)
- `…Succeeds`, `…ArrayRankChangeRegeneratesTypeMap`, `…DoesNotHitCopyIfChangedMismatch`, R2R/multi‑RID packaging tests
**Emulator (`MSBuildDeviceIntegration`, API 36 / arm64):** `DotNetRun` deploy+run for trimmable CoreCLR (Debug + Release) and NativeAOT (Release), and `TrimmableTypeMapInheritedVirtualOverrideUsesCorrectUco`.1 parent 222aecd commit 174e481
5 files changed
Lines changed: 545 additions & 18 deletions
File tree
- src
- Microsoft.Android.Sdk.TrimmableTypeMap
- Xamarin.Android.Build.Tasks
- Microsoft.Android.Sdk/targets
- Tasks
- Tests/Xamarin.Android.Build.Tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
Lines changed: 32 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
51 | 66 | | |
52 | 67 | | |
| 68 | + | |
53 | 69 | | |
54 | 70 | | |
55 | | - | |
| 71 | + | |
56 | 72 | | |
57 | | - | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | 73 | | |
62 | 74 | | |
63 | 75 | | |
| |||
73 | 85 | | |
74 | 86 | | |
75 | 87 | | |
| 88 | + | |
76 | 89 | | |
77 | 90 | | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
78 | 101 | | |
79 | 102 | | |
80 | 103 | | |
81 | 104 | | |
| 105 | + | |
| 106 | + | |
82 | 107 | | |
83 | 108 | | |
84 | 109 | | |
85 | 110 | | |
86 | 111 | | |
87 | 112 | | |
| 113 | + | |
| 114 | + | |
88 | 115 | | |
89 | 116 | | |
90 | 117 | | |
| |||
0 commit comments