Skip to content

Commit 452e2fb

Browse files
tyrielvtyrielv
committed
Release pipeline: route vcpkg through Terrapin asset cache
The official release pipeline has been failing since libgit2 was statically linked via vcpkg, because the pool's build agents do not have direct public-internet access. vcpkg was reaching out to sourceforge.net (and other origins) for libgit2 transitive dependencies (pcre, zlib, http_parser, ...) which the network layer blocked, causing WinHttpSendRequest 12029 errors. Route vcpkg asset downloads through the internal Terrapin cache (the canonical CI asset-caching service). This is opt-in. External contributors and the public GitHub Actions PR-validation workflow (.github/workflows/build.yaml) continue to use the standard vcpkg flow with direct downloads from public origins. No internal feed authentication is required to build VFS for Git from a fresh clone, and the repo source code intentionally contains no msbuild SDK references that would pull internal packages into a default restore. Changes: Directory.Build.props Add IsLocalBuild and UseTerrapinAssetCache properties, both defaulted to false. IsLocalBuild controls Terrapin upload-on-fetch (allowed on dev boxes only); UseTerrapinAssetCache opts the build into the Terrapin asset-cache machinery (release pipeline only). Directory.Build.targets Build a --x-asset-sources string that combines the Terrapin retrieval script with x-block-origin, gated on UseTerrapinAssetCache=true, and append it to both vcpkg install invocations. Add a guard <Error> that requires the pipeline to supply TerrapinRetrievalToolPath up front when asset caching is enabled. Also validate static git2.lib (was only checking dynamic git2.dll). .azure-pipelines/official-release-nuget.config (new) Pipeline-only NuGet config that points at an internal feed. Used solely by a one-off `nuget install` of Microsoft.Build.Vcpkg, not by any csproj restore. Not consulted by external contributors or by GitHub Actions. .azure-pipelines/release.yml Add NuGetAuthenticate@1 task. Add a NuGetCommand@2 task that out-of-band installs the internal Microsoft.Build.Vcpkg package (which ships TerrapinRetrievalTool.exe) into $(Agent.TempDirectory). Add an explicit "Restore vcpkg native libraries (Terrapin cache)" step that invokes _RestoreVcpkgDependencies on GVFS.Common.csproj with -p:UseTerrapinAssetCache=true and TerrapinRetrievalToolPath pointing at the extracted tool. Build.bat's own vcpkg install step then skips because the libs are already present. Assisted-by: Claude Opus 4.7 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
1 parent c53aea6 commit 452e2fb

4 files changed

Lines changed: 113 additions & 3 deletions

File tree

‎.azure-pipelines/official-release-nuget.config‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
<?xml version="1.0" encoding="utf-8"?>
2+
<!--
3+
Official-release NuGet configuration. Pipeline-only.
4+
5+
Used by the official release pipeline ONLY, supplied at build time via
6+
-ConfigFile to a one-off `nuget install Microsoft.Build.Vcpkg` command
7+
that fetches the Terrapin retrieval tool. The root nuget.config (the
8+
one external contributors and the public GitHub Actions workflow use)
9+
does NOT reference this internal feed, so building VFS for Git from a
10+
fresh clone requires no internal authentication.
11+
12+
Why this file exists:
13+
Microsoft.Build.Vcpkg is an internal-only NuGet package that ships
14+
the proprietary TerrapinRetrievalTool.exe. The release pipeline
15+
downloads the package out-of-band (i.e. not via msbuild SDK
16+
resolution, which would force the internal feed into the root
17+
nuget.config and expose it to every consumer) and passes the path to
18+
the extracted retrieval tool via -p:TerrapinRetrievalToolPath. vcpkg
19+
then routes its asset downloads through the Terrapin cache, because
20+
the release pipeline's build agents have x-block-origin enforced at
21+
the network layer and cannot fetch from public origins
22+
(sourceforge.net, github.com release artifacts, etc.).
23+
-->
24+
<configuration>
25+
<packageSources>
26+
<clear />
27+
<add key="GitClient" value="https://pkgs.dev.azure.com/mseng/1ES/_packaging/GitClient/nuget/v3/index.json" />
28+
</packageSources>
29+
</configuration>

‎.azure-pipelines/release.yml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,9 @@ extends:
111111
inputs:
112112
versionSpec: '6.x'
113113

114+
- task: NuGetAuthenticate@1
115+
displayName: 'Authenticate to internal NuGet feed (for Microsoft.Build.Vcpkg)'
116+
114117
- task: PowerShell@2
115118
displayName: 'Install VS C++ workload (NativeAOT prerequisite)'
116119
inputs:
@@ -121,6 +124,37 @@ extends:
121124
inputs:
122125
filePath: $(Build.SourcesDirectory)\.azure-pipelines\scripts\enable-projfs.ps1
123126

127+
# Download the Microsoft.Build.Vcpkg NuGet package out-of-band so we
128+
# can hand the build a path to TerrapinRetrievalTool.exe via
129+
# -p:TerrapinRetrievalToolPath. The package is pulled from an
130+
# internal NuGet feed (see .azure-pipelines/official-release-nuget.config).
131+
# Downloading it this way -- rather than as an msbuild Sdk import --
132+
# keeps the internal feed out of the root nuget.config that
133+
# external contributors and the public GitHub Actions workflow see.
134+
- task: NuGetCommand@2
135+
displayName: 'Download Microsoft.Build.Vcpkg package (Terrapin retrieval tool)'
136+
inputs:
137+
command: custom
138+
arguments: 'install Microsoft.Build.Vcpkg -Version 2026.5.25.434-aa40adda53 -ConfigFile $(Build.SourcesDirectory)\.azure-pipelines\official-release-nuget.config -OutputDirectory $(Agent.TempDirectory)\nuget-internal -ExcludeVersion -DirectDownload -NonInteractive'
139+
140+
# Restore vcpkg native dependencies through the Terrapin asset
141+
# cache (the release pipeline's build agents have x-block-origin
142+
# enforced and cannot download from the public internet). Runs the
143+
# _RestoreVcpkgDependencies MSBuild target with
144+
# UseTerrapinAssetCache=true and TerrapinRetrievalToolPath pointing
145+
# at the binary extracted by the previous step. vcpkg downloads
146+
# then route through https://vcpkg.storage.devpackages.microsoft.io.
147+
# Build.bat's own vcpkg install step then skips because the libs
148+
# are already present.
149+
- script: |
150+
dotnet build "$(Build.SourcesDirectory)\GVFS\GVFS.Common\GVFS.Common.csproj" ^
151+
/t:_RestoreVcpkgDependencies ^
152+
-c $(BuildConfiguration) ^
153+
-p:UseTerrapinAssetCache=true ^
154+
-p:TerrapinRetrievalToolPath=$(Agent.TempDirectory)\nuget-internal\Microsoft.Build.Vcpkg\trt\TerrapinRetrievalTool.exe ^
155+
-v:detailed
156+
displayName: 'Restore vcpkg native libraries (Terrapin cache)'
157+
124158
- script: |
125159
$(Build.SourcesDirectory)\scripts\Build.bat ^
126160
$(BuildConfiguration) ^

‎Directory.Build.props‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,25 @@
1818
but our .NET projects do.
1919
-->
2020
<RestorePackagesConfig>true</RestorePackagesConfig>
21+
22+
<!--
23+
Set to true when building locally to allow Terrapin to upload missing
24+
vcpkg assets to the shared cache. CI agents must keep this false so
25+
vcpkg uses x-block-origin (fail fast instead of attempting direct
26+
downloads from the public internet, which is blocked at the agent
27+
network layer).
28+
-->
29+
<IsLocalBuild Condition="'$(IsLocalBuild)' == ''">false</IsLocalBuild>
30+
31+
<!--
32+
Set to true on the official release pipeline to route vcpkg asset
33+
downloads through the internal Terrapin cache via the
34+
Microsoft.Build.Vcpkg SDK (an internal-only package). Defaults to
35+
false so external contributors and the public GitHub Actions
36+
PR-validation workflow continue to use the standard vcpkg flow
37+
with direct downloads from the public internet.
38+
-->
39+
<UseTerrapinAssetCache Condition="'$(UseTerrapinAssetCache)' == ''">false</UseTerrapinAssetCache>
2140
</PropertyGroup>
2241

2342
<!-- Managed project properties -->

‎Directory.Build.targets‎

Lines changed: 31 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,22 @@
3232
<_VcpkgManifestFile>$(RepoSrcPath)vcpkg.json</_VcpkgManifestFile>
3333
<_VcpkgConfigFile>$(RepoSrcPath)vcpkg-configuration.json</_VcpkgConfigFile>
3434
<_VcpkgStampFile>$(RepoOutPath)vcpkg_installed\.msbuildstamp</_VcpkgStampFile>
35+
36+
<!--
37+
Terrapin asset caching for vcpkg downloads (only when explicitly
38+
enabled by the release pipeline). -a $(IsLocalBuild) controls whether
39+
the retrieval script may upload missing assets back to the Terrapin
40+
cache: true on dev boxes (write permission available), false on CI
41+
agents (read-only; x-block-origin causes immediate failure rather
42+
than attempting public-internet downloads).
43+
44+
$(TerrapinRetrievalToolPath) is supplied by the release pipeline
45+
after it downloads the internal Microsoft.Build.Vcpkg NuGet package
46+
that ships the retrieval tool. The repo intentionally does not
47+
reference that internal package directly; the pipeline is the only
48+
bridge between this repo and the internal feed.
49+
-->
50+
<_VcpkgAssetSources Condition="'$(UseTerrapinAssetCache)' == 'true'">"--x-asset-sources=x-script,$(TerrapinRetrievalToolPath) -b https://vcpkg.storage.devpackages.microsoft.io/artifacts/ -a $(IsLocalBuild) -p {url} -s {sha512} -d {dst};x-block-origin"</_VcpkgAssetSources>
3551
</PropertyGroup>
3652

3753
<ItemGroup Condition="'$(MSBuildProjectExtension)' == '.csproj'">
@@ -70,13 +86,25 @@
7086
<Message Importance="high" Text="[vcpkg] Installing native dependencies to $(RepoOutPath)vcpkg_installed\" />
7187

7288
<MakeDir Directories="$(RepoOutPath)vcpkg_installed\" />
73-
<Exec Command="&quot;$(_VcpkgExe)&quot; install --x-wait-for-lock --triplet x64-windows-static-aot --x-install-root=&quot;$(RepoOutPath)vcpkg_installed\static&quot; $(_VcpkgManifestArg)"
89+
90+
<!--
91+
Guard: if asset caching was requested but the tool path was not
92+
supplied, the asset-source string would be malformed and vcpkg
93+
would fail with a confusing error. Surface the misconfiguration up
94+
front.
95+
-->
96+
<Error Condition="'$(UseTerrapinAssetCache)' == 'true' and ('$(TerrapinRetrievalToolPath)' == '' or !Exists('$(TerrapinRetrievalToolPath)'))"
97+
Text="UseTerrapinAssetCache=true but TerrapinRetrievalToolPath is empty or does not exist. The release pipeline must pre-download the Microsoft.Build.Vcpkg NuGet package and pass -p:TerrapinRetrievalToolPath=&lt;path-to-TerrapinRetrievalTool.exe&gt;." />
98+
99+
<Exec Command="&quot;$(_VcpkgExe)&quot; install --x-wait-for-lock --triplet x64-windows-static-aot --x-install-root=&quot;$(RepoOutPath)vcpkg_installed\static&quot; $(_VcpkgManifestArg) $(_VcpkgAssetSources)"
74100
StandardOutputImportance="High" StandardErrorImportance="High" />
75-
<Exec Command="&quot;$(_VcpkgExe)&quot; install --x-wait-for-lock --triplet x64-windows-dynamic --x-install-root=&quot;$(RepoOutPath)vcpkg_installed\dynamic&quot; $(_VcpkgManifestArg)"
101+
<Exec Command="&quot;$(_VcpkgExe)&quot; install --x-wait-for-lock --triplet x64-windows-dynamic --x-install-root=&quot;$(RepoOutPath)vcpkg_installed\dynamic&quot; $(_VcpkgManifestArg) $(_VcpkgAssetSources)"
76102
StandardOutputImportance="High" StandardErrorImportance="High" />
77103

78104
<Error Condition="!Exists('$(RepoOutPath)vcpkg_installed\dynamic\x64-windows-dynamic\bin\git2.dll')"
79-
Text="vcpkg install completed but expected output files are missing. Check vcpkg output above for errors." />
105+
Text="vcpkg install completed but git2.dll (dynamic) is missing. Check vcpkg output above for errors." />
106+
<Error Condition="!Exists('$(RepoOutPath)vcpkg_installed\static\x64-windows-static-aot\lib\git2.lib')"
107+
Text="vcpkg install completed but git2.lib (static) is missing. Check vcpkg output above for errors." />
80108

81109
<Touch Files="$(_VcpkgStampFile)" AlwaysCreate="true" />
82110
</Target>

0 commit comments

Comments
 (0)