Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 15 additions & 9 deletions docs/api/Inside-the-Template-Engine.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ designed to separate gathering, instantiating and processing templates.

# Overview of the subsystems

## [IEngineEnvironmentSettings](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.Abstractions/IEngineEnvironmentSettings.cs)
## [IEngineEnvironmentSettings](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.Abstractions/IEngineEnvironmentSettings.cs)

This is responsible for holding all properties of the environment.

Expand Down Expand Up @@ -69,7 +69,7 @@ and

`ComponentManager` is responsible for loading [components](#components) provided by the host from
`ITemplateEngineHost.BuiltIns`. Additional components can be added via
`IComponentManager.AddComponent`. Similar to templates, components are also
`IComponentManager.AddComponent`. Similar to templates, components are also
dynamically loaded from template packages during scanning. `ComponentManager`
implementation is not publicly accessible; however, it is used when default
implementation of `IEngineEnvironmentSettings` is used.
Expand Down Expand Up @@ -120,9 +120,9 @@ instantiate the class when needed.

### Supplying parameters conditions results

It is possible to supply evaluation results of parameters conditions when instantiating template via Edge API [`TemplateCreator.InstantiateAsync`](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.Edge/Template/TemplateCreator.cs#L89). Example use case is instantiation from Visual Studio host, that will leverage condition evaluator integrated within the New Project Dialog.
It is possible to supply evaluation results of parameters conditions when instantiating template via Edge API [`TemplateCreator.InstantiateAsync`](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.Edge/Template/TemplateCreator.cs#L89). Example use case is instantiation from Visual Studio host, that will leverage condition evaluator integrated within the New Project Dialog.

This can be achieved by passing the structured [`InputDataSet`](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.Edge/Template/InputDataSet.cs) argument that is populated with [`EvaluatedInputParameterData`](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.Edge/Template/EvaluatedInputParameterData.cs) objects for evaluated parameters.
This can be achieved by passing the structured [`InputDataSet`](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.Edge/Template/InputDataSet.cs) argument that is populated with [`EvaluatedInputParameterData`](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.Edge/Template/EvaluatedInputParameterData.cs) objects for evaluated parameters.

It is currently not possible to provide just partial external evaluation - meaning that the template engine evaluates either all the parameter conditions or none. If the [`InputDataSet`](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.Edge/Template/InputDataSet.cs) collection contains at least one [`EvaluatedInputParameterData`](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.Edge/Template/EvaluatedInputParameterData.cs) element, results of all parameter conditions are expected to be passed.

Expand Down Expand Up @@ -178,7 +178,7 @@ be split into two main categories:

- mount points

## Template package providers
## Template package providers

Template package providers responsible for providing template packages the
template engine needs to manage. There are two kinds of providers:
Expand Down Expand Up @@ -243,6 +243,12 @@ Template engine provides two installer implementations:
You can use them when implementing your own providers, they are available from
IComponentManager.

When package verification is enabled, downloaded signed NuGet packages are
verified using `dotnet nuget verify --all` before installation. Unsigned packages
are accepted only when NuGet's `signatureValidationMode` is `accept` and the
source does not declare `allRepositorySigned=true`. A client policy of `require`
or a repository signing requirement rejects unsigned packages.

## [Mount point](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.Abstractions/Mount/IMountPoint.cs)

Template package location is represented by mount point URI. Mount point that
Expand All @@ -268,14 +274,14 @@ Template engine host works with constraint using `TemplateConstraintManager` cla

The template may define `bind` symbols. Those symbols might be bound to values provided externally using bind symbol source components.
Default sources are:
- host provided parameters - uses prefix `host:`. Binds symbol to parameters provided by the host.
- host provided parameters - uses prefix `host:`. Binds symbol to parameters provided by the host.
- environment variables - uses prefix `env:`. Binds symbol to environment variables.

These components are defined in `Microsoft.TemplateEngine.Edge` and are part of mandatory components collection to be loaded.

If the `binding` does not have prefix, or prefixed value cannot be evaluated the evaluation without prefix is performed as the fallback.
If the `binding` does not have prefix, or prefixed value cannot be evaluated the evaluation without prefix is performed as the fallback.
In case there are more than one component which returns the value, the component priority will be used to define the value to be bounded. In case of same priorities, the evaluation results in error.
The template author needs to explicitly specify the prefix in this case. It is possible to restrict usage of prefix with setting `IBindSymbolSource.RequiresPrefixMatch` to `true`.
The template author needs to explicitly specify the prefix in this case. It is possible to restrict usage of prefix with setting `IBindSymbolSource.RequiresPrefixMatch` to `true`.


## Registering the components
Expand Down Expand Up @@ -325,7 +331,7 @@ It is possible to register additional components to in the following way:
Note that components are not persisted, so they should be added each time
EngineEnvironmentSettings instance is created.

# Microsoft.TemplateEngine.IDE
# Microsoft.TemplateEngine.IDE

The package provides
[Bootstrapper](https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.IDE/Bootstrapper.cs)
Expand Down
4 changes: 2 additions & 2 deletions eng/Version.Details.props
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,12 @@ This file should be imported by eng/Versions.props
<PropertyGroup>
<!-- dotnet-dotnet dependencies -->
<MicrosoftBclAsyncInterfacesPackageVersion>10.0.11</MicrosoftBclAsyncInterfacesPackageVersion>
<MicrosoftDotNetArcadeSdkPackageVersion>10.0.0-beta.26512.108</MicrosoftDotNetArcadeSdkPackageVersion>
<MicrosoftDotNetArcadeSdkPackageVersion>10.0.0-beta.26570.123</MicrosoftDotNetArcadeSdkPackageVersion>
<MicrosoftExtensionsDependencyInjectionAbstractionsPackageVersion>10.0.11</MicrosoftExtensionsDependencyInjectionAbstractionsPackageVersion>
<MicrosoftExtensionsLoggingPackageVersion>10.0.11</MicrosoftExtensionsLoggingPackageVersion>
<MicrosoftExtensionsLoggingAbstractionsPackageVersion>10.0.11</MicrosoftExtensionsLoggingAbstractionsPackageVersion>
<MicrosoftExtensionsLoggingConsolePackageVersion>10.0.11</MicrosoftExtensionsLoggingConsolePackageVersion>
<SystemCommandLinePackageVersion>2.0.14</SystemCommandLinePackageVersion>
<SystemCommandLinePackageVersion>2.0.15</SystemCommandLinePackageVersion>
<SystemDiagnosticsDiagnosticSourcePackageVersion>10.0.11</SystemDiagnosticsDiagnosticSourcePackageVersion>
<SystemFormatsAsn1PackageVersion>10.0.11</SystemFormatsAsn1PackageVersion>
<SystemIOPipelinesPackageVersion>10.0.11</SystemIOPipelinesPackageVersion>
Expand Down
12 changes: 6 additions & 6 deletions eng/Version.Details.xml
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
<?xml version="1.0" encoding="utf-8"?>
<Dependencies>
<Source Uri="https://github.com/dotnet/dotnet" Mapping="templating" Sha="f009a9309c13506b779f3163c50c61381627a1e6" BarId="336211" />
<Source Uri="https://github.com/dotnet/dotnet" Mapping="templating" Sha="e00bf123e3b50b162a3811740bf9fe093efc966d" BarId="342569" />
<ProductDependencies>
<Dependency Name="System.CommandLine" Version="2.0.14">
<Uri>https://github.com/dotnet/dotnet</Uri>
<Sha>f009a9309c13506b779f3163c50c61381627a1e6</Sha>
<Dependency Name="System.CommandLine" Version="2.0.15">
<Uri>https://dev.azure.com/dnceng/internal/_git/dotnet-dotnet</Uri>
<Sha>8e0f8ed2312954727f8919d7d1c2f78b483b4ee7</Sha>
</Dependency>
</ProductDependencies>
<ToolsetDependencies>
<Dependency Name="Microsoft.DotNet.Arcade.Sdk" Version="10.0.0-beta.26512.108">
<Dependency Name="Microsoft.DotNet.Arcade.Sdk" Version="10.0.0-beta.26570.123">
<Uri>https://github.com/dotnet/dotnet</Uri>
<Sha>f009a9309c13506b779f3163c50c61381627a1e6</Sha>
<Sha>e00bf123e3b50b162a3811740bf9fe093efc966d</Sha>
</Dependency>
<!-- Dependencies required for source build. We'll still update manually -->
<Dependency Name="System.Formats.Asn1" Version="10.0.11">
Expand Down
2 changes: 1 addition & 1 deletion eng/Versions.props
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<Project>
<Import Project="Version.Details.props" Condition="Exists('Version.Details.props')" />
<PropertyGroup>
<VersionPrefix>10.0.403</VersionPrefix>
<VersionPrefix>10.0.406</VersionPrefix>
<!-- When StabilizePackageVersion is set to 'true', this branch will produce stable outputs for 'Shipping' packages.
NOTE: This repository stabilizes via VMR and this should not be set except in dev scenarios -->
<StabilizePackageVersion Condition="'$(StabilizePackageVersion)' == ''">true</StabilizePackageVersion>
Expand Down
103 changes: 50 additions & 53 deletions eng/common/Get-GitHubAppToken.ps1
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
# Mints a short-lived GitHub App installation access token by signing a JWT
# with an RSA private key (RS256). The signed JWT is exchanged with the GitHub
# API for a token scoped to a single installation.
# via Azure Key Vault's RSA signing operation (RS256). The private key never
# leaves Key Vault. The JWT is exchanged for a single installation's token.
#
# Requirements:
# - A GitHub App ID and PEM private key stored as Azure Key Vault secrets.
# - The federated Azure service connection running this script must have
# `Get` access to those two secrets.
# - The GitHub App's private key imported as a Key Vault RSA key.
# - Read access to the configured App-ID secret projection.
# - The federated Azure service connection must have the `Sign` action on
# that key. Reading private-key secrets is neither required nor supported.
# - The App must be installed on the target organization/account
# (`InstallationOwner`) with the permissions/repositories it needs.
#
Expand All @@ -14,17 +15,19 @@

[CmdletBinding()]
param(
# Name of the Key Vault holding the GitHub App credentials.
# Key Vault holding the App-ID projection and RSA signing key.
[Parameter(Mandatory = $true)]
[string] $KeyVaultName,

# Secret Manager projection containing the GitHub App ID.
# Name of the RSA signing key, not a Key Vault secret.
[Parameter(Mandatory = $true)]
[string] $AppIdSecretName,
[ValidateNotNullOrEmpty()]
[string] $KeyName,

# Secret Manager projection containing the PEM private key.
# Secret Manager projection containing the GitHub App's public ID.
[Parameter(Mandatory = $true)]
[string] $AppPrivateKeySecretName,
[ValidateNotNullOrEmpty()]
[string] $AppIdSecretName,

# Login of the organization or user account whose installation we should
# mint the token for (e.g. `dotnet`, `microsoft`).
Expand All @@ -51,55 +54,33 @@ function ConvertTo-Base64Url([byte[]] $bytes) {
return [Convert]::ToBase64String($bytes).TrimEnd('=').Replace('+', '-').Replace('/', '_')
}

Write-Host "Reading GitHub App ID from secret '$AppIdSecretName' in vault '$KeyVaultName'..."
$previousNativeCommandErrorPreference = $PSNativeCommandUseErrorActionPreference
try {
# Azure CLI can emit non-fatal Python warnings to stderr.
$PSNativeCommandUseErrorActionPreference = $false
$keyVaultAccessToken = az account get-access-token `
# az keyvault secret show fails for this projection with Errno 22; az rest keeps authentication in Azure CLI.
$escapedSecretName = [Uri]::EscapeDataString($AppIdSecretName)
$appId = az rest `
--method get `
--url "https://$KeyVaultName.vault.azure.net/secrets/${escapedSecretName}?api-version=7.4" `
--resource https://vault.azure.net `
--query accessToken `
--query value `
--output tsv `
--only-show-errors
$tokenExitCode = $LASTEXITCODE
$secretExitCode = $LASTEXITCODE
}
catch {
Write-PipelineTelemetryError -Category 'Build' -Message "Failed to acquire an Azure Key Vault access token: $_"
Write-PipelineTelemetryError -Category 'Build' -Message "Failed to read App ID secret '$AppIdSecretName' from vault '$KeyVaultName' with Azure CLI: $_"
exit 1
}
finally {
$PSNativeCommandUseErrorActionPreference = $previousNativeCommandErrorPreference
}
if ($tokenExitCode -ne 0 -or [string]::IsNullOrWhiteSpace($keyVaultAccessToken)) {
Write-PipelineTelemetryError -Category 'Build' -Message "'az account get-access-token' exited with code $tokenExitCode while acquiring an Azure Key Vault access token."
if ($secretExitCode -ne 0 -or [string]::IsNullOrWhiteSpace($appId)) {
Write-PipelineTelemetryError -Category 'Build' -Message "'az rest' failed with exit code $secretExitCode while reading App ID secret '$AppIdSecretName' from vault '$KeyVaultName'. Verify the service connection can read this App-ID projection."
exit 1
}

function Get-KeyVaultSecret([string] $SecretName) {
# Use the data-plane REST API because `az keyvault secret show` can fail
# with Errno 22 on hosted Windows agents when reading these projections.
$escapedSecretName = [Uri]::EscapeDataString($SecretName)
$secretUri = "https://$KeyVaultName.vault.azure.net/secrets/$escapedSecretName`?api-version=7.4"
try {
$response = Invoke-RestMethod `
-Uri $secretUri `
-Headers @{ Authorization = "Bearer $keyVaultAccessToken" } `
-Method Get
}
catch {
Write-PipelineTelemetryError -Category 'Build' -Message "Failed to read secret '$SecretName' from vault '$KeyVaultName': $_. Verify the secret exists and the service connection has 'Key Vault Secrets User' access to it."
exit 1
}
if ([string]::IsNullOrWhiteSpace($response.value)) {
Write-PipelineTelemetryError -Category 'Build' -Message "Secret '$SecretName' in vault '$KeyVaultName' is empty."
exit 1
}
return [string] $response.value
}

Write-Host "Reading GitHub App credentials from vault '$KeyVaultName'..."
$appId = Get-KeyVaultSecret $AppIdSecretName
$privateKey = Get-KeyVaultSecret $AppPrivateKeySecretName

# Build JWT header and payload. Use [ordered] hashtables so JSON
# serialization is deterministic.
$jwtHeader = [ordered]@{
Expand All @@ -125,23 +106,34 @@ finally {
$sha256.Dispose()
}

Write-Host 'Signing JWT with the GitHub App private key...'
$rsa = [System.Security.Cryptography.RSA]::Create()
$digestBase64 = [Convert]::ToBase64String($digestBytes)
Write-Host "Signing JWT with key '$KeyName' in vault '$KeyVaultName'..."
$previousNativeCommandErrorPreference = $PSNativeCommandUseErrorActionPreference
try {
$rsa.ImportFromPem($privateKey)
$signatureBytes = $rsa.SignHash(
$digestBytes,
[System.Security.Cryptography.HashAlgorithmName]::SHA256,
[System.Security.Cryptography.RSASignaturePadding]::Pkcs1)
$signatureUrl = ConvertTo-Base64Url $signatureBytes
# Azure CLI can emit non-fatal Python warnings to stderr.
$PSNativeCommandUseErrorActionPreference = $false
$signatureBase64 = az keyvault key sign `
--vault-name $KeyVaultName `
--name $KeyName `
--algorithm RS256 `
--digest $digestBase64 `
--query signature `
--output tsv `
--only-show-errors
$signExitCode = $LASTEXITCODE
}
catch {
Write-PipelineTelemetryError -Category 'Build' -Message "Failed to sign the GitHub App JWT with the supplied private key: $_"
Write-PipelineTelemetryError -Category 'Build' -Message "Failed to sign the JWT via Key Vault (key '$KeyName', vault '$KeyVaultName'): $_. Verify the service connection has the Sign action on the key."
exit 1
}
finally {
$rsa.Dispose()
$PSNativeCommandUseErrorActionPreference = $previousNativeCommandErrorPreference
}
if ($signExitCode -ne 0 -or [string]::IsNullOrWhiteSpace($signatureBase64)) {
Write-PipelineTelemetryError -Category 'Build' -Message "'az keyvault key sign' exited with code $signExitCode for key '$KeyName' in vault '$KeyVaultName'. Verify the service connection has the Sign action on the key."
exit 1
}
$signatureUrl = $signatureBase64.Trim().TrimEnd('=').Replace('+', '-').Replace('/', '_')
$jwt = "$signingInput.$signatureUrl"

$headers = @{
Expand Down Expand Up @@ -198,6 +190,11 @@ catch {
exit 1
}

if ([string]::IsNullOrWhiteSpace($tokenResponse.token)) {
Write-PipelineTelemetryError -Category 'Build' -Message "GitHub returned an empty installation token for '$InstallationOwner'."
exit 1
}

Write-Host "Got installation token for '$InstallationOwner' (expires $($tokenResponse.expires_at))."
if ($OutputVariableName) {
Write-Host "Setting pipeline variable '$OutputVariableName'."
Expand Down
4 changes: 2 additions & 2 deletions eng/common/core-templates/job/onelocbuild.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ parameters:
GitHubAppServiceConnection: 'dnceng-oneloc-githubapp'
GitHubAppKeyVaultName: 'EngKeyVault'
GitHubAppIdSecretName: 'oneloc-localization-app-app-id'
GitHubAppPrivateKeySecretName: 'oneloc-localization-app-app-private-key'
GitHubAppKeyName: 'oneloc-localization-app-key'

SourcesDirectory: $(System.DefaultWorkingDirectory)
CreatePr: true
Expand Down Expand Up @@ -102,7 +102,7 @@ jobs:
azureSubscription: ${{ parameters.GitHubAppServiceConnection }}
keyVaultName: ${{ parameters.GitHubAppKeyVaultName }}
appIdSecretName: ${{ parameters.GitHubAppIdSecretName }}
appPrivateKeySecretName: ${{ parameters.GitHubAppPrivateKeySecretName }}
keyName: ${{ parameters.GitHubAppKeyName }}
installationOwner: ${{ parameters.GitHubOrg }}
outputVariableName: 'GitHubAppInstallationToken'
condition: ${{ parameters.condition }}
Expand Down
4 changes: 2 additions & 2 deletions eng/common/core-templates/job/publish-build-assets.yml
Original file line number Diff line number Diff line change
Expand Up @@ -187,10 +187,10 @@ jobs:
PromoteToChannelIds: ${{ parameters.PromoteToChannelIds }}
is1ESPipeline: ${{ parameters.is1ESPipeline }}

# Darc is targeting 8.0, so make sure it's installed
# Darc is targeting 10.0, so make sure it's installed
- task: UseDotNet@2
inputs:
version: 8.0.x
version: 10.0.x

- task: AzureCLI@2
displayName: Publish Using Darc
Expand Down
4 changes: 2 additions & 2 deletions eng/common/core-templates/post-build/post-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -355,10 +355,10 @@ stages:

- template: /eng/common/templates/steps/enable-internal-runtimes.yml

# Darc is targeting 8.0, so make sure it's installed
# Darc is targeting 10.0, so make sure it's installed
- task: UseDotNet@2
inputs:
version: 8.0.x
version: 10.0.x

- task: AzureCLI@2
displayName: Publish Using Darc
Expand Down
Loading
Loading