[cert.ci.jenkins.io] Run ephemeral VM agents in the sponsored subscription

[dduportal]

Service(s)

cert.ci.jenkins.io

Summary

As per #5003, we have credits to spend in the Jenkins sponsored subscription.

Let's move the cert.ci.jenkins.io ephemeral agents in this new subscription, which includes the data transiting from/to the NAT gateway. It means the following costs will be moved to the new subscription (excerpt of the past 6 months):

Prerequisites:

• [Azure Sponsored Subscription 2026] Set up permissions #5005
• Check old "cleanup" PR or commits from 2025 former sponsored subscription
  • Azure Net: jenkins-infra/azure-net@a13538f
  • Azure: cleanup(cert.ci.jio) remove resources from the Azure Sponsored subscription azure#1115
• Check current code (for both repositories azure and azure-net) to ensure we have the same objects and naming conventions (as the old cleanup might be stuck on former techniques or naming we dropped)

Task list:

• Azure Net: a new vnet + subnet is required for ephemeral VM agents. We should use the same pattern as last year, except that the controller might need to move in this vnet as well: we should increase its size compared to 2025 right at creation so we won't have vnet overlap or increase in the upcoming weeks
• Azure: the following resources are expected (same as old setup with 2025 subscription):
  • data sources to the new vnet,subnet and their resource groups (RG)
  • A new RG for the "non agent" resources. Usually we name it xxx_ci_jenkins_io_controller_jenkins_sponsored with xxx the specific name (cert here). Will also be used if we move the controller VM of course.
  • A new "azure-vm" module instantiation in this new subscription (to create the usual resources) - RG, storage, Network Security Group (NSG), etc.
  • Missing permissions such as vnet reader
    • Nit: I'm not sure why it's not in the module. Might be missing OR there might have been a reason 🤔
  • A new User-Assigned Managed Identity (UAID) for the azure VM agents (required to be in the same subscription as the role assignment and their scopes) and its assignments to allow management by the controler Service Principal (SP) to allow writing to the buildreports file share for agents
    • Nit: might be useful to have this UAID integrated into the module in the future as we want this by default
  • Azure Container Registry (ACR) setup: a Private Endpoint (PE) in the agents subnet to reach the ACR's Private Link Service (PLS), the NSG rules associated with it
  • Output the required values for cert.ci.jenkins.io JCasC Puppet setup (see below)
    • Nit: can be done as a second non functional PR if need be
• Puppet: set up cert.ci controller to use the new vnet, subnet, their RG and the agent UAID (if I recall correctly, should be all)
  • Tip: testing the values manually in the controller UI and triggering the "agent health" helps to verify the minimum is set up. If agent do not allocate after 2-3 min, then check the controller logs and correct discovered errors
  • Once the manual tests are ok, puppet hieradata can be updated and deployed
• Setup VPN access (required to access agents with SSH bounce through the VPN):
  • Add the new vnet in the VPN routes (Docker image, e.g. VPN client side)
  • With the new image tagged, update it in puppet along with server side routes (automated PR recently fixed by Jay)
  • Allocate an agent from cert.ci (pipeline replay) and verify you can SSH to it through your machine. If cannot access, then try through OpenVPN VM and compare.
• Verify ACR from a cert.ci.jenkins.io agent
  • From an allocated agent (with SSH), check access with a curl -v https://<acr DNS name>. Fix missing requirements based on eventual errors (DNS record absent from private network? TCP not able to establish connection? etc. - see https://docs.azure.cn/en-us/container-registry/container-registry-troubleshoot-access).
  • Reminder: ACR must stay private using PE/PLS. No network peering, no public access, because it cannot be authenticated (limitation of Docker/Podman registry-mirror)
  • Once curl allows reaching the ACR, check if Docker Engine is able to use it (need root access to the agent, with journalctl -u docker -f
• Finally, cleanup: with cert.ci.jenkins.io using new subscription for agents
  • Remove old agents resources in Azure (including data source unless used somewhere else)
  • Then remove old subnet/resources from Azure Net in the CDF subscription (does not cost much, but better to cleanup for clarity)
  • Finally remove routes from OpenVPN

[github-actions]

Take a look at these similar issues to see if there isn't already a response to your problem:

1. 95% [trusted.ci.jenkins.io] Run ephemeral VM agents in the sponsored subscription #5015

[lemeurherve]

Similar to jenkins-infra/azure-net#492, when trying to deploy jenkins-infra/azure#1307, we got the following terraform error:

[2026-02-24T18:41:12.758Z] Error: creating User Assigned Identity (Subscription: "1e7d5219-acbc-4495-8629-bdbb22e9b3ed"
[2026-02-24T18:41:12.758Z] Resource Group Name: "cert-ci-jenkins-io-controller"
[2026-02-24T18:41:12.758Z] Name: "cert-ci-jenkins-io-agents-sponsored"): unexpected status 409 (409 Conflict) with error: MissingSubscriptionRegistration: The subscription is not registered to use namespace 'Microsoft.ManagedIdentity'. See https://aka.ms/rps-not-found for how to register subscriptions.
[2026-02-24T18:41:12.759Z] 
[2026-02-24T18:41:12.759Z]   with azurerm_user_assigned_identity.cert_ci_jenkins_io_azurevm_agents_jenkins_sponsored,
[2026-02-24T18:41:12.759Z]   on cert.ci.jenkins.io.tf line 163, in resource "azurerm_user_assigned_identity" "cert_ci_jenkins_io_azurevm_agents_jenkins_sponsored":
[2026-02-24T18:41:12.759Z]  163: resource "azurerm_user_assigned_identity" "cert_ci_jenkins_io_azurevm_agents_jenkins_sponsored"
[2026-02-24T18:41:12.759Z] 
[2026-02-24T18:41:12.759Z] creating User Assigned Identity (Subscription:
[2026-02-24T18:41:12.759Z] "1e7d5219-acbc-4495-8629-bdbb22e9b3ed"
[2026-02-24T18:41:12.759Z] Resource Group Name: "cert-ci-jenkins-io-controller"
[2026-02-24T18:41:12.759Z] Name: "cert-ci-jenkins-io-agents-sponsored"): unexpected status 409 (409
[2026-02-24T18:41:12.759Z] Conflict) with error: MissingSubscriptionRegistration: The subscription is
[2026-02-24T18:41:12.759Z] not registered to use namespace 'Microsoft.ManagedIdentity'. See
[2026-02-24T18:41:12.759Z] https://aka.ms/rps-not-found for how to register subscriptions.

To fix that, I've activated this resource provider:

$ az provider list --output table | grep ManagedIdentity
Microsoft.ManagedIdentity                                RegistrationRequired  NotRegistered
$ az provider register --namespace Microsoft.ManagedIdentity
Registering is still on-going. You can monitor using 'az provider show -n Microsoft.ManagedIdentity'
$ az provider show -n Microsoft.ManagedIdentity
{
  "id": "/subscriptions/1e7d5219-acbc-4495-8629-bdbb22e9b3ed/providers/Microsoft.ManagedIdentity",
  "namespace": "Microsoft.ManagedIdentity",
  "providerAuthorizationConsentState": null,
  "registrationPolicy": "RegistrationRequired",
  "registrationState": "Registered",
  "resourceTypes": [
    {
      "aliases": null,
      "apiProfiles": null,
      "apiVersions": [
        "2025-01-31-PREVIEW",
        "2024-11-30",
        "2023-07-31-PREVIEW",
        "2023-01-31",
        "2022-01-31-PREVIEW",
        "2021-09-30-PREVIEW",
        "2018-11-30",
        "2015-08-31-PREVIEW"
      ],
      "capabilities": "SupportsExtension",
      "defaultApiVersion": "2024-11-30",
      <snip>
      "properties": null,
      "resourceType": "userAssignedIdentities/federatedIdentityCredentials",
      "zoneMappings": null
    }
  ]
}