[Feature Request] Create GitHub-linked pipeline with variables unattended

[jaraco]

Is your feature request related to a problem? Please describe.

With other CI systems like Travis-CI, enrolling a project is about as simple as invoking travis enable. Currently with Azure Devops and Pipelines, it's dramatically more manually-intensive. There az project create to create the project, but from there, it's still necessary to manually:

• Create a pipeline
• Link that pipeline to a GitHub (or other) hosting
• Create a variable group
• Authorize that group to an Azure Key Vault (multiple steps)
• Authorize the pipeline to azure to be run.
• Run the pipeline.

See the screetcapture that illustrates the workflow that should be straightforward to automate through the command-line.

I've enacted this process 19 times now and it's getting rather tedious.

Describe the solution you'd like

As I maintain over 140 packages in the Python ecosystem and I'm working to migrate them over to Azure, it would be seriously valuable to me if there were APIs or command-line invocations I could make to fully automate the process of enrolling a project with Azure pipelines.

It's possible that portions of this request are duplicates with existing requests or that some of this functionality has been implemented, but I wanted to capture this whole-workflow need for your consideration.

[ishitam8]

Hi @jaraco ,

Did you check az pipelines create cmd and az pipelines variable-group cmds?
You can use pipeline create cmd to create your pipeline linked with Github repo.
Sorry, currently creating variable groups associated with key-vault is not supported, since there is no publicly available API for the same.
Also check if you can use az devops invoke cmd in your case. More details #723 for creating variable-groups linked to Azure key-vault.

[jaraco]

Great suggestions. I'll take a look and report back.

[jaraco]

While trying az pipelines create, I learned I needed to upgrade both azure-cli and azure-devops. Once doing that, I have the az pipelines create command, but I ran into two issues. The first was that the examples in help didn't work. None of them specify --project but when running the commands, an error is emitted:

--project must be specified. The value should be the ID or name of a team project. You can set a default value by running: az devops configure --defaults project=<ProjectName>.

So looking at what the web UI creates, I tried using the project name for both --name and --project, and that seemed to work, but it prompted for GitHub credentials. There's a note there, though:

Service connection required for non Azure Repos can be optionally provided in the command run it non interatively

But I don't know what my service connection is. I checked the project and it doesn't have a service connection... which is weird, because the other projects have service connections for "jaraco (Github)" and my Azure account.

I tried using "jaraco" as the service connection, but that didn't work.

svg.charts master $ az pipelines create --name svg.charts --project svg.charts --service-connection jaraco                                                                
This command is in preview. It may be changed/removed in a future release.
Value cannot be null.
Parameter name: serviceConnectionId

I then tried finding the ID of the resource by navigating to one of the other projects and finding its Github service connection and inspecting the URL, but it also didn't accept that service connection.

svg.charts master $ az pipelines create --name svg.charts --project svg.charts --service-connection ec35cd1c-163b-49b0-9318-fcfaca77321e                                  
This command is in preview. It may be changed/removed in a future release.
Service connection ec35cd1c-163b-49b0-9318-fcfaca77321e not found in the project 9ac28141-0b2c-4cf2-b50f-c8feb95102fb.

Is it possible, given a freshly-created project, to associate a service connection with that project from the login, based on whatever technique the web UI is using?

[ishitam8]

@jaraco ,

You will need to pass a project parameter wherever it is required , project and organization parameters are usually omitted in examples, assuming user has configured them as defaults.

You can list the service connections for your project here:
az devops service-endpoint list

You can create a new github service connection
az devops service-endpoint github create -h

And then later add this to your pipeline create command

[jaraco]

Thanks ishitam8. This is great! I think I'm making progress.

After supplying a Github Personal Access token, I was able to create the endpoint. I used my username/org as the --name, as that seems to be the name that was used endpoints as created through the web.

jaraco.functools master $ $AZURE_DEVOPS_EXT_GITHUB_PAT=keyring.get_password('Github', 'jaraco')
jaraco.functools master $ az devops service-endpoint github create --github-url https://github.com/jaraco/jaraco.functools --name jaraco --project jaraco.functools
{
...
}

When I create the service-endpoint through the web, I don't have to supply a GitHub access token. Is that because my browser is already authorized with GitHub, so the web UI is relying on that access to create the endpoint?

After creating the service endpoint, I'm able to create the pipeline:

jaraco.functools master $ az pipelines create --name jaraco.functools --project jaraco.functools --service-connection ee6c39d1-5c39-48ab-a8be-4cbe65e0b9ce --yaml-path azu
re-pipelines.yml                                                                                                                                                          
This command is in preview. It may be changed/removed in a future release.
Successfully created a pipeline with Name: jaraco.functools, Id: 20.
Could not queue the build because there were validation errors or warnings.

However, as you can see, it worked well, but gave an error about queuing a build. Probably that has to do with the missing variable group. I'll be investigating that next.

[jaraco]

I tried to create the variable group with secrets linked to Azure key vault, but I don't that's possible through the current UI. Trying to create the variables:

jaraco.functools master $ az pipelines variable-group create --name 'Azure secrets' --project jaraco.functools --authorize true
az pipelines variable-group create: error: the following arguments are required: --variables
usage: az pipelines variable-group create [-h] [--verbose] [--debug] [--output {json,jsonc,table,tsv,yaml,yamlc,none}] [--query JMESPATH] [--subscription _SUBSCRIPTION]
                                          --name NAME --variables [VARIABLES [VARIABLES ...]] [--description DESCRIPTION] [--authorize [{true,false}]]
                                          [--organization ORGANIZATION] [--project PROJECT] [--detect [{true,false}]]

Indeed, that's what you said above. I'll look into #723.

[jaraco]

OMG!

It worked:

jaraco.functools master $ cat > vars.json                                               
{
    "description": "",
    "name": "Azure secrets",
    "providerData": {
        "serviceEndpointId": "87c71fed-f2b9-408a-bc32-e9160f980958",
        "vault": "secrets007"
    },
    "type": "AzureKeyVault",
    "variables": {
        "PyPI-token": {
          "contentType": "",
          "enabled": true,
          "isSecret": true,
          "value": null
        },
        "Tidelift-token": {
          "contentType": "",
          "enabled": true,
          "isSecret": true,
          "value": null
        }
    }
}
jaraco.functools master $ az devops invoke --http-method post --area distributedtask --resource variablegroups --debug --in-file vars.json --encoding utf-8 --route-parameters project=jaraco.functools --api-version 5.0-preview
...
az_command_data_logger : exit code: 0
{
  "createdBy": {
    "displayName": null,
    "id": "92e806d6-b002-6f2f-a5f6-cc2bb7277fda"
  },
  "createdOn": "2020-04-02T00:26:10.2033333Z",
  "id": 20,
  "isShared": false,
  "modifiedBy": {
    "displayName": null,
    "id": "92e806d6-b002-6f2f-a5f6-cc2bb7277fda"
  },
  "modifiedOn": "2020-04-02T00:26:10.2033333Z",
  "name": "Azure secrets",
  "providerData": {
    "serviceEndpointId": "87c71fed-f2b9-408a-bc32-e9160f980958",
    "vault": "secrets007"
  },
  "type": "AzureKeyVault",
  "variableGroupProjectReferences": [
    {
      "name": "Azure secrets",
      "projectReference": {
        "id": "194510b8-89df-4d0d-83a2-eff1c76a6286",
        "name": "jaraco.functools"
      }
    }
  ],
  "variables": {
    "PyPI-token": {
      "contentType": "",
      "enabled": true,
      "isSecret": true,
      "value": null
    },
    "Tidelift-token": {
      "contentType": "",
      "enabled": true,
      "isSecret": true,
      "value": null
    }
  }
}
telemetry.save : Save telemetry record of length 2695 in cache
telemetry.check : Negative: The /Users/jaraco/.azure/telemetry.txt was modified at 2020-04-01 20:21:13.194501, which in less than 600.000000 s
command ran in 2.690 seconds.

However, now when I try to run the pipeline, I see this error (web UI):

An error occurred while loading the YAML build pipeline. Service connection : 87c71fed-f2b9-408a-bc32-e9160f980958 used in variable group : Azure secrets is not valid. Either service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz.

Do you know how I can authorize those resources to the newly-created pipeline?

[jaraco]

A few other notes - after creating this variable group, it was not yet authorized with any Azure subscription nor key vault. Is it possible to authorize it with the service and endpoint?

It's not configured to "Allow access to all pipelines".

I also find I'm unable to edit the variable group. When I attempt to edit it through the UI, I see this error:

TFS.WebApi.Exception: Value cannot be null. Parameter name: variableGroupParameters

Any ideas?

[ishitam8]

@jaraco ,

I think since your pipeline is not configured for "Allow access to all pipelines" , you will need to separately authorize variable group.
You can authorize the pipeline using the az devops invoke command from the CLI . To authorize the resource using the az devops invoke command, you can follow this syntax
az devops invoke --area pipelinepermissions --resource pipelinePermissions --route-parameters project= resourceType=variablegroup resourceId=<var_group_ID> --in-file params.json --http-method PATCH --api-version "5.1-preview1"

params.json
{ "pipelines": [ { "id": "<Build-Id>", "authorized": "true" } ] }

[jaraco]

I think since your pipeline is not configured for "Allow access to all pipelines" , you will need to separately authorize variable group.

I'd like to configure my pipeline (did you mean "variable group"?) to "Allow access to all pipelines".

Also, I'm unsure I've selected the correct "serviceEndpointId" for the variable group. I created the variable group using the endpoint ending in 958, but I don't think that was correct and I think that's what led to the error. When I create the variable group manually, I see it assigns a different endpoint id: 9f84f829-0e6c-403e-a90a-cd949913301c, so I've changed the vars.json to contain that content.

Furthermore, even if I use that serviceEndpointId, I continue to get TFS.WebApi.Exception: Value cannot be null errors in the web UI when I attempt to manipulate the variable group created through the command-line.

In your latest suggestion, you suggest running a command to authorize the pipeline (or variable group or resource?) explicitly, but it's not obvious to me where I get the var_group_ID or Build-Id from.

I have questions:

• Can I configure the pipeline/variableGroup to "Allow access to all pipelines", either when I create it or after?
• How do I reason about the correct "providerData.serviceEndpointId" for the variable group when creating it? How do I discover that endpoint ID in an automated fashion?
• Can you help me construct a command that creates a variable group that's not broken?
• Where can I find the var_group_ID and Build-Id as indicated in the example? Is the var_group_id the integer id found when retrieving the variablegroups (20 in the output from post to --resource variablegroups above)?