Azure cloud provider configuration and upstream direction of external cloud configuration

[cehoffman]

What keywords did you search in tectonic-installer issues before filing this one?

cloud

Is this a BUG REPORT or FEATURE REQUEST?

FEATURE REQUEST and partial FYI to help others

Versions

Tectonic version (release or commit hash):

1.9.6-tectonic.1

Feature Request

Environment

azure

Desired Feature

Bring the azure cloud provider configuration up to date with changes in configuration and enhanced kubernetes features link. Mainly there are known (Azure/acs-engine#863) issues with the Azure API resulting in intermittent loss of partial cluster functionality that can degrade into full cluster downtime if the issues occur on masters simultaneously (has happened). The referenced issue claims it was problems with the go client library, but we suffered multiple NodeNotReady issues on azure resulting from failures in the Azure API in 1.8.9 (improved from 1.7) until we moved to use the instance metadata service.

Current configuration link lacks updated configuration useInstanceMetadata to smooth these problems.

There are other feature like Managed Service Identity (MSI) which limit the exposed capabilities of individual worker kubelets to read only. Azure/acs-engine#479

Also upstream work on external cloud provider controller manager link may mitigate both the NodeNotReady and worker credential exposure.

Our cluster has diverged from canonical Tectonic on Azure for cloud provider configuration for the above reasons and is looking at moving to the external cloud provider configuration because of limitations in the MSI administration surface area. The MSI lacks an off the shelf service like kube2iam to prevent worker pods from accessing the metadata service and getting authentication tokens to act as the VM to the Azure API.