V0.16.x (#4)

* KIAM updates to support assumeRoleArn functionalilty

* Add compute.internal to etcd san when using private zones, because the aws controller does not support private zones

* Fix issue with node names in the clusters

* Fix tests

* Whitespace.

* Forced rebuild.

* Update cloud-config-controller

* Update cloud-config-controller

* Update test

* Remove verbose json output.

* Allow dnsmasq to be backed by a local copy of CoreDNS

This commit allows the user to specify that dnsmasq should be
backed by a pod-local copy of CoreDNS rather than relying on
the global CoreDNS service. If enabled, the dnsmasq-node
DaemonSet will be configured to use a local copy of CoreDNS
for its resolution while setting the global CoreDNS service as
a fallback. This is handy in situations where the number of DNS
requests within a cluster grows large and causes resolution issues
as dnsmasq reaches out to the global CoreDNS service.

Additionally, several values passed to dnsmasq are now configurable
including its `--cache-size` and `--dns-forward-max`.

See [this postmortem](https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/docs/postmortems/jan-2019-dns-outage.md)
for an investigation into this situation which was instrumental in
understanding issues we were facing. Many thanks to dominicgunn
for providing the manifests which I codified into this commit.

---

These features can be enabled and tuned by setting the following
values within cluster.yaml:

```yaml
kubeDns:
  dnsmasq:
    coreDNSLocal:
      # When enabled, this will run a copy of CoreDNS within each DNS-masq pod and
      # configure the utility to use it for resolution.
      enabled: true

      # Defines the resource requests/limits for the coredns-local container.
      # cpu and/or memory constraints can be removed by setting the appropriate value(s)
      # to an empty string.
      resources:
        requests:
          cpu: 50m
          memory: 100Mi
        limits:
          cpu: 50m
          memory: 100Mi

    # The size of dnsmasq's cache.
    cacheSize: 50000

    # The maximum number of concurrent DNS queries.
    dnsForwardMax: 500

    # This option gives a default value for time-to-live (in seconds) which dnsmasq
    # uses to cache negative replies even in the absence of an SOA record.
    negTTL: 60
```

* Always create required dnsmasq resources

The dnsmasq-node ServiceAccount must exist whether or not CoreDNS-local
has been enabled. Therefore, it is created alongside the DaemonSet rather
than as part of the coredns-local manifest.

Additionally, always create dnsmasq-node-coredns-local.yaml If this file
does not exist (as would be the case if the CoreDNS local feature has
not been enabled), controller nodes will fail to come up with the error:
> error: the path "/srv/kubernetes/manifests/dnsmasq-node-coredns-local.yaml" does not exist
This is caused when `kubectl delete` is called against the file because
of the line `remove "${mfdir}/dnsmasq-node-coredns-local.yaml`.

This manifest must always be generated because the CoreDNS-local
feature cannot be enabled and then later disabled without otherwise
requiring manual operator intervention.

Co-authored-by: Dominic Gunn <[email protected]>
Co-authored-by: Dominic Gunn <[email protected]>
Co-authored-by: Kevin Richardson <[email protected]>

Author: 3 people

builtin/files/cluster.yaml.tmpl

@@ -1167,7 +1167,7 @@ kubernetes:
   # Tells Kubernetes to enable the autoscaler rest client (not using heapster) without the requirement to use metrics-server.
   podAutoscalerUseRestClient:
     enabled: false
-    
+
 #  controllerManager:
 #   resources:
 #     requests:
@@ -1329,6 +1329,35 @@ kubeDns:
   # - --neg-ttl=10
   # - --no-ping
 
+  # Settings for the dnsmasq-node DaemonSet which must be enabled by setting
+  # `kubeDns.nodeLocalResolver` to true.
+  dnsmasq:
+    coreDNSLocal:
+      # When enabled, this will run a copy of CoreDNS within each DNS-masq pod and
+      # configure the utility to use it for resolution.
+      enabled: false
+
+      # Defines the resource requests/limits for the coredns-local container.
+      # cpu and/or memory constraints can be removed by setting the appropriate value(s)
+      # to an empty string.
+      resources:
+        requests:
+          cpu: 50m
+          memory: 100Mi
+        limits:
+          cpu: 50m
+          memory: 100Mi
+
+    # The size of dnsmasq's cache.
+    cacheSize: 50000
+
+    # The maximum number of concurrent DNS queries.
+    dnsForwardMax: 500
+
+    # This option gives a default value for time-to-live (in seconds) which dnsmasq
+    # uses to cache negative replies even in the absence of an SOA record.
+    # negTTL: 60
+
   # When enabled, will modify the TTL of the coredns service.
   # ttl: 30
 
@@ -1351,6 +1380,8 @@ kubeDns:
   #
   # This configuration is injected into the CoreDNS config map after the root
   # zone (".") and can be used to add configuration for additional zones.
+  # If coreDNSLocal has been enabled, this configuration will additionally be injected
+  # into its ConfigMap.
   # additionalZoneCoreDNSConfig: |
   #  global:53 {
   #      errors
@@ -1378,7 +1409,7 @@ kubeProxy:
 # When enabled, a security group rule is included on the generated kube-aws SG to allow ICMP Ping from all traffic (0.0.0.0/0).
 # This is applied to all nodes (worker & control plane) in the cluster.
 openICMP: true
- 
+
 # Addon features
 addons:
   # When enabled, Kubernetes rescheduler is deployed to the cluster controller(s)