v0.9.8-rc.1

Changelog since v0.9.7

Component versions

Kubernetes: v1.7.3
Etcd: v3.2.5

Actions required

• #820: Simplify configuration for OIDC Authenticator.

  • The key dex and its children in cluster.yaml has been basically renamed to oidc and the correspondents in apiserver flags. See #820 for more information

• #832: Update Calico to v2.4.1

  • To maintain existing behavior when upgrading your existing cluster, follow these steps:
    • In Namespaces that previously did not have the “DefaultDeny” annotation, you should delete any existing NetworkPolicy objects.
    • In Namespaces that previously did have the “DefaultDeny” annotation, you can create the equivalent semantics by creating a NetworkPolicy that selects all pods but does not allow any traffic.
    • See kubernetes/kubernetes#39164 (comment) for more details

Features

• #731: Add cluster kube-aws version to outputs(Thanks to @Vrtak-CZ)
• #742: Install Tiller by default
• #752: Deny direct command execution on privileged containers(Thanks to @ytsarev)
• #760: Support cross-stack references of VPC, IGW
• #761: More flexible IAM configuration for etcd nodes
• #778: Better encryption error message(Thanks to @redbaron)
• #789: Ability to propagate custom options to kubelet(Thanks to @ytsarev)
• #791: Plugin System
• #792: Make PODs to resolve DNS names via locally running dnsmasq(Thanks to @dvdthms)
• #809: Automatically configure kube2iam.(Thanks to @camilb)
• #820: Simplify configuration for OIDC Authenticator.(Thanks to @camilb)
• #821: Add Spot Fleet support for the automatic ALB target group attachment

Improvements

• #740: Update the default etcd version to 3.2.1
• #743: Update CA to 0.6.0
• #746: Update Kubernetes to v1.7.0
• #755: Rename experimental.nodeLabels to controller.nodeLabels
• #756: Explicitly disallow tainting controller nodes
• #757: Remove deprecated keys in cluster.yaml
• #774: Update Kubernetes to v1.7.1
• #780: Stop using unnecessary autoscaling notification target/role
• #787: Rescheduler logs now piped to docker(Thanks to @c-knowles)
• #788: Additional permissions for heapster nanny(Thanks to @c-knowles)
• #794: Refactor node drainer implementation(Thanks to @danielfm)
• #817: Bump default k8s to 1.7.2(Thanks to @c-knowles)
• #818: Put kube2iam update strategy in the correct place(Thanks to @c-knowles)
• #828: Bump tiller to 2.5.1(Thanks to @c-knowles)
• #830: Bump default k8s to 1.7.3(Thanks to @c-knowles)
• #832: Update Calico to v2.4.1(Thanks to @tmjd)
• #835: cfn-signal depends on install-kube-system(Thanks to @dvdthms)
• #844: update default version of kubernetes dashboard to 1.6.3(Thanks to @Vrtak-CZ)
• #845: update default version of ETCd to 3.2.5(Thanks to @Vrtak-CZ)

Bug fixes

• #713: Fix kube-resources-autosave when kube2iam is enabled(Thanks to @camilb)
• #749: Fix kubelet bootstrap for Kubernetes 1.7(Thanks to @danielfm)
• #763: Fix node labeling to allow scheduling cluster-autoscaler to workers
• #773: Fix --ami-id
• #797: Issue #796 - cluster.yaml missing dnsMasqMetricsImage.repo key(Thanks to @wallentx)
• #814: Fix for typo introduced in #792(Thanks to @redbaron)
• #824: Fix managed role name validation(Thanks to @adyromantika)
• #827: Fixed typo in the error message(Thanks to @sathiyas)
• #840: Fix Typo to improve GoReport Card(Thanks to @asifdxtreme)
• #849: Fix failing worker and controller nodes when Calico is enabled

Documentation

• #733: Bugfix: CloudWatchLogging always disabled for Worker nodes(Thanks to @jollinshead)
• #748: Provide real-time feedback from Journald logs when updating/creating …(Thanks to @jollinshead)
• #781: Fix (Journald logging) localStreaming typo.(Thanks to @jollinshead)
• #801: CloudFormation events stream to stdout during kube-aws up/update(Thanks to @jollinshead)
• #826: Updating instructions for MFA token(Thanks to @sathiyas)
• #829: Add AWS_PROFILE to FAQ(Thanks to @Vrtak-CZ)

Refactorings

• #777: Use consistent names for validation functions(Thanks to @danielfm)

Other changes

• #739: Update OWNERS
• #741: Update golang to v1.8.3
• #751: Initial plugins proposal(Thanks to @c-knowles)
• #770: Containerized test run(Thanks to @ytsarev)
• #807: Update ROADMAP
• #836: Remove redundant step(Thanks to @asifdxtreme)
• #838: Fix Typo in Events Code of Conduct(Thanks to @mbssaiakhil)

v0.9.7: NVIDIA GPU, Flexible IAM, Improved supports for RBAC, Node Drainer, cluster-autoscaler, Fixes to etcd automatic disaster recovery

Notable changes since v0.9.6

Full changelog can be seen at v0.9.6...v0.9.7

Please see our roadmap for details on upcoming releases.

Actions required

• #639: Users of the experimental TLS bootstrap feature are required to run the following actions:

1. Run kube-aws render stack to update controller/worker user data templates

2. Move the bootstrap token from ./credentials/tokens.csv to its own file ./credentials/kubelet-tls-bootstrap-token

   # Remove the following line from ./credentials/tokens.csv, and move <token>
   # (with no leading/trailing blank chars and lines) to it's own file
   # ./credentials/kubelet-tls-bootstrap-token
   <token>,kubelet-bootstrap,10001,system:kubelet-bootstrap
   

3. Run kube-aws update to update the cluster. This operation will cause controllers and workers to be replaced

• #629: experimenetal.clusterAutoscalerSupport.enabled was removed in favor of addons.clusterAutoscaler
• #629: worker.nodePools[].clusterAutoscaler.(min|max)Size was removed in favor of worker.nodePools[].autoscaling.clusterAutoscaler.enabled

Known issues

• etcdDataVolumeEphemeral is broken. Please don't turn it until it is fixed and keep using EBS based data volumes instead. #446

Features

• Kubernetes 1.6.3
  • #624: Update default dashboard to 1.6.0 (Thanks to @Vrtak-CZ)
  • #646: Update versions for various components.(Thanks to @camilb)
  • #659: Update Kubernetes dashboard to v1.6.1. Update calico to v2.2.1.(Thanks to @camilb)
• #559: Add sprig templating functions (Thanks to @tyrannasaurusbanks)
• #607: More flexible configuration for IAM and stable naming for roles (Thanks to @Fsero)
• #629: Re: cluster-autoscaler support(Thanks to @redbaron for reviewing)
  • Follow-ups
    • #724: Add critical-pod annotation to CA
    • #738: Add CA a CriticalAddonsOnly toleration
• #645: NVIDIA driver installation support on GPU instances(Thanks to @everpeace)
• #707: Send Journald logs to AWS CloudWatch(Thanks to @jollinshead)
• #737: Drain node on spot instance termination notice as well(Thanks to @danielfm)

Enhancements

• #615: kube-dns improvements(Thanks to @danielfm)
• #616: Improve taints validation(Thanks to @danielfm)
• #618: RBAC setup improvements(Thanks to @danielfm)
• #625: Fix dashboard version in labels and remove version from RC name (Thanks to @Vrtak-CZ)
• #650: Label masters with 'node-role.kubernetes.io/master' label(Thanks to @redbaron)
• #652: Label masters with 'node-role.kubernetes.io/master' label(Thanks to @redbaron)
• #663: Make kubelet flags more consistent(Thanks to @redbaron)
• #693: Spot fleet detailed monitoring(Thanks to @paalkr)
• #682: Allow userdata to be split across multiple parts(Thanks to @redbaron)
• #701: core: update cluster-proportional-autoscaler to v1.1.2(Thanks to @harsha-y)
• #718: Report errors if empty nodepool is specified(Thanks to @redbaron)

Fixes

• #619: Fix IamFleetRole syntax (Thanks to @danielfm)
• #632: Fix unable to backup namespaced resources(Thanks to @cheungpat)
• #634: Fixes leading slash on s3:prefix removed to fix ListObject permission denied error during etcdadm save(Thanks to @jeremyd)
• #639: Avoid unnecessary node replacements when TLS bootstrapping is enabled(Thanks to @danielfm)
• #641: adding --rm to docker run aws_cli commands in etcdadm(Thanks to @jeremyd)
• #657: Fix typo in help message(Thanks to @ytsarev)
• #662: Fix taint being assigned as labels(Thanks to @redbaron)
• #671: Fix "install-kube-system" script when "clusterAutoscaler" is disabled(Thanks to @camilb)
• #674: Improved node drainer(Thanks to @danielfm)
• #676: Ensure TLSBootstrapToken related code is not written if TLSBootstrap is disabled(Thanks to @jollinshead)
• #687: calico-node - DaemonSet tolerating all taints(Thanks to @jeffersongirao)
• #686: Heapster service account and cluster role binding(Thanks to @c-knowles)
  Follow-up: #692: Apply heapster RBAC setup properly(Thanks to @c-knowles)
• #695: core: Add ListBucket explicitly to EtcdSnapshotsS3Bucket(Thanks to @trinitronx)
• #697: Fix the issue that cluster-autoscaler never scale-down the cluster
• #705: Use docker instead of rkt for regular etcdadm tasks(Thanks to @ytsarev)
• #710: Fix node drain error when trying to evict pods from jobs(Thanks to @danielfm)
• #711: Remove unused sysctl override for nf_conntrack_max(Thanks to @danielfm)
• #722: Additional propagation of etcd version for etcdadm(Thanks to @ytsarev)
• #735: Inject stack name into userdata for nodepool workers(Thanks to @redbaron)

Documentation

• #626: Fix quote on "Launch the secondary node"(Thanks to @velo)
• #635: cluster.yaml: Additional note for etcd.count
• #637: Fix destroy doc(Thanks to @jorge07)
• #689: added required --s3-url parameter(Thanks to @spatronis)
• #696: Introduce kube-aws slack channel in README
• #700: Enhance cluster.yaml documentation(Thanks to @ytsarev)

Refactoring

• #680: Remove ancient CoreOS version check which is not relevant anymore(Thanks to @redbaron)
• #678: Kill dead code and restore DNS config validation(Thanks to @redbaron)
• #670: Remove obsolete etcd locking logic(Thanks to @redbaron)

Testing

• #669: Make go test timeout longer enough for Travis
• #723: e2e: Fix flag provided but not defined: -check_version_skew error

Project

• #739: Update OWNERS

v0.9.7-rc.4

Notable changes since v0.9.7-rc.3

Full changelog can be seen at v0.9.7-rc.3...v0.9.7-rc.4

Features

• #707: Send Journald logs to AWS CloudWatch(Thanks to @jollinshead)

Fixes

• #705: Use docker instead of rkt for regular etcdadm tasks(Thanks to @ytsarev)
• #710: Fix node drain error when trying to evict pods from jobs(Thanks to @danielfm)
• #711: Remove unused sysctl override for nf_conntrack_max(Thanks to @danielfm)

Documentation

• #700: Enhance cluster.yaml documentation(Thanks to @ytsarev)

Enhancements

• #682: Allow userdata to be split across multiple parts(Thanks to @redbaron)
• #701: core: update cluster-proportional-autoscaler to v1.1.2(Thanks to @harsha-y)

v0.9-7-rc.3: Fixes to RBAC, Etcd Disaster Recovery, Cluster Autoscaler, Refactorings, Improved Node Drainer, and more

Notable changes since v0.9.7-rc.2

Full changelog can be seen at v0.9.7-rc.2...v0.9.7-rc.3

Fixes

• #674: Improved node drainer(Thanks to @danielfm)
• #676: Ensure TLSBootstrapToken related code is not written if TLSBootstrap is disabled(Thanks to @jollinshead)
• #687: calico-node - DaemonSet tolerating all taints(Thanks to @jeffersongirao)
• #686: Heapster service account and cluster role binding(Thanks to @c-knowles)
  • Follow-up: #692: Apply heapster RBAC setup properly(Thanks to @c-knowles)
• #695: core: Add ListBucket explicitly to EtcdSnapshotsS3Bucket(Thanks to @trinitronx)
• #697: Fix the issue that cluster-autoscaler never scale-down the cluster

Documentation

• #689: added required --s3-url parameter(Thanks to @spatronis)
• #696: Introduce kube-aws slack channel in README

Enhancements

• #693: Spot fleet detailed monitoring(Thanks to @paalkr)

Refactorings

• #680: Remove ancient CoreOS version check which is not relevant anymore(Thanks to @redbaron)
• #678: Kill dead code and restore DNS config validation(Thanks to @redbaron)
• #670: Remove obsolete etcd locking logic(Thanks to @redbaron)

v0.9.7-rc.2

Notable changes since v0.9.7-rc.1

Full changelog can be seen at v0.9.7-rc.1...v0.9.7-rc.2

Fixes

• #671: Fix "install-kube-system" script when "clusterAutoscaler" is disabled(Thanks to @camilb)

[broken] v0.9.7-rc.1: K8S 1.6.3, Sprig, More flexible IAM, cluster-autoscaler, NVIDIA GPU

Notable changes since v0.9.6

Full changelog can be seen at v0.9.6...v0.9.7-rc.1

Actions required

• #639: Users of the experimental TLS bootstrap feature are required to run the following actions:

1. Run kube-aws render stack to update controller/worker user data templates

2. Move the bootstrap token from ./credentials/tokens.csv to its own file ./credentials/kubelet-tls-bootstrap-token

   # Remove the following line from ./credentials/tokens.csv, and move <token>
   # (with no leading/trailing blank chars and lines) to it's own file
   # ./credentials/kubelet-tls-bootstrap-token
   <token>,kubelet-bootstrap,10001,system:kubelet-bootstrap
   

3. Run kube-aws update to update the cluster. This operation will cause controllers and workers to be replaced

• #629: experimenetal.clusterAutoscalerSupport.enabled was removed in favor of addons.clusterAutoscaler
• #629: worker.nodePools[].clusterAutoscaler.(min|max)Size was removed in favor of worker.nodePools[].autoscaling.clusterAutoscaler.enabled

Known issues

• etcdDataVolumeEphemeral is broken. Please don't turn it until it is fixed and keep using EBS based data volumes instead. #446
• subPath is broken in k8s 1.6.3 - v1.6.4 with the fix is already released in upstream
  • ref kubernetes/kubernetes#45749

Features

• Kubernetes 1.6.3
  • #624: Update default dashboard to 1.6.0 (Thanks to @Vrtak-CZ)
  • #646: Update versions for various components.(Thanks to @camilb)
  • #659: Update Kubernetes dashboard to v1.6.1. Update calico to v2.2.1.(Thanks to @camilb)
• #559: Add sprig templating functions (Thanks to @tyrannasaurusbanks)
• #607: More flexible configuration for IAM and stable naming for roles (Thanks to @Fsero)
• #629: Re: cluster-autoscaler support(Thanks to @redbaron for reviewing)
• #645: NVIDIA driver installation support on GPU instances(Thanks to @everpeace)

Fixes

• #619: Fix IamFleetRole syntax (Thanks to @danielfm)
• #632: Fix unable to backup namespaced resources(Thanks to @cheungpat)
• #634: Fixes leading slash on s3:prefix removed to fix ListObject permission denied error during etcdadm save(Thanks to @jeremyd)
• #639: Avoid unnecessary node replacements when TLS bootstrapping is enabled(Thanks to @danielfm)
• #641: adding --rm to docker run aws_cli commands in etcdadm(Thanks to @jeremyd)
• #662: Fix taint being assigned as labels(Thanks to @redbaron)

Improvements

• #615: kube-dns improvements(Thanks to @danielfm)
• #616: Improve taints validation(Thanks to @danielfm)
• #618: RBAC setup improvements(Thanks to @danielfm)
• #625: Fix dashboard version in labels and remove version from RC name (Thanks to @Vrtak-CZ)
• #650: Label masters with 'node-role.kubernetes.io/master' label(Thanks to @redbaron)
• #652: Label masters with 'node-role.kubernetes.io/master' label(Thanks to @redbaron)
• #663: Make kubelet flags more consistent(Thanks to @redbaron)

Documentation updates

• #626: Fix quote on "Launch the secondary node"(Thanks to @velo)
• #635: cluster.yaml: Additional note for etcd.count
• #637: Fix destroy doc(Thanks to @jorge07)

Other updates

• #657: Fix typo in help message(Thanks to @ytsarev)
• #669: Make go test timeout longer enough for Travis

v0.9.6: Kubernetes 1.6.2, Automatic Disaster Recovery for Etcd3, K8s Resources Autosaving, Dex support, TLS bootstrapping, Multi k8s API endpoints

Notable changes since v0.9.5

Full changelog can be seen at v0.9.5...v0.9.6

Actions required

• Due to the changes in how API endpoint load balancers and etcd clusters are provisioned, you may need to recreate your kube-aws clusters from scratch as always. Please see #455 for more information
• The support for Kubernetes 1.5.x is dropped. To whom still like to stick with 1.5.x, we have an open issue dedicated for bringing the support. You can also keep using kube-aws v0.9.5

Breaking changes

• #565 EFS PersistentVolume introduces Recycle policy which is a breaking change. Be aware that if all the persistent volume claims are removed, the EFS will recycle and remove all data

Known issues

• etcdDataVolumeEphemeral is broken. Please don't turn it until it is fixed and keep using EBS based data volumes instead. #446

Features

• Etcd3 & Automatic Disaster Recovery
  • #417: Automatic recovery from permanent failures of etcd3 nodes
  • #511: etcd unit should unconditionally depend on cfn-etcd-environment (Thanks to @redbaron)
  • #517: Fix a race between systemd services: cfn-etc-environment and etcdadm-reconfigure
  • #531: Fix the dead-lock while bootstrapping etcd cluster (Thanks to @redbaron for reporting)
• Kubernetes 1.6.2
  • #492: Bump to Kubernetes v1.6.1
  • #504: Fix RBAC in Kubernetes 1.6. Fix etcdadm when terminated instances still exist (Thanks to @camilb)
  • #508: Bump rescheduler to 0.3.0 which uses k8s 1.6 (Thanks to @c-knowles)
  • #558: Fix to calico configuration file etcd endpoints (Thanks to @kevtaylor)
  • #564: bump kube-1.6.2 (Thanks to @redbaron)
  • #575: Quote security group refs for etcd, controller, and apiendpoints (Thanks to @soellman)
  • #576: Set --storage-backend to etcd2 if not using etcd3 (Thanks to @cheungpat)
  • #581: Update kubelet flags (Thanks to @c-knowles for reporting)
  • #582: Update kube-dns to 1.14.1 (Thanks to @c-knowles for reporting)
  • #590: Fix etcd snapshots locations in S3 (Thanks to @cmcconnell1 for providing the important info to
  • #594: Fix syntax error (Thanks to @danielfm)
  • #606: Fix certs path when TLS bootstrapping is enabled (Thanks to @danielfm)
    locate the issue)
• #449, #489: Kubelet TLS bootstrapping (Thanks to @danielfm)
• #441, #486: Introduce the rescheduler (Thanks to @c-knowles)
• #468: Support for multiple k8s API endpoints
  • #514: Fix API endpoint from HA controllers (Thanks to @c-knowles)
  • #521: Fix incorrect validations on apiEndpoints
  • #526: Fix up API endpoints config (Thanks to @c-knowles)
  • #529: Follow-up for the multi API endpoints support
• Kubernetes Resources Autosaving & Restore
  • #507: 'Cluster-dump' feature to export Kubernetes Resources to S3 (Thanks to @jollinshead)
  • #535: 'Restore' feature to restore Kubernetes Resources from S3 backup (Thanks to @jollinshead)
  • #538: Bugfix: Add missing '/' when constructing the Autosave S3 put path (Thanks to @jollinshead)
  • #570: Kubernetes-Autosave save as Kubernetes/List. (Thanks to @jollinshead and @c-knowles)
  • #609: Additional notes for autosave regarding s3 error (Thanks to @jollinshead)
• #568: Dex integration (Thanks to @camilb)
• #589: Bump to calico 2.1.4 (Thanks to @redbaron)
• #577: Add controller node labels if specified (Thanks to @cheungpat)
• #551: Allow customizing network ranges from which SSH accesses to nodes are allowed
• #552: Allow customizing network ranges from which Kubernetes API accesses are allowed
• #471: Shared Persistent Volume (Thanks to @kevtaylor)
  • #565: Add reclaim policy (Thanks to @kevtaylor)
• #510: New options: customFiles and customSystemdUnits (Thanks to @jeremyd)
• New settings: nodeMonitorGracePeriod, disableSecurityGroupIngress for controller-manager, nodeStatusUpdateFrequency for worker kubelet (#473, thanks to @jeremyd)

Fixes

• #476: Setup net.netfilter.nf_conntrack_max and fix error "nf_conntrack: table full, dropping packet" (Thanks to @gianrubio)
• #503: Perform docker post-start check (Thanks to @redbaron)
• #555: Don't mount /var/lib/rkt into kubelet (Thanks to @redbaron)
• #561: Fix unwanted AWS resource creation/Add extra validation on internetGatewayID + vpcID
• #563: Make cfn-signal more robust against image fetch failures (Thanks to @redbaron)
• #579: Fix no space left on device when audit loggin enabled for apiserver (Thanks to @ankon and @whereisaaron for reporting)
• elasticFileSystemId
  • F#530: Fix elasticFileSystemId to be propagated to node pools (Thanks to @drywheat for reporting)
  • #610: Fix elasticFileSystemId only on node pools (Thanks to @Vrtak-CZ for reporting)
• #613: Fix typo in command listing (Thanks to @simonwydooghe)

Improvements

• #472: Update kube-system using kubectl (Thanks to @jollinshead)
• #481: Deprecate verbose legacy keys in favor of corresponding nested keys
• #515: Make AMI fetching even more reliable
• #519: Wait until kube-system becomes ready
• #516: Retry userdata download (Thanks to @redbaron)
• #518: Make the validation error message when KMS failed more friendly
• #528: Minor fixup for etcd unit files (Thanks to @redbaron)
• #554: Deprecate externalDNSName/createRecordSet/hostedZoneId
• #556: Export worker stack names and worker IAM role ARNs (Thanks to @jpb)
• #603: controller.loadBalancer is deprecated use apiEndpoints[].loadBalancer (Thanks to @Vrtak-CZ)
• #604: Change API endpoint ELB health check to SSL:443 (Thanks to @cheungpat)

Documentation updates

• #533: Add documentation for administrating etcd cluster
• #557: Fix hyperlink to restore script in Readme.md (Thanks to @jollinshead and @c-knowles)

v0.9.6-rc.7

Fixes

• #613: Fix typo in command listing (Thanks to @simonwydooghe)

v0.9.6-rc.6

Features

• #568: Dex integration (Thanks to @camilb)
• #589: Bump to calico 2.1.4 (Thanks to @redbaron)
• #602: Bump calico ctl to 1.1.3 (Thanks to @redbaron)

Fixes

• #594: Fix syntax error (Thanks to @danielfm)
• #606: Fix certs path when TLS bootstrapping is enabled (Thanks to @danielfm)
• #610: Fix elasticFileSystemId only on node pools (Thanks to @Vrtak-CZ for reporting)

Improvements

• #603: controller.loadBalancer is deprecated use apiEndpoints[].loadBalancer (Thanks to @Vrtak-CZ)
• #604: Change API endpoint ELB health check to SSL:443 (Thanks to @cheungpat)
• #609: Additional notes for autosave regarding s3 error (Thanks to @jollinshead)

Full change log
v0.9.6-rc.5...v0.9.6-rc.6

v0.9.6-rc.5

Actions required

Breaking changes

• #565 EFS PersistentVolume introduces Recycle policy which is a breaking change. Be aware that if all the persistent volume claims are removed, the EFS will recycle and remove all data.

Features

• #577: Add controller node labels if specified (Thanks to @cheungpat)
• #582: Update kube-dns to 1.14.1 (Thanks to @c-knowles for reporting)

Fixes

• #575: Quote security group refs for etcd, controller, and apiendpoints (Thanks to @soellman)
• #579: Fix no space left on device when audit loggin enabled for apiserver (Thanks to @ankon and @whereisaaron for reporting)
• #590: Fix etcd snapshots locations in S3 (Thanks to @cmcconnell1 for providing the important info to locate the issue)

Improvements

• #556: Export worker stack names and worker IAM role ARNs (Thanks to @jpb)
• #565: Add reclaim policy (Thanks to @kevtaylor)
• #576: Set --storage-backend to etcd2 if not using etcd3 (Thanks to @cheungpat)
• #581: Update kubelet flags (Thanks to @c-knowles for reporting)

Full change log
v0.9.6-rc.4...v0.9.6-rc.5