Add support for bootstrap pod annotations

[patelsmit32123]

Description

Currently, bootstrap pod spec doesn't support adding annotations. This change adds the support.

Annotations were supported for node pools but was missing in helm charts, so added that as well.

Issues Resolved

#1098

Check List

• Commits are signed per the DCO using --signoff
• Unittest added for the new/changed functionality and all unit tests are successful
• Customer-visible features documented
• No linter warnings (make lint)

If CRDs are changed:

• CRD YAMLs updated (make manifests) and also copied into the helm chart
• Changes to CRDs documented

Please refer to the PR guidelines before submitting this pull request.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[patelsmit32123]

@prudhvigodithi please review when you can

[prudhvigodithi]

Thanks @patelsmit32123, i'm ok with this change, but just curious the lifetime of bootstrap pod is short and start a new cluster without this bootstrap pod, is there a strong reason to add the annotations? I would love to see a proposal/RFC to completely remove the bootstrap logic and start the cluster directly with user defined node pools while handling the security settings as well.
CC @rootxrishabh

[patelsmit32123]

Thanks @patelsmit32123, i'm ok with this change, but just curious the lifetime of bootstrap pod is short and start a new cluster without this bootstrap pod, is there a strong reason to add the annotations?

At Uber, we don't have per-pod-ip (only host ip) and use dynamic ports for colocation of pods, so the k8s scheduler assigns ports via specific annotations and we need to assign one for bootstrap pod as well.

[synhershko]

LGTM