Helm chart and production installation guidance

[ajaysundark]

1. Introduce a Helm Chart to facilitate the installation of node-readiness-controller
2. Create a installation doc that details how to install the controller in a production-like environment, including the recommended 3-node setup (control-plane, platform-nodes, worker-nodes).

[ajaysundark]

Instructions:

1. Run make build-installer command from the root of the project directory. This will create a file named install.yaml in the dist/ directory. This will be the source installation file for creating the helm chart.
2. Use Chart name: node-readiness-controller
3. Follow best-practices (https://helm.sh/docs/chart_best_practices/#values).

Some suggestions -

a. Folder structure (scaffolding from here: https://helm.sh/docs/helm/helm_create/)

charts/
  node-readiness-controller/
    - Chart.yaml
    - values.yaml
    - templates/
       - deployment.yaml
       - namespace.yaml
       - rbac.yaml 
       - serviceaccount.yaml
       - ^decompose dist/install.yaml into separate k8s yaml files for better template management..
       - tests/
    - crds/
       - crd.yaml
    

b. separate 'crds' directory (https://helm.sh/docs/chart_best_practices/custom_resource_definitions)

c. extract default values into values.yaml

eg:

// values.yaml
image:
   repository: controller
   pullPolicy: IfNotPresent
   tag: "latest"

resources:
   limits:
     cpu: 500m
     memory: 128Mi
   requests:
     cpu: 10m
     memory: 64Mi
 
nodeSelector:
  node-role.kubernetes.io/control-plane: ""
 
tolerations:
- key: node-role.kubernetes.io/control-plane
   operator: Exists

4. Manually test the helm install command in a kind cluster. Ref: docs/TEST_README.md

[ajaysundark]

cc @sreeram-venkitesh

[sreeram-venkitesh]

/assign