add ipamd readiness latency metrics

[oliviassss]

What type of PR is this?

improvement

Which issue does this PR fix?:

What does this PR do / Why do we need it?:
Add 2 historgram prom metrics to track:

1. awscni_ipamd_startup_duration_seconds, the whole latency during IPAMD startup
2. awscni_ipamd_startup_aws_duration_secondsthe latency, the latency majorly from aws side during ipamd.New()
3. add p99 and p50 bucket in existing metrics awscni_aws_api_latency_ms

Testing done on this change:

deployed and tested in prom and grafana

# HELP awscni_ipamd_startup_aws_duration_seconds The duration of AWS initialization during IPAMD startup
# TYPE awscni_ipamd_startup_aws_duration_seconds histogram
awscni_ipamd_startup_aws_duration_seconds_bucket{success="true",le="0.005"} 0
awscni_ipamd_startup_aws_duration_seconds_bucket{success="true",le="0.01"} 0
awscni_ipamd_startup_aws_duration_seconds_bucket{success="true",le="0.025"} 0
awscni_ipamd_startup_aws_duration_seconds_bucket{success="true",le="0.05"} 0
awscni_ipamd_startup_aws_duration_seconds_bucket{success="true",le="0.1"} 0
awscni_ipamd_startup_aws_duration_seconds_bucket{success="true",le="0.25"} 0
awscni_ipamd_startup_aws_duration_seconds_bucket{success="true",le="0.5"} 0
awscni_ipamd_startup_aws_duration_seconds_bucket{success="true",le="1"} 0
awscni_ipamd_startup_aws_duration_seconds_bucket{success="true",le="2.5"} 1
awscni_ipamd_startup_aws_duration_seconds_bucket{success="true",le="5"} 1
awscni_ipamd_startup_aws_duration_seconds_bucket{success="true",le="10"} 1
awscni_ipamd_startup_aws_duration_seconds_bucket{success="true",le="+Inf"} 1
awscni_ipamd_startup_aws_duration_seconds_sum{success="true"} 1.845853322
awscni_ipamd_startup_aws_duration_seconds_count{success="true"} 1
# HELP awscni_ipamd_startup_duration_seconds The duration of IPAMD startup from process start to ready to serve CNI requests
# TYPE awscni_ipamd_startup_duration_seconds histogram
awscni_ipamd_startup_duration_seconds_bucket{failure_reason="",success="true",with_api_server="true",le="0.005"} 0
awscni_ipamd_startup_duration_seconds_bucket{failure_reason="",success="true",with_api_server="true",le="0.01"} 0
awscni_ipamd_startup_duration_seconds_bucket{failure_reason="",success="true",with_api_server="true",le="0.025"} 0
awscni_ipamd_startup_duration_seconds_bucket{failure_reason="",success="true",with_api_server="true",le="0.05"} 0
awscni_ipamd_startup_duration_seconds_bucket{failure_reason="",success="true",with_api_server="true",le="0.1"} 0
awscni_ipamd_startup_duration_seconds_bucket{failure_reason="",success="true",with_api_server="true",le="0.25"} 0
awscni_ipamd_startup_duration_seconds_bucket{failure_reason="",success="true",with_api_server="true",le="0.5"} 0
awscni_ipamd_startup_duration_seconds_bucket{failure_reason="",success="true",with_api_server="true",le="1"} 0
awscni_ipamd_startup_duration_seconds_bucket{failure_reason="",success="true",with_api_server="true",le="2.5"} 1
awscni_ipamd_startup_duration_seconds_bucket{failure_reason="",success="true",with_api_server="true",le="5"} 1
awscni_ipamd_startup_duration_seconds_bucket{failure_reason="",success="true",with_api_server="true",le="10"} 1
awscni_ipamd_startup_duration_seconds_bucket{failure_reason="",success="true",with_api_server="true",le="+Inf"} 1
awscni_ipamd_startup_duration_seconds_sum{failure_reason="",success="true",with_api_server="true"} 1.9611958980000002
awscni_ipamd_startup_duration_seconds_count{failure_reason="",success="true",with_api_server="true"} 1

Will this PR introduce any new dependencies?:

Will this break upgrades or downgrades? Has updating a running cluster been tested?:

Does this change require updates to the CNI daemonset config files to work?:

Does this PR introduce any user-facing change?:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[jaydeokar]

nit - Can we name this something like "awscni_ipamd_node_initialization_seconds" ? We do more than just aws api calls..

[oliviassss]

sg!

[jaydeokar]

This metric might not be that useful for cases where success: false as when ipamd initialization fails, it will be in a crashloop continuously.