xdsclient: Fix flakyness in TestResourceUpdateMetrics in the case of repeated NACKs

[purnesh42H]

Fixes: #8344

Successful forge test

In a rare case scenario of NACK, the xDS client can emit metrics in quick succession due to the interaction sequence between the xDS client (in grpc-go) and the test management server (using go-control-plane) after the initial invalid update. The test uses testMetricsReporter for xdsclient to report metrics which calls metricsReporter.ReportMetric() to report invalid and valid xds resource updates. The flakiness is due to timeout in r.metricsCh.Send(m) in the case burst of redundant updates because the metricsCh in testMetricsReporter is a testutils.Channel initialized with a size of 1. A Send operation on this channel will block if the channel is full. See #8344 (comment)

We can mitigate the race by making the metric sends non-block by dropping redundant update and replacing with the new one. Similar to what is being done in internal grpc metrics recorder https://github.com/grpc/grpc-go/blob/master/internal/testutils/stats/test_metrics_recorder.go#L125-L126

RELEASE NOTES: None

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.34%. Comparing base (4cab0e6) to head (a507ef2).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8363      +/-   ##
==========================================
+ Coverage   82.27%   82.34%   +0.06%     
==========================================
  Files         419      419              
  Lines       42039    42039              
==========================================
+ Hits        34587    34616      +29     
+ Misses       5994     5968      -26     
+ Partials     1458     1455       -3     

see 33 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.