Add Metrics to base class

[damccorm]

Add a metric that will track prediction failures.

num_failed_inferences

``

Add metrics that track data on side loaded models

{}num_model_loads{}, {}num_model_versions{}, and curr_model_version

``

curr_model_version is less a metric and more of a static value. Part of this FR will be to figure out how that fits.

Imported from Jira BEAM-14043. Original Jira may contain additional context.
Reported by: Ryan.Thompson.
Subtask of issue #21435

[damccorm]

This probably comes after #22970

[damccorm]

Actually, I think there could be value in adding num_failed_inferences before worrying about the side input model loading stuff. That would probably involve putting some error handling around our run_inference call:

beam/sdks/python/apache_beam/ml/inference/base.py

Line 418 in 2d4f61c

result_generator = self._model_handler.run_inference(

and have that update the metrics count. @BjornPrime as you free up would you mind working on this? We probably can't totally close out the issue, but we can make a dent in it.

cc/ @yeandy

[BjornPrime]

.take-issue

[yeandy]

Several questions around the error handling:

1. We're working with a batch of data. If inference fails, do we consider that to be 1 num_failed_inferences? Or perhaps it should be renamed to num_failed_batch_inferences? It may depend on framework, but is there a way to figure out which specific data points in a batch fail? In which case we can also track num_failed_inferences.
2. Do we want to do any retries for the inference call? Maybe we shouldn't, since the default behavior for a model inference failure is to throw an exception. But if we do want retries, how many times? If it succeeds in, say 5 tries, do we swallow the error and ignore increasing the counter? Or do we still increase the counter?
3. Or should the pipeline always fail?
4. Should we give users a flag to choose between the behavior?
   @rezarokni Do you have any idea on points 2-4?