feat: Add training progression tracking feature for RHAITrainer type implementation

[abhijeet-dhumal]

What this PR does / why we need it:

RHOAIENG-38273
Implement controller support for polling and tracking training job progression from HTTP metrics endpoints exposed by experimental trainers (e.g., TransformersTrainer).

Related to :
opendatahub-io/kubeflow-sdk#21

Sample TrainJob tested it with :
wrapper-test.yaml

apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainJob
metadata:
  annotations:
    trainer.opendatahub.io/framework: transformers
    trainer.opendatahub.io/metrics-poll-interval: '30'
    trainer.opendatahub.io/metrics-port: '28080'
    trainer.opendatahub.io/progression-tracking: enabled
    trainer.opendatahub.io/trainerStatus: '{
            "progressPercentage": 35,
            "estimatedRemainingSeconds": 354,
            "estimatedRemainingTimeSummary": "5 minutes",
            "currentStep": 132,
            "totalSteps": 375,
            "currentEpoch": 1,
            "totalEpochs": 3,
            "trainMetrics": {
              "grad_norm": 0.034348633140325546,
              "learning_rate": 0.000013120000000000001,
              "loss": 0.0015
            },
            "evalMetrics": {
              "eval_loss": 0.0012287567369639874,
              "eval_runtime": 9.2987,
              "eval_samples_per_second": 21.508,
              "eval_steps_per_second": 2.689
            },
            "lastUpdatedTime": "2025-11-20T09:50:13Z"
  }'
...

Sample suceeded trainjob status progression metrics: (the annotation is chnaged a bit in below picture to make all annotations viewable) -

Pre-stop hook injected by controller :

Checklist:

• Docs included if any changes are user facing

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[robert-bell]

A few small nits, but lgtm otherwise. I'm happy for this to be merged after those nits are resolved.
/lgtm

[astefanutti]

Awesome work!

[astefanutti]

Thanks @abhijeet-dhumal!

[abhijeet-dhumal]

Thanks a million @astefanutti @robert-bell 🙌