Skip to content

feat: add support for tracking TrainJob progress and training metrics #3227

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
andreyvelich merged 65 commits into from
Mar 13, 2026
Merged

feat: add support for tracking TrainJob progress and training metrics #3227

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
0f13bb7
feat(api): add progress types to api
robert-bell Feb 2, 2026
0cbc25f
feat: progress server stub
robert-bell Feb 3, 2026
d2e1162
feat: setup tls for server
robert-bell Feb 4, 2026
dac72c7
refactor: pull out middleware
robert-bell Feb 5, 2026
260059f
refactor: remove logger message noise from tests
robert-bell Feb 5, 2026
69a6e6a
fix: send server error logs to main logger
robert-bell Feb 5, 2026
3c4578f
feat: update the trainer status
robert-bell Feb 5, 2026
fc06c46
feat: use separate client to avoid impacting main reconciler client-s…
robert-bell Feb 10, 2026
ee90a96
feat: configure progress server in helm chart and configmap
robert-bell Feb 10, 2026
e99228b
fix: prevent race condition
robert-bell Feb 10, 2026
13973d1
feat: inject progress runtime config
robert-bell Feb 13, 2026
c056bf1
fix: progress client config
robert-bell Feb 13, 2026
0e49980
fix: update generated config
robert-bell Feb 13, 2026
d6a7739
feat: validate auth tokens in endpoint
robert-bell Feb 13, 2026
d075c31
test: validate auth tokens in endpoint
robert-bell Feb 13, 2026
583bfcb
refactor: tidy tests to use same format as other tests
robert-bell Feb 13, 2026
6ef2806
fix: look up the cluster issuer from the service account token
robert-bell Feb 13, 2026
caba54a
refactor: rename authn middleware
robert-bell Feb 16, 2026
8e0d083
feat: add authn to status update endpoint
robert-bell Feb 16, 2026
b15a818
refactor: tidy up reading in-cluster service account token
robert-bell Feb 16, 2026
bb508ee
fix: rbac permissions
robert-bell Feb 16, 2026
4cbae62
fix: better error response
robert-bell Feb 16, 2026
1a7cd28
refactor: move kubebuilder annotation higher up
robert-bell Feb 16, 2026
55b80c0
fix: make lastUpdateTime required
robert-bell Feb 17, 2026
bf3b439
chore: fix config tests after rebase
robert-bell Feb 19, 2026
08eccde
test: update config test with new progress server config
robert-bell Feb 19, 2026
f9ddc20
test: add progress tracking e2e test
robert-bell Feb 23, 2026
9b86419
fix: allow feature gates to be configurable from command line argument
robert-bell Feb 24, 2026
f9ffc34
refactor: reuse shared progress StatusUrl
robert-bell Feb 23, 2026
0bd2a0e
test: tidy up tests
robert-bell Feb 23, 2026
34501ec
refactor: allow plugins to use config
robert-bell Feb 23, 2026
77cff75
refactor: centralise shared constants
robert-bell Feb 24, 2026
a5ba603
fix: disable http2 in progress server tls opts if disabled by config
robert-bell Feb 26, 2026
7c2f523
fix: minor improvements suggested by copilot
robert-bell Feb 26, 2026
a493c4e
chore: rename progress server => status server
robert-bell Feb 26, 2026
20e7126
chore: update type doc
robert-bell Feb 26, 2026
6ca0ed6
chore: update name of feature gate
robert-bell Feb 26, 2026
a2e029b
fix: after rebase
robert-bell Feb 26, 2026
f702ab8
chore: fix typo
robert-bell Feb 26, 2026
e93cc03
chore: rename handler
robert-bell Feb 27, 2026
0c3bb3f
fix: run status server on all replicas
robert-bell Mar 2, 2026
d00695d
fix: deep copy the rest config
robert-bell Mar 2, 2026
5cb36f6
refactor: use ssa
robert-bell Mar 2, 2026
8c0be22
refactor: separate token audience per train job
robert-bell Mar 2, 2026
1bd4d23
chore: rename status server
robert-bell Mar 3, 2026
96a6841
fix: revert adding status pkg to manifest gen paths
robert-bell Mar 3, 2026
89ab448
chore: update log messages
robert-bell Mar 3, 2026
83a29b8
chore: update api docs
robert-bell Mar 3, 2026
4a691ce
chore: update format
robert-bell Mar 3, 2026
5181861
fix: revert inadvertent change
robert-bell Mar 5, 2026
0b86114
chore: update api name and type docs
robert-bell Mar 5, 2026
441f00f
refactor: initialise oidc provider in manager startup
robert-bell Mar 6, 2026
bdad285
fix: allow empty update request
robert-bell Mar 6, 2026
c0d7440
fix: wait for server shut down
robert-bell Mar 6, 2026
fbf1b5b
fix: handle missing train job
robert-bell Mar 6, 2026
3fc1347
refactor: update env var names
robert-bell Mar 12, 2026
18eaccb
chore: rename packages, plugins, feature gate
robert-bell Mar 13, 2026
3bfc54e
refactor: update env var names
robert-bell Mar 13, 2026
8b6910a
chore: regenerate after rebase
robert-bell Mar 13, 2026
053b59e
chore: fixes after rebase
robert-bell Mar 13, 2026
4ee1a06
chore: minor doc updates
robert-bell Mar 13, 2026
70529fe
chore: rename status server config
robert-bell Mar 13, 2026
899149c
chore: replace placeholder config with nil
robert-bell Mar 13, 2026
126de75
chore: revert separate runtime utils package
robert-bell Mar 13, 2026
0ef0b91
chore: remove unused config from tests
robert-bell Mar 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions api/openapi-spec/swagger.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions api/python_api/kubeflow_trainer_api/models/__init__.py
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

89 changes: 89 additions & 0 deletions api/python_api/kubeflow_trainer_api/models/trainer_v1alpha1_metric.py
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 8 additions & 2 deletions api/python_api/kubeflow_trainer_api/models/trainer_v1alpha1_train_job_status.py
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

102 changes: 102 additions & 0 deletions api/python_api/kubeflow_trainer_api/models/trainer_v1alpha1_trainer_status.py
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading