Add support for dataset and model initializers in container backend

[Fiona-Waters]

What you would like to be added?

Kubeflow Trainer uses Initializers to pull datasets and models required for TrainJobs. These initialializers pull the required dataset and/or model into a shared volume prior to starting the TrainJob nodes. We should add support for this in the container backend.

Why is this needed?

This is needed to bring the container backend implementation in line with the kubernetes backend.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[kramaranya]

/area local
/remove-label lifecycle/needs-triage
/good-first-issue

[google-oss-prow]

@kramaranya:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

Details

In response to this:

/area local
/remove-label lifecycle/needs-triage
/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[aniketpati1121]

/assign

[HKanoje]

Hi @Fiona-Waters ! I'm interested in working on this issue. I've analyzed the Kubernetes backend implementation and have a plan to add initializer support to the container backend following the same patterns.

My approach would be to:

• Create separate initializer containers that run before training starts
• Add utility functions for building initializer commands/env vars
• Ensure the implementation is consistent with the Kubernetes backend

Happy to discuss the approach!

[immortal71]

/assign

[adity1raut]

/assign

[Fiona-Waters]

Hey everyone, I'm reassigning this based on who had tried to assign it first. @aniketpati1121 please let us know if you are currently working on it, otherwise someone else can take it over.
@adity1raut @immortal71 @HKanoje please take a look at other open issues, we appreciate your eagerness to contribute.
/assign @aniketpati1121
/unassign @adity1raut
/unassign @immortal71