Using GitHub Large Runners for Kubeflow Infra

[andreyvelich]

Since the Kubeflow GitHub is part of the CNCF enterprise account, we have access to the GitHub Large Runners for our test infrastructure.

The following runners are currently available:

ubuntu-latest-4-cores
ubuntu-latest-8-cores
ubuntu-latest-16-cores
github-arm64-2c-8gb

To utilize these runners, we need to enable Repository Access in the Kubeflow GitHub organization setting. We've successfully implemented the ubuntu-latest-16-cores runner for Kubeflow Trainer E2Es: kubeflow/trainer#2470

As of now, we've enabled it for these repos:

kubeflow/trainer
kubeflow/manifests
kubeflow/mpi-operator

If other Kubeflow WGs require access to large runners for their testing infrastructure, please reach out, and we will help to enable it.

cc @kubeflow/kubeflow-steering-committee @kubeflow/wg-training-leads @kubeflow/wg-automl-leads @kubeflow/wg-notebooks-leads @kubeflow/wg-data-leads @kubeflow/wg-pipeline-leads @kubeflow/wg-manifests-leads @kubeflow/release-team

[juliusvonkohout]

How much memory do they have? I could need a large one to test a full kubeflow installation with 8 cores and 32GB

[andreyvelich]

Here is the spec: https://docs.github.com/en/actions/using-github-hosted-runners/using-larger-runners/about-larger-runners#specifications-for-general-larger-runners

I believe, 8 cores runner has 32GB

[juliusvonkohout]

Here is the spec: https://docs.github.com/en/actions/using-github-hosted-runners/using-larger-runners/about-larger-runners#specifications-for-general-larger-runners

I believe, 8 cores runner has 32GB

Thank you, can you enable this one then for kubeflow/manifests? What are the labels I need to add to use it selectively ?

[andreyvelich]

Thank you, can you enable this one then for kubeflow/manifests? What are the labels I need to add to use it selectively ?

Sure, let me enable it.
Just use the same labels as I did here: https://github.com/kubeflow/trainer/blob/master/.github/workflows/test-e2e.yaml#L9-L11

[tenzen-y]

Hey @andreyvelich, could you enable it in mpi-operator repo?
This is really helpful since mpi-operator maintains a lot of heavy MPI foundational container images.

[tenzen-y]

cc @terrytangyuan @alculquicondor

[andreyvelich]

@tenzen-y Sounds great! I enabled them for MPI Operator also.

[tenzen-y]

@tenzen-y Sounds great! I enabled them for MPI Operator also.

Thanks.