Add question-answer example for v2 trainer #2580

solanyn · 2025-04-01T16:19:50Z

What this PR does / why we need it:

This PR adds an example for the V2 Training Operator to train a question-answer model based on the HuggingFace recipe adapted to the training operator API.

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Adds a question-answer example mentioned in #2040 using the pytorch runtime.

Checklist:

Docs included if any changes are user facing

google-oss-prow · 2025-04-01T16:19:55Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign andreyvelich for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

review-notebook-app · 2025-04-01T16:19:55Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

Signed-off-by: solanyn <14799876+solanyn@users.noreply.github.com>

Electronic-Waste · 2025-04-02T03:58:04Z

examples/pytorch/question-answering/fine-tune-distilbert.ipynb

@@ -0,0 +1,592 @@
+{


Line #1. !pip install git+https://github.com/kubeflow/trainer.git@master#subdirectory=sdk
Could you please add a comment like: "# Change the version of SDK when we have the first release of Trainer SDK"?

Reply via ReviewNB

Electronic-Waste · 2025-04-02T03:58:04Z

examples/pytorch/question-answering/fine-tune-distilbert.ipynb

@@ -0,0 +1,592 @@
+{


Line #8. if r.name == "torch-distributed":
Can we remove this? Since torch_runtime is unused.

Reply via ReviewNB

Electronic-Waste · 2025-04-02T03:58:04Z

examples/pytorch/question-answering/fine-tune-distilbert.ipynb

@@ -0,0 +1,592 @@
+{


Line #17. # Uncomment this to distribute the TrainJob using GPU nodes.
It might be better to remove this comment:)

Reply via ReviewNB

Electronic-Waste · 2025-04-02T04:00:29Z

@solanyn Thanks for this amazing work! Appreciate your precious usecases for Trainer. I left my initial reviews for you.

/cc @kubeflow/wg-training-leads @astefanutti

Signed-off-by: solanyn <14799876+solanyn@users.noreply.github.com>

solanyn · 2025-04-02T04:24:16Z

Thanks @Electronic-Waste, I've updated the branch to address your comments!

Electronic-Waste · 2025-04-02T06:50:22Z

@solanyn Thanks for this! And welcome to the Kubeflow community!

/lgtm
/assign @kubeflow/wg-training-leads @astefanutti

coveralls · 2025-04-02T23:02:02Z

Pull Request Test Coverage Report for Build 14674591093

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.9%) to 67.356%

Totals
Change from base Build 14160634151:	0.9%
Covered Lines:	1758
Relevant Lines:	2610

💛 - Coveralls

andreyvelich

Thank you for this great contribution @solanyn !
I left a few comments.
It would be awesome to have this example as part of our first Kubeflow Trainer 2.0 release.
/assign @tenzen-y @saileshd1402 @kubeflow/wg-training-leads @astefanutti @shravan-achar @akshaychitneni

andreyvelich · 2025-04-24T12:45:02Z

examples/pytorch/question-answering/fine-tune-distilbert.ipynb

+    "        resources_per_node={\n",
+    "            \"cpu\": \"3\",\n",
+    "            \"memory\": \"8Gi\",\n",
+    "            \"nvidia.com/gpu\": 1,\n",


Is there a way for us to test test script on CPU maybe just for 1 Epoch or distilbert-base-uncased model is too large for CPU testing ?

Yeah, I can try it out on CPU and make the changes!

Thanks, it would be nice to see if we could run this example as part of our E2Es: https://github.com/kubeflow/trainer/blob/master/.github/workflows/test-e2e.yaml#L55

examples/pytorch/question-answering/fine-tune-distilbert.ipynb

andreyvelich · 2025-04-25T19:45:10Z

examples/pytorch/question-answering/fine-tune-distilbert.ipynb

+   "id": "c31bc8f2",
+   "metadata": {},
+   "source": [
+    "# Install the KubeFlow SDK and dependencies"


Let's remove the installation of Kubeflow SDK as prerequisite for this Notebook.
It is better to redirect users to this guide to get started with Kubeflow Trainer:
https://www.kubeflow.org/docs/components/trainer/getting-started/

andreyvelich · 2025-04-25T19:47:30Z

examples/pytorch/question-answering/fine-tune-distilbert.ipynb

+    "    \n",
+    "    trainer.train()\n",
+    "\n",
+    "    CloudPath(f'gs://{args[\"BUCKET\"]}/{args[\"MODEL_NAME\"]}').upload_from(args[\"MODEL_NAME\"])"


Can we put the location (e.g. gs://bucket-name) under args, so users can use s3 or gs with the CloudPath API ?

andreyvelich · 2025-04-25T19:49:58Z

examples/pytorch/question-answering/fine-tune-distilbert.ipynb

+    "\n",
+    "    squad = squad.train_test_split(test_size=0.2)\n",
+    "    \n",
+    "    tokenizer = AutoTokenizer.from_pretrained(\"distilbert/distilbert-base-uncased\")\n",


Can we use the MODEL_NAME arg for the model here

Suggested change

" tokenizer = AutoTokenizer.from_pretrained(\"distilbert/distilbert-base-uncased\")\n",

" tokenizer = AutoTokenizer.from_pretrained(\"distilbert/args["MODEL_NAME"]\")\n",

Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Signed-off-by: Andrew Chen <14799876+solanyn@users.noreply.github.com>

google-oss-prow · 2025-04-25T22:33:23Z

New changes are detected. LGTM label has been removed.

andreyvelich · 2025-04-25T22:40:26Z

/ok-to-test

google-oss-prow bot requested a review from Electronic-Waste April 1, 2025 16:19

google-oss-prow bot requested a review from jinchihe April 1, 2025 16:19

google-oss-prow bot added the size/XL label Apr 1, 2025

Add question-answer example

75062fd

Signed-off-by: solanyn <14799876+solanyn@users.noreply.github.com>

solanyn force-pushed the solanyn/question-answer-example branch from 8e336ff to 75062fd Compare April 1, 2025 16:20

Electronic-Waste reviewed Apr 2, 2025

View reviewed changes

google-oss-prow bot requested review from astefanutti and a team April 2, 2025 04:00

chore: remove unused lines, add TODO comment

a7f044b

Signed-off-by: solanyn <14799876+solanyn@users.noreply.github.com>

google-oss-prow bot assigned Electronic-Waste Apr 2, 2025

google-oss-prow bot added the lgtm label Apr 2, 2025

andreyvelich reviewed Apr 25, 2025

View reviewed changes

chore: update example description

ec8d1ea

Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Signed-off-by: Andrew Chen <14799876+solanyn@users.noreply.github.com>

google-oss-prow bot removed the lgtm label Apr 25, 2025

google-oss-prow bot added the ok-to-test label Apr 25, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add question-answer example for v2 trainer #2580

Add question-answer example for v2 trainer #2580

solanyn commented Apr 1, 2025 •

edited

Loading

google-oss-prow bot commented Apr 1, 2025

review-notebook-app bot commented Apr 1, 2025

Electronic-Waste Apr 2, 2025 •

edited

Loading

Electronic-Waste Apr 2, 2025 •

edited

Loading

Electronic-Waste Apr 2, 2025 •

edited

Loading

Electronic-Waste commented Apr 2, 2025

solanyn commented Apr 2, 2025

Electronic-Waste commented Apr 2, 2025

coveralls commented Apr 2, 2025 •

edited

Loading

andreyvelich left a comment •

edited

Loading

andreyvelich Apr 24, 2025

solanyn Apr 25, 2025

andreyvelich Apr 25, 2025

andreyvelich Apr 25, 2025

andreyvelich Apr 25, 2025

andreyvelich Apr 25, 2025

google-oss-prow bot commented Apr 25, 2025

andreyvelich commented Apr 25, 2025

	" tokenizer = AutoTokenizer.from_pretrained(\"distilbert/distilbert-base-uncased\")\n",
	" tokenizer = AutoTokenizer.from_pretrained(\"distilbert/args["MODEL_NAME"]\")\n",

Add question-answer example for v2 trainer #2580

Are you sure you want to change the base?

Add question-answer example for v2 trainer #2580

Conversation

solanyn commented Apr 1, 2025 • edited Loading

google-oss-prow bot commented Apr 1, 2025

review-notebook-app bot commented Apr 1, 2025

Electronic-Waste Apr 2, 2025 • edited Loading

Choose a reason for hiding this comment

Electronic-Waste Apr 2, 2025 • edited Loading

Choose a reason for hiding this comment

Electronic-Waste Apr 2, 2025 • edited Loading

Choose a reason for hiding this comment

Electronic-Waste commented Apr 2, 2025

solanyn commented Apr 2, 2025

Electronic-Waste commented Apr 2, 2025

coveralls commented Apr 2, 2025 • edited Loading

Pull Request Test Coverage Report for Build 14674591093

Warning: This coverage report may be inaccurate.

Details

💛 - Coveralls

andreyvelich left a comment • edited Loading

Choose a reason for hiding this comment

andreyvelich Apr 24, 2025

Choose a reason for hiding this comment

solanyn Apr 25, 2025

Choose a reason for hiding this comment

andreyvelich Apr 25, 2025

Choose a reason for hiding this comment

andreyvelich Apr 25, 2025

Choose a reason for hiding this comment

andreyvelich Apr 25, 2025

Choose a reason for hiding this comment

andreyvelich Apr 25, 2025

Choose a reason for hiding this comment

google-oss-prow bot commented Apr 25, 2025

andreyvelich commented Apr 25, 2025

solanyn commented Apr 1, 2025 •

edited

Loading

Electronic-Waste Apr 2, 2025 •

edited

Loading

Electronic-Waste Apr 2, 2025 •

edited

Loading

Electronic-Waste Apr 2, 2025 •

edited

Loading

coveralls commented Apr 2, 2025 •

edited

Loading

andreyvelich left a comment •

edited

Loading