What happened?
In the TorchTune fine-tuning path, passing a plain string where a Loss or DataFormat
enum is expected raises AttributeError from inside the SDK instead of the ValueError
that the surrounding code clearly intends. dtype in the same function already behaves
correctly, so the three fields are inconsistent.
from kubeflow.trainer.backends.kubernetes import utils
from kubeflow.trainer.types import types
# dtype: guarded, reports the problem clearly
utils.get_args_using_torchtune_config(types.TorchTuneConfig(dtype="bf16"))
# ValueError: Invalid dtype: bf16.
# loss: no guard at all
utils.get_args_using_torchtune_config(
types.TorchTuneConfig(loss="torchtune.modules.loss.CEWithChunkedOutputLoss")
)
# AttributeError: 'str' object has no attribute 'value'
# dataset source: guard exists but its own error message crashes
utils.get_args_from_dataset_preprocess_config(
types.TorchTuneInstructDataset(source="json")
)
# AttributeError: 'str' object has no attribute 'value'1. loss is never type-checked. kubeflow/trainer/backends/kubernetes/utils.py
compares as follows:
# dtype - guarded
if fine_tuning_config.dtype:
if not isinstance(fine_tuning_config.dtype, types.DataType):
raise ValueError(f"Invalid dtype: {fine_tuning_config.dtype}.")
args.append(f"dtype={fine_tuning_config.dtype.value}")
# loss - not guarded
if fine_tuning_config.loss:
args.append(f"loss={fine_tuning_config.loss.value}")2. The ValueError for an invalid dataset source is unreachable. In
get_args_from_dataset_preprocess_config:
if dataset_preprocess_config.source:
if not isinstance(dataset_preprocess_config.source, types.DataFormat):
raise ValueError(f"Invalid data format: {dataset_preprocess_config.source.value}.")The branch is entered only when source is not a DataFormat, but the message then
evaluates .value on that non-enum object. Building the exception raises AttributeError
first, so the intended ValueError can never be raised.
This is reproducible in pure Python without a cluster, since it only involves the SDK's
argument-building logic.
What did you expect to happen?
Both cases should raise ValueError at the point where the configuration is converted to
trainer args, matching the existing dtype behaviour, so the user sees which field is
wrong instead of an AttributeError from inside the SDK.
TorchTuneConfig(loss="...") -> ValueError: Invalid loss: ...TorchTuneInstructDataset(source="json") -> ValueError: Invalid data format: json.
Passing a string instead of the enum member is an easy mistake to make when writing a
fine-tuning configuration by hand, and DataFormat.JSON.value is literally "json", so
the invalid value often looks correct at a glance.
Environment
Kubernetes version:
$ kubectl version
Not applicable - reproducible without a cluster.
Kubeflow Trainer version:
$ kubectl get pods -n kubeflow -l app.kubernetes.io/name=trainer -o jsonpath="{.items[*].spec.containers[*].image}"
Not applicable - reproducible without a cluster.Kubeflow Python SDK version:
$ pip show kubeflow
Version: 0.5.0rc0
Also present on main at commit 97ec647.
Impacted by this bug?
Give it a 👍 We prioritize the issues with most 👍
What happened?
In the TorchTune fine-tuning path, passing a plain string where a
LossorDataFormatenum is expected raises
AttributeErrorfrom inside the SDK instead of theValueErrorthat the surrounding code clearly intends.
dtypein the same function already behavescorrectly, so the three fields are inconsistent.
1.
lossis never type-checked.kubeflow/trainer/backends/kubernetes/utils.pycompares as follows:
2. The
ValueErrorfor an invalid dataset source is unreachable. Inget_args_from_dataset_preprocess_config:The branch is entered only when
sourceis not aDataFormat, but the message thenevaluates
.valueon that non-enum object. Building the exception raisesAttributeErrorfirst, so the intended
ValueErrorcan never be raised.This is reproducible in pure Python without a cluster, since it only involves the SDK's
argument-building logic.
What did you expect to happen?
Both cases should raise
ValueErrorat the point where the configuration is converted totrainer args, matching the existing
dtypebehaviour, so the user sees which field iswrong instead of an
AttributeErrorfrom inside the SDK.TorchTuneConfig(loss="...")->ValueError: Invalid loss: ...TorchTuneInstructDataset(source="json")->ValueError: Invalid data format: json.Passing a string instead of the enum member is an easy mistake to make when writing a
fine-tuning configuration by hand, and
DataFormat.JSON.valueis literally"json", sothe invalid value often looks correct at a glance.
Environment
Kubernetes version:
Kubeflow Trainer version:
$ kubectl get pods -n kubeflow -l app.kubernetes.io/name=trainer -o jsonpath="{.items[*].spec.containers[*].image}" Not applicable - reproducible without a cluster.Kubeflow Python SDK version:
Also present on
mainat commit 97ec647.Impacted by this bug?
Give it a 👍 We prioritize the issues with most 👍