Repository navigation
Commit b628a94
[SPARK-59985][ML][PYTHON] Fix malformed warning in TorchDistributor w…
### What changes were proposed in this pull request?
Fix the warning that `TorchDistributor._run_distributed_training` emits when the log streaming server fails to start. The message had a stray comma, so the error text was passed to `logger.warning` as an extra positional argument. Formatting the record then raised `TypeError: not all arguments converted during string formatting`. The warning now uses a single format string with the exception as its argument (`"... error: %r.", e`).
### Why are the changes needed?
The fallback added in SPARK-44909 still works (training continues with the port set to `-1`), but the warning that tells users why worker logs are missing is broken:
- With default logging settings, users see a `--- Logging error ---` traceback that looks like a crash instead of a one-line warning.
- With `logging.raiseExceptions = False`, the warning is silently dropped.
- `record.getMessage()` raises, so the warning cannot be asserted on.
### Does this PR introduce _any_ user-facing change?
Yes. When the log streaming server fails to start, users now get the intended warning instead of a logging error traceback.
Before:
```
--- Logging error ---
Traceback (most recent call last):
...
TypeError: not all arguments converted during string formatting
Call stack:
File ".../pyspark/ml/torch/distributor.py", line 815, in _run_distributed_training
self.logger.warning(
Message: 'Start torch distributor log streaming server failed, You cannot receive logs sent from distributor workers, '
Arguments: ("error: OSError('Address already in use').",)
```
After:
```
Start torch distributor log streaming server failed, You cannot receive logs sent from distributor workers, error: OSError('Address already in use').
```
### How was this patch tested?
Added `test_log_streaming_server_start_failure_warns` to `TorchDistributorBaselineUnitTestsMixin`. It makes `LogStreamingServer.start` raise and checks that execution reaches task-function creation with log streaming disabled and that exactly one warning is logged containing the error. The test fails with the `TypeError` above before the fix and passes after it. The rest of `TorchDistributorBaselineUnitTests` still passes.
```
python/run-tests --testnames 'pyspark.ml.torch.tests.test_distributor TorchDistributorBaselineUnitTests'
```
### Was this patch authored or co-authored using generative AI tooling?
Co Authered with claude.
Closes #59242 from firasbouzazi/SPARK-59985-fix-torch-distributor-warning.
Authored-by: Firas Bouzazi <115628206+firasbouzazi@users.noreply.github.com>
Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com>
(cherry picked from commit d7635d0)
Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com>1 parent 72ec2bf commit b628a94
2 files changed
Lines changed: 28 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
809 | 809 | | |
810 | 810 | | |
811 | 811 | | |
812 | | - | |
813 | | - | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
814 | 815 | | |
815 | 816 | | |
816 | 817 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
275 | 275 | | |
276 | 276 | | |
277 | 277 | | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
278 | 303 | | |
279 | 304 | | |
280 | 305 | | |
| |||
0 commit comments