System Info
I can run accelerate smoothly with 2 GPUs without using torch.compile, but when I enable torch.compile, I get an error: **Dynamo does not know how to trace method `set_runtime_stats_and_log` of class `Logger`**
Additional informaltion: My model run very well with torch.compile when I train my model from scraft, not using accelerate
Here is my config
dynamo_plugin = TorchDynamoPlugin(
ackend="inductor", # Options: "inductor", "aot_eager", "aot_nvfuser", etc.
mode="default", # Options: "default", "reduce-overhead", "max-autotune"
fullgraph=False,
dynamic=True,
)
accelerator = Accelerator(dynamo_plugin=dynamo_plugin)
Here is the error:
Epoch 1 Train: 0%| | 0/3750 [00:00<?, ?it/s]
[rank0]: Traceback (most recent call last):
[rank0]: File "/kaggle/working/train_class.py", line 450, in <module>
[rank0]: main()
[rank0]: File "/kaggle/working/train_class.py", line 436, in main
[rank0]: train_model(
[rank0]: File "/kaggle/working/train_class.py", line 230, in train_model
[rank0]: outputs = model(input_ids, attention_mask=attention_mask)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 465, in __call__
[rank0]: return super().__call__(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1787, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 963, in compile_wrapper
[rank0]: raise e.with_traceback(None) from e.__cause__ # User compiler error
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: torch._dynamo.exc.Unsupported: Unsupported method call
[rank0]: Explanation: Dynamo does not know how to trace method `set_runtime_stats_and_log` of class `Logger`
[rank0]: Hint: Avoid calling `Logger.set_runtime_stats_and_log` in your code.
[rank0]: Hint: Please report an issue to PyTorch.
[rank0]: Developer debug context: call_method UserDefinedObjectVariable(Logger) set_runtime_stats_and_log [] {}
[rank0]: For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0156.html
[rank0]: from user code:
[rank0]: File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/external_utils.py", line 69, in inner
[rank0]: return fn(*args, **kwargs)
[rank0]: File "/usr/local/lib/python3.12/dist-packages/torch/nn/parallel/distributed.py", line 1662, in forward
[rank0]: inputs, kwargs = self._pre_forward(*inputs, **kwargs)
[rank0]: File "/usr/local/lib/python3.12/dist-packages/torch/nn/parallel/distributed.py", line 1533, in _pre_forward
[rank0]: self.logger.set_runtime_stats_and_log()
[rank0]: Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
tôi thấy lỗi này do chương trình gọi self.logger.set_runtime_stats_and_log() trong /usr/local/lib/python3.12/dist-packages/torch/nn/parallel/distributed.py
Information
Tasks
Reproduction
outputs = model(input_ids, attention_mask=attention_mask)
Expected behavior
the model can be compile
System Info
Information
Tasks
no_trainerscript in theexamplesfolder of thetransformersrepo (such asrun_no_trainer_glue.py)Reproduction
outputs = model(input_ids, attention_mask=attention_mask)
Expected behavior
the model can be compile