Skip to content

Isssue when using torch.compile #3991

@tritam593

Description

@tritam593

System Info

I can run accelerate smoothly with 2 GPUs without using torch.compile, but when I enable torch.compile, I get an error: **Dynamo does not know how to trace method `set_runtime_stats_and_log` of class `Logger`**

Additional informaltion: My model run very well with torch.compile when I train my model from scraft, not using accelerate

Here is my config 
dynamo_plugin = TorchDynamoPlugin(
ackend="inductor",  # Options: "inductor", "aot_eager", "aot_nvfuser", etc.
     mode="default",      # Options: "default", "reduce-overhead", "max-autotune"
     fullgraph=False,
     dynamic=True,
   
 )

accelerator = Accelerator(dynamo_plugin=dynamo_plugin)


Here is the error:

Epoch 1 Train:   0%|                                   | 0/3750 [00:00<?, ?it/s]
[rank0]: Traceback (most recent call last):
[rank0]:   File "/kaggle/working/train_class.py", line 450, in <module>
[rank0]:     main()
[rank0]:   File "/kaggle/working/train_class.py", line 436, in main
[rank0]:     train_model(
[rank0]:   File "/kaggle/working/train_class.py", line 230, in train_model
[rank0]:     outputs = model(input_ids, attention_mask=attention_mask)
[rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 465, in __call__
[rank0]:     return super().__call__(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1787, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 963, in compile_wrapper
[rank0]:     raise e.with_traceback(None) from e.__cause__  # User compiler error
[rank0]:     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: torch._dynamo.exc.Unsupported: Unsupported method call
[rank0]:   Explanation: Dynamo does not know how to trace method `set_runtime_stats_and_log` of class `Logger`
[rank0]:   Hint: Avoid calling `Logger.set_runtime_stats_and_log` in your code.
[rank0]:   Hint: Please report an issue to PyTorch.

[rank0]:   Developer debug context: call_method UserDefinedObjectVariable(Logger) set_runtime_stats_and_log [] {}

[rank0]:  For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0156.html

[rank0]: from user code:
[rank0]:    File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/external_utils.py", line 69, in inner
[rank0]:     return fn(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/nn/parallel/distributed.py", line 1662, in forward
[rank0]:     inputs, kwargs = self._pre_forward(*inputs, **kwargs)
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/nn/parallel/distributed.py", line 1533, in _pre_forward
[rank0]:     self.logger.set_runtime_stats_and_log()

[rank0]: Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"

tôi thấy lỗi này do chương trình gọi self.logger.set_runtime_stats_and_log() trong /usr/local/lib/python3.12/dist-packages/torch/nn/parallel/distributed.py

Information

  • The official example scripts
  • My own modified scripts

Tasks

  • One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
  • My own task or dataset (give details below)

Reproduction

outputs = model(input_ids, attention_mask=attention_mask)

Expected behavior

the model can be compile

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions