the error when save optimizer state

[zyuanzi]

Hi,

I also encountered issues with the optimizer state when I fine tuning the llama model in ada 6000.
My torch is 2.7.1

error:
File "/root/.local/lib/python3.10/site-packages/torch/distributed/fsdp/_optim_utils.py", line 1382, in _convert_all_state_info
assert dtype == info.dtype
AssertionError

can you help me to solve this problem?

best regards!

[haochengxi]

Hi, I believe you are trying to save the optimizer states in an unsharded way. We currently only support sharding the optimizer in a sharded way in FSDP1. But if you meet this issue when trying to save the checkpoints in a sharded way, please let me know.

[zyuanzi]

Thank you for your answer. I will check my code again. Thank you！！