Usage of composer.utils.dist.get_node_signal_file_name()

[Andrew-Wyn]

I encountered a bug during the usage of composer.utils.dist.get_node_signal_file_name.

Setup

• llm-foundry==release/v0.17.1

If I execute a training script on a single node I have no issue and the training starts smoothly. When I set up the multinode configuration, an error comes out.

[rank0]: Traceback (most recent call last):
[rank0]:   File "/leonardo/home/userexternal/lmoroni0/__Work/llm-foundry/scripts/train/train.py", line 9, in <module>
[rank0]:     train_from_yaml(yaml_path, args_list)
[rank0]:   File "/leonardo/home/userexternal/lmoroni0/__Work/llm-foundry/llmfoundry/command_utils/train.py", line 662, in train_from_yaml
[rank0]:     return train(yaml_cfg)
[rank0]:            ^^^^^^^^^^^^^^^
[rank0]:   File "/leonardo/home/userexternal/lmoroni0/__Work/llm-foundry/llmfoundry/command_utils/train.py", line 366, in train
[rank0]:     tokenizer = build_tokenizer(tokenizer_name, tokenizer_kwargs)
[rank0]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/leonardo/home/userexternal/lmoroni0/__Work/llm-foundry/llmfoundry/utils/builders.py", line 545, in build_tokenizer
[rank0]:     os.remove(signal_file_path)
[rank0]: FileNotFoundError: [Errno 2] No such file or directory: '._signal_file_node0_KfmwNg'

After a bit of debugging, I noticed that the get_node_signal_file_name return the same name for each node, resulting in a race condition, since each node use the same file to assess inter-node concurrency.

I fixed such error using a previous methodology:

llmfoundry/utils/builders.py line:497

 f'.node_{dist.get_node_rank()}_local_rank0_completed_tokenizer_setup'

I think that is something related to the composer library. If this workaround is something sound, I can open an pull request.

[dakinggg]

What version of Comopser do you have? the get_node_signal_file_name function in Composer does have a node rank in it. And actually the string you shared looks like it also does (the 0 after node).

[Andrew-Wyn]

I am using mosaicml==0.28.0 which is the minimum version supported by the release/v0.17.1.

I noticed that in main now mosaicml is supported started from 0.29.0

actually the string you shared looks like it also does (the 0 after node).

Yes, is correct, but in my setup get_node_signal_file_name return always 0 as node-rank (while should be incremental for each node). Looking at such function the only difference (w.r.t. my workaround) is the usage of broadcast_object_list.

[gusario]

Hey! I've got stumbled across the very same bug. I've tried to fix it with code provided by @Andrew-Wyn, but my train failed on further usage of get_node_signal_file_name in method build_inner_model in composer_model and in Trainer.
For reference: I used the latest version of llm-foundry and composer
This error only happens in my case only if number of nodes > 1 (I've tested using 2 nodes which had 8h100 each) If I launched using single node it worked without errors.

To fix problem in case of multi-node I've put this couple lines of code at the start of my train file and it worked:

def get_node_signal_file_name_kostyl(rng: Optional[random.Random] = None) -> str:
    """Returns a file name to use for a file based wait within a node.

    The file name will contain a randomly generated string to avoid conflicts.
    Note: This file name will be the same on each node, so that it can be used for a file based wait.

    Returns:
        str: The name of the file that will be created to signal the end of a node's training.
    """

    random_string = "kek"
    node_rank = dist.get_node_rank()
    file_name = f"._signal_file_node{node_rank}_{random_string}"
    return file_name

dist.get_node_signal_file_name = get_node_signal_file_name_kostyl


def main(...) -> None:
    ...

What I've tried and it didn't work for me:

1. Get rid of dist.broadcast_object_list:

def get_node_signal_file_name_kostyl(rng: Optional[random.Random] = None) -> str:
    """Returns a file name to use for a file based wait within a node.

    The file name will contain a randomly generated string to avoid conflicts.
    Note: This file name will be the same on each node, so that it can be used for a file based wait.

    Returns:
        str: The name of the file that will be created to signal the end of a node's training.
    """
    if rng is None:
        rng = random.Random()

    random_string = ''.join(rng.choices(string.ascii_letters + string.digits, k=6))
    node_rank = dist.get_node_rank()
    file_name = f"._signal_file_node{node_rank}_{random_string}"
    return file_name

Error I've got was:

.pdm_venv_new/lib/python3.11/site-packages/llmfoundry/utils/builders.py", line 542, in build_tokenizer
2025-03-13T14:35:20.793244541Z [1,0]<stderr>:    os.remove(signal_file_path)
2025-03-13T14:35:20.793251051Z [1,0]<stderr>:FileNotFoundError: [Errno 2] No such file or directory: '._signal_file_node0_ADm9Xb'

2. Get rid of rng name and keep broadcast:

def get_node_signal_file_name_kostyl(rng: Optional[random.Random] = None) -> str:
    """Returns a file name to use for a file based wait within a node.

    The file name will contain a randomly generated string to avoid conflicts.
    Note: This file name will be the same on each node, so that it can be used for a file based wait.

    Returns:
        str: The name of the file that will be created to signal the end of a node's training.
    """

    random_string = "kek"
    node_rank = dist.get_node_rank()
    file_name_list = [f"._signal_file_node{node_rank}_{random_string}"]
    dist.broadcast_object_list(file_name_list, src=0)
    return file_name_list[0]

Error I've got:

pdm_venv_new/lib/python3.11/site-packages/llmfoundry/utils/builders.py", line 542, in build_tokenizer
2025-03-14T11:05:06Z [1,0]<stderr>:    os.remove(signal_file_path)
2025-03-14T11:05:06Z [1,0]<stderr>:FileNotFoundError: [Errno 2] No such file or directory: '._signal_file_node0_kek'

I hope that this will shed some light on root cause of this problem