The train/val split in domainnet

[FrankZhangRp]

we find that the train sets are kept the same size on all domains as the 'min_data_len' and we think there is a mistake:

FedBN/singleset/single_domainnet.py

Line 120 in 1caf83c

min_data_len = int(min_data_len * 0.05)

[FrankZhangRp]

if the 'trainset' is only 5% of the total dataset, the training process is too toy and unstable with about 100+ samples per client without pretrained weights. We think the train/val split may be 95% / 5%, but in this code is 5% / 5%.

[dwelcaslu]

I re-implemented this part like this for the domainet dataset (federated/fed_domainnet.py):

Replaced this:

    min_data_len = min(len(clipart_trainset), len(infograph_trainset), len(painting_trainset), len(quickdraw_trainset), len(real_trainset), len(sketch_trainset))
    val_len = int(min_data_len * 0.05)
    min_data_len = int(min_data_len * 0.05)

    clipart_valset   = torch.utils.data.Subset(clipart_trainset, list(range(len(clipart_trainset)))[-val_len:])
    clipart_trainset = torch.utils.data.Subset(clipart_trainset, list(range(min_data_len)))
    
    infograph_valset   = torch.utils.data.Subset(infograph_trainset, list(range(len(infograph_trainset)))[-val_len:])
    infograph_trainset = torch.utils.data.Subset(infograph_trainset, list(range(min_data_len)))
    
    painting_valset   = torch.utils.data.Subset(painting_trainset, list(range(len(painting_trainset)))[-val_len:])
    painting_trainset = torch.utils.data.Subset(painting_trainset, list(range(min_data_len)))

    quickdraw_valset   = torch.utils.data.Subset(quickdraw_trainset, list(range(len(quickdraw_trainset)))[-val_len:])
    quickdraw_trainset = torch.utils.data.Subset(quickdraw_trainset, list(range(min_data_len)))

    real_valset   = torch.utils.data.Subset(real_trainset, list(range(len(real_trainset)))[-val_len:])
    real_trainset = torch.utils.data.Subset(real_trainset, list(range(min_data_len)))

    sketch_valset   = torch.utils.data.Subset(sketch_trainset, list(range(len(sketch_trainset)))[-val_len:])
    sketch_trainset = torch.utils.data.Subset(sketch_trainset, list(range(min_data_len)))

By this:

    idx_train, idx_valid = train_test_split(range(len(clipart_trainset)), test_size=valid_size, random_state=42)
    clipart_valset =  torch.utils.data.Subset(clipart_trainset, idx_valid)
    clipart_trainset =  torch.utils.data.Subset(clipart_trainset, idx_train)

    idx_train, idx_valid = train_test_split(range(len(infograph_trainset)), test_size=valid_size, random_state=42)
    infograph_valset =  torch.utils.data.Subset(infograph_trainset, idx_valid)
    infograph_trainset =  torch.utils.data.Subset(infograph_trainset, idx_train)

    idx_train, idx_valid = train_test_split(range(len(painting_trainset)), test_size=valid_size, random_state=42)
    painting_valset =  torch.utils.data.Subset(painting_trainset, idx_valid)
    painting_trainset =  torch.utils.data.Subset(painting_trainset, idx_train)

    idx_train, idx_valid = train_test_split(range(len(quickdraw_trainset)), test_size=valid_size, random_state=42)
    quickdraw_valset =  torch.utils.data.Subset(quickdraw_trainset, idx_valid)
    quickdraw_trainset =  torch.utils.data.Subset(quickdraw_trainset, idx_train)

    idx_train, idx_valid = train_test_split(range(len(real_trainset)), test_size=valid_size, random_state=42)
    real_valset =  torch.utils.data.Subset(real_trainset, idx_valid)
    real_trainset =  torch.utils.data.Subset(real_trainset, idx_train)

    idx_train, idx_valid = train_test_split(range(len(sketch_trainset)), test_size=valid_size, random_state=42)
    sketch_valset =  torch.utils.data.Subset(sketch_trainset, idx_valid)
    sketch_trainset =  torch.utils.data.Subset(sketch_trainset, idx_train)

If you use valid_size=0.05 then you get train/val split with 95% / 5% percentages.