About the CIFAR-10 dataset

[Tam-JQK]

This is an excellent piece of work！I like it.
However, when I ran:
python federated_main.py --mode task_heter --dataset cifar10 --num_classes 10 --num_users 20 --ways 5 --shots 100 --stdev 2 --rounds 110 --train_shots_max 110 --ld 0.1
I found that the results on cifar-10 dataset were far from the results in the paper.
I think my parameter settings are wrong, I read the paper carefully, but still haven't corrected the experimental results.
So I wanted to borrow your parameter settings.
Best regards.

[yuetan031]

Hi,

As I mentioned in the experimental parts of this paper, a ResNet18 pre-trained on ImageNet is used as the initial model.
“For CIFAR10, ResNet18 pre-trained on ImageNet (Krizhevsky, Sutskever, and Hinton 2017) is used as the initial model. The initial average test accuracy of the pre-trained network on CIFAR10 is 27.55%.”

Have you followed this setting when you reproduce the results?

[973891422]

Hi,

As I mentioned in the experimental parts of this paper, a ResNet18 pre-trained on ImageNet is used as the initial model. “For CIFAR10, ResNet18 pre-trained on ImageNet (Krizhevsky, Sutskever, and Hinton 2017) is used as the initial model. The initial average test accuracy of the pre-trained network on CIFAR10 is 27.55%.”

Have you followed this setting when you reproduce the results?

Hi, I followed the settings in your code

            resnet = resnet18(args, pretrained=False, num_classes=args.num_classes)
            initial_weight = model_zoo.load_url(model_urls['resnet18'])
            local_model = resnet
            initial_weight_1 = local_model.state_dict()
            for key in initial_weight.keys():
                if key[0:3] == 'fc.' or key[0:5] == 'conv1' or key[
                        0:3] == 'bn1':
                    initial_weight[key] = initial_weight_1[key]

            local_model.load_state_dict(initial_weight)

But the result is still not good, I don't know where the problem is
Your work is outstanding and I look forward to your guidance

[yuetan031]

Hi,
As I mentioned in the experimental parts of this paper, a ResNet18 pre-trained on ImageNet is used as the initial model. “For CIFAR10, ResNet18 pre-trained on ImageNet (Krizhevsky, Sutskever, and Hinton 2017) is used as the initial model. The initial average test accuracy of the pre-trained network on CIFAR10 is 27.55%.”
Have you followed this setting when you reproduce the results?

Hi, I followed the settings in your code

            resnet = resnet18(args, pretrained=False, num_classes=args.num_classes)
            initial_weight = model_zoo.load_url(model_urls['resnet18'])
            local_model = resnet
            initial_weight_1 = local_model.state_dict()
            for key in initial_weight.keys():
                if key[0:3] == 'fc.' or key[0:5] == 'conv1' or key[
                        0:3] == 'bn1':
                    initial_weight[key] = initial_weight_1[key]

            local_model.load_state_dict(initial_weight)

But the result is still not good, I don't know where the problem is Your work is outstanding and I look forward to your guidance

Thanks for your attention.
Have you checked the initial average test accuracy before training? If it is not around 27%, please double check the data splitting manner.

[973891422]

Hi,
As I mentioned in the experimental parts of this paper, a ResNet18 pre-trained on ImageNet is used as the initial model. “For CIFAR10, ResNet18 pre-trained on ImageNet (Krizhevsky, Sutskever, and Hinton 2017) is used as the initial model. The initial average test accuracy of the pre-trained network on CIFAR10 is 27.55%.”
Have you followed this setting when you reproduce the results?

Hi, I followed the settings in your code

            resnet = resnet18(args, pretrained=False, num_classes=args.num_classes)
            initial_weight = model_zoo.load_url(model_urls['resnet18'])
            local_model = resnet
            initial_weight_1 = local_model.state_dict()
            for key in initial_weight.keys():
                if key[0:3] == 'fc.' or key[0:5] == 'conv1' or key[
                        0:3] == 'bn1':
                    initial_weight[key] = initial_weight_1[key]

            local_model.load_state_dict(initial_weight)

But the result is still not good, I don't know where the problem is Your work is outstanding and I look forward to your guidance

Thanks for your attention. Have you checked the initial average test accuracy before training? If it is not around 27%, please double check the data splitting manner.

Thanks for your reply, I will try it
I have another question. When resnet18 is migrated to cifra10, the final fc layer is modified to output 10 dimensions. Can the random initialization of the fc layer parameters achieve the accuracy of about 27%?

[973891422]

I ran it, resnet18 pre-trained model in cifar10 initial test accuracy rate is only about 5%, completely using the data division and pre-trained model loading method provided in your code.
I can't understand where the problem is.
Can you give me some help? Thanks

[yuetan031]

It seems weird because other people have successfully reproduced the results reported in the paper. You should make sure the hyper parameters and the model architectures are the same with the paper. Furthermore, the output dimension should be set to 10 because there are 10 classes in CIFAR10 dataset.

[eepLearning]

Hi, your excellent work gives us several things to think about.

I am having the same problem that test accuracy is not restored about CIFAR-10.
And this gives me some confusion.

One thing I'm confused about is that the code outputs the test accuracy for two cases. Which of the two is the accuracy in the performance table of the paper? ("without proto"/ "with proto")

I cloned your code completely as-is. Therefore, there is no possibility that the code I run is different from the settings in the repository.

"data splitting manner"

Could it be that the problem you mentioned above can occur even though I haven't made any modifications at all?

Also, when looking at the output training log, there seems to be no problem in data splitting.

[yuetan031]

Hi,

Towards the first question, have you check whether the initial test accuracy on CIFAR10 is about 27%? Pre-trained model weight should be downloaded and loaded appropriately before federated training.

Towards the second question, the test accuracy can be obtained in two manner, we choose to inference from the prototypes, so "with proto" is the one reported in this paper.

[yuetan031]

Hi, your excellent work gives us several things to think about.

I am having the same problem that test accuracy is not restored about CIFAR-10. And this gives me some confusion.

One thing I'm confused about is that the code outputs the test accuracy for two cases. Which of the two is the accuracy in the performance table of the paper? ("without proto"/ "with proto")

I cloned your code completely as-is. Therefore, there is no possibility that the code I run is different from the settings in the repository.

"data splitting manner"

Could it be that the problem you mentioned above can occur even though I haven't made any modifications at all?

Also, when looking at the output training log, there seems to be no problem in data splitting.

I have rerun the code of this project and it seems that there exists the issue reported by you. Could you please increase the batch size from 4 to 32 to see if the initial accuracy is around 27%?

[eepLearning]

Thank you so much for the quick and kind reply.
As you said, changing the batch_size to 32 miraculously doubled the performance.

Final accuracy increased from 30% to 60%
Initial accuracy increased from 5% to 12%.

First of all, I am amazed that batch_size can have such a big effect. Is this a data split problem or does it appear to be a training problem in the model network?

The initial accuracy was performed with the global proto obtained in the first round.
(Is the initial accuracy you mentioned the inference performance without the proto of the pretrain network?)

Thanks again for your reply.

[eepLearning]

I add the simple code for check the initial accuracy in the following way

    train_loss, train_accuracy = [], []

    for round in tqdm(range(args.rounds)):
        local_weights, local_losses, local_protos = [], [], {}
        print(f'\n | Global Training Round : {round + 1} |\n')

        proto_loss = 0
        for idx in idxs_users:
            local_model = LocalUpdate(args=args, dataset=train_dataset, idxs=user_groups[idx])
            w, loss, acc, protos = local_model.update_weights_het(args, idx, global_protos, model=copy.deepcopy(local_model_list[idx]), global_round=round)
            agg_protos = agg_func(protos)

            local_weights.append(copy.deepcopy(w))
            local_losses.append(copy.deepcopy(loss['total']))

            local_protos[idx] = agg_protos
            summary_writer.add_scalar('Train/Loss/user' + str(idx + 1), loss['total'], round)
            summary_writer.add_scalar('Train/Loss1/user' + str(idx + 1), loss['1'], round)
            summary_writer.add_scalar('Train/Loss2/user' + str(idx + 1), loss['2'], round)
            summary_writer.add_scalar('Train/Acc/user' + str(idx + 1), acc, round)
            proto_loss += loss['2']

        # update global weights
        local_weights_list = local_weights
        global_protos = proto_aggregation(local_protos)
        if round == 0:
           print("Initial Accuracy")
            acc_list_l, acc_list_g = test_inference_new_het_lt(args, local_model_list, test_dataset, classes_list,
                                                               user_groups_lt, global_protos)
            print('For all users (with protos), mean of test acc is {:.5f}, std of test acc is {:.5f}'.format(
                np.mean(acc_list_g), np.std(acc_list_g)))
            print('For all users (w/o protos), mean of test acc is {:.5f}, std of test acc is {:.5f}'.format(
                np.mean(acc_list_l), np.std(acc_list_l)))

        for idx in idxs_users:
            local_model = copy.deepcopy(local_model_list[idx])
            local_model.load_state_dict(local_weights_list[idx], strict=True)
            local_model_list[idx] = local_model
           
        loss_avg = sum(local_losses) / len(local_losses)
        train_loss.append(loss_avg)

[yuetan031]

Thank you so much for the quick and kind reply. As you said, changing the batch_size to 32 miraculously doubled the performance.

Final accuracy increased from 30% to 60% Initial accuracy increased from 5% to 12%.

First of all, I am amazed that batch_size can have such a big effect. Is this a data split problem or does it appear to be a training problem in the model network?

The initial accuracy was performed with the global proto obtained in the first round. (Is the initial accuracy you mentioned the inference performance without the proto of the pretrain network?)

Thanks again for your reply.

I think the batch_size parameter has such an effect is due to the fact that the local dataset in each client is relatively small (only n classes, k samples/class). But how it actually affects the model may deserve more exploration in the future.

Besides, thanks for your modification on the code. I will follow your suggestion and update this project:)

[TsingZ0]

I guess the reason may be that the Batch Normalization layer in the ResNet-18 requires a large batch_size to approximate the statistics (\mu and \sigma) accurately.

[TsingZ0]

Towards the second question, the test accuracy can be obtained in two manner, we choose to inference from the prototypes, so "with proto" is the one reported in this paper.

I cannot find this test metric in the paper. Did I miss something?

[yuetan031]

Have you checked the "test_inference_new_het_lt" function in ./lib/update.py? It returns acc_list_g and acc_list_l consisting of the accuracy obtained with and without prototypes, which corresponds to the two manners mentioned above.