SimulatedAnnealingPruner prune iteration issue

[sungh66]

Describe the issue:
When i use the SimulatedAnnealingPruner, it has iteration of temperature, but i found that its evaluation results is always the same every times, is there anything wrong with the evaluate function?
PS: I checked the Simulatedpruner source code, and found that it must have comparation with the acc every iteration in order to find which search space is better, do I understand this correctly? So i have the doubt.

[J-shang]

Hi @sungh66 , yes, you are right, it needs to compare the acc to find the best one. Can you show your evaluation results and log? Let's find what makes wrong.

[sungh66]

I run your base cifar10 and it has the same problem.During pruning, the cifar10 evaluate acc result is always 10%, so it may not be my code's fault?
My code may be a little complicated, but the model is simple, is the resnet50

def test(model, test_set):
    model.eval()
    fe = DeepMAR_ResNet50_ExtractFeature(model=model)
    print('att test with feat_func_att')
    result = attribute_evaluate(fe, test_set, **{})
    print( '-' * 60)
    print( 'Evaluation on %s set:' % (args.test_split))
    ma = np.mean(result['label_acc'])
    print( 'Label-based evaluation: \n mA: %.4f'%(ma))
    print( 'Instance-based evaluation: \n Acc: %.4f, Prec: %.4f, Rec: %.4f, F1: %.4f' \
           %(result['instance_acc'], result['instance_precision'], result['instance_recall'], result['instance_F1']))
    print( '-' * 60)
    return ma

class DeepMAR_ResNet50_ExtractFeature(object):
    """
    A feature extraction function
    """
    def __init__(self, model, **kwargs):
        self.model = model
    def __call__(self, imgs):
        old_train_eval_model = self.model.training
        # set the model to be eval
        self.model.eval()
        # imgs should be Variable
        if not isinstance(imgs, Variable):
            print('imgs should be type: Variable')
            raise ValueError
        score = self.model(imgs)
        score = score.data.cpu().numpy()
        self.model.train(old_train_eval_model)
        return score

def attribute_evaluate(feat_func, dataset, **kwargs):
    print("extracting features for attribute recognition")
    pt_result = extract_feat(feat_func, dataset)

    # obain the attributes from the attribute dictionary
    print("computing attribute recognition result")
    N = pt_result.shape[0]
    L = pt_result.shape[1]
    gt_result = np.zeros(pt_result.shape)
    # get the groundtruth attributes
    for idx, label in enumerate(dataset.label):
        gt_result[idx, :] = label
    pt_result[pt_result>=0] = 1
    pt_result[pt_result<0] = 0
    return attribute_evaluate_lidw(gt_result, pt_result)

def extract_feat(feat_func, dataset, **kwargs):
    """
    extract feature for images
    """
    test_loader = torch.utils.data.DataLoader(
        dataset = dataset, batch_size = 32,
        num_workers = 2, pin_memory = True)
    # extract feature for all the images of test/val identities
    start_time = time.time()
    total_eps = len(test_loader)
    N = len(dataset.image)
    start = 0
    with torch.no_grad():
        for ep, (imgs, labels) in enumerate(test_loader):
            imgs_var = imgs.cuda()
            feat_tmp = feat_func( imgs_var )
            batch_size = feat_tmp.shape[0]
            # print(feat_tmp.size)
            # print(feat_tmp.size/batch_size)
            # print(N.type())
            temp = feat_tmp.size/batch_size
            if ep == 0:
                feat = np.zeros((N, int(temp)))
            feat[start:start+batch_size, :] = feat_tmp.reshape((batch_size, -1))
            start += batch_size
    end_time = time.time()
    print('{} batches done, total {:.2f}s'.format(total_eps, end_time-start_time))
    return feat

def attribute_evaluate_lidw(gt_result, pt_result):
    """
    Input:
    gt_result, pt_result, N*L, with 0/1
    Output:
    result
    a dictionary, including label-based and instance-based evaluation
    label-based: label_pos_acc, label_neg_acc, label_acc
    instance-based: instance_acc, instance_precision, instance_recall, instance_F1
    """
    # obtain the label-based and instance-based accuracy
    # compute the label-based accuracy
    if gt_result.shape != pt_result.shape:
        print('Shape beteen groundtruth and predicted results are different')
    # compute the label-based accuracy
    result = {}
    gt_pos = np.sum((gt_result == 1).astype(float), axis=0)
    gt_neg = np.sum((gt_result == 0).astype(float), axis=0)
    pt_pos = np.sum((gt_result == 1).astype(float) * (pt_result == 1).astype(float), axis=0)
    pt_neg = np.sum((gt_result == 0).astype(float) * (pt_result == 0).astype(float), axis=0)
#    print(gt_pos,gt_neg,len(gt_pos),len(gt_neg))
#    print(pt_pos,pt_neg,len(pt_pos),len(pt_neg))
    label_pos_acc = 1.0*pt_pos/gt_pos
    label_neg_acc = 1.0*pt_neg/gt_neg
    label_acc = (label_pos_acc + label_neg_acc)/2
#    print(label_pos_acc,label_neg_acc)
    result['label_pos_acc'] = label_pos_acc
    result['label_neg_acc'] = label_neg_acc
    result['label_acc'] = label_acc
    # compute the instance-based accuracy
    # precision
    gt_pos = np.sum((gt_result == 1).astype(float), axis=1)
    print('gtpos.ptpos',gt_pos,pt_pos)
    pt_pos = np.sum((pt_result == 1).astype(float), axis=1)
    floatersect_pos = np.sum((gt_result == 1).astype(float)*(pt_result == 1).astype(float), axis=1)
    union_pos = np.sum(((gt_result == 1)+(pt_result == 1)).astype(float),axis=1)
    # avoid empty label in predicted results
    cnt_eff = float(gt_result.shape[0])
    for iter, key in enumerate(gt_pos):
        if key == 0:
            union_pos[iter] = 1
            pt_pos[iter] = 1
            gt_pos[iter] = 1
            cnt_eff = cnt_eff - 1
            continue
        if pt_pos[iter] == 0:
            pt_pos[iter] = 1
    instance_acc = np.sum(floatersect_pos/union_pos)/cnt_eff
    instance_precision = np.sum(floatersect_pos/pt_pos)/cnt_eff
    instance_recall = np.sum(floatersect_pos/gt_pos)/cnt_eff
    floatance_F1 = 2*instance_precision*instance_recall/(instance_precision+instance_recall)
    result['instance_acc'] = instance_acc
    result['instance_precision'] = instance_precision
    result['instance_recall'] = instance_recall
    result['instance_F1'] = floatance_F1
    return result

[J-shang]

Got it, do you pre-train your model before pruning? and what's the sparsity? In the paper, the sparsity increase very small during each iteration. So if you use SA Pruner independence, please set a small sparsity. In the paper, the sparsity is 0.02 in each iteration. If you set a big sparsity, the evaluation result will approximately random, because too many weight are pruned.
FYI, https://arxiv.org/abs/1907.03141

[sungh66]

Got it, do you pre-train your model before pruning? and what's the sparsity? In the paper, the sparsity increase very small during each iteration. So if you use SA Pruner independence, please set a small sparsity. In the paper, the sparsity is 0.02 in each iteration. If you set a big sparsity, the evaluation result will approximately random, because too many weight are pruned. FYI, https://arxiv.org/abs/1907.03141

Thanks, though i haven't fix the problem, the finetune result is god and i had pretrain my model.

[sungh66]

Got it, do you pre-train your model before pruning? and what's the sparsity? In the paper, the sparsity increase very small during each iteration. So if you use SA Pruner independence, please set a small sparsity. In the paper, the sparsity is 0.02 in each iteration. If you set a big sparsity, the evaluation result will approximately random, because too many weight are pruned. FYI, https://arxiv.org/abs/1907.03141

I had pretrained my model, and the same pruned result exactly because of the sparsity is too big as you said.So i set the sparsity to 0.1.Besides, do i need to finetune the model every pruning iteration?How many finetune epochs per pruning iteration is more appropriate compared to one-time finetuneing？

[J-shang]

I had pretrained my model, and the same pruned result exactly because of the sparsity is too big as you said.So i set the sparsity to 0.1.Besides, do i need to finetune the model every pruning iteration?How many finetune epochs per pruning iteration is more appropriate compared to one-time finetuneing？

Yes, the model should speed up and finetune before the next iteration. The epoch number is hard to recommend, finetune to stable is ok.

And you can have a try autocompress pruner. https://nni.readthedocs.io/en/stable/Compression/Pruner.html#autocompress-pruner

[typhoonlee]

Describe the issue: When i use the SimulatedAnnealingPruner, it has iteration of temperature, but i found that its evaluation results is always the same every times, is there anything wrong with the evaluate function? PS: I checked the Simulatedpruner source code, and found that it must have comparation with the acc every iteration in order to find which search space is better, do I understand this correctly? So i have the doubt.

hi, i use AGPPruner now, but how to apply pruning in the process of finetune Where should this API be placed in the code?