prune pointpillar with mmdetection3d

[qingzhouzhen]

I want to prune pointpillar on mmdetection3d framework, but the model has more than one inputdef forward(self, img, img_metas, return_loss=True, **kwargs):, which makes it not suitable for speedup: ModelSpeedup(model, dummy_input=torch.rand([10, 3, 32, 32]).to(device), masks_file=masks).speedup_model()
so I just prune and speedup the backbone: model.backbone follow steps:
step1 : load origin model
step2: prune model, speedup model, save the speeduped model
step3 : load speeduped model and finetune it
the reason I save model in step2 is I can not finetune it in mmdetection3d immediately if I prune the model in mmdetection3d.
the sparse ratio is 0.9(cut 10% weight with L1Norm), accuracy drops about 2%
question is, am I right?
ps: finetune loss

[Louis-J]

for many mmdetection3d models, there is a forward_dummy function in the model, so you can try to prune it by:

    forward_orig = model.forward
    model.forward = model.forward_dummy
    # prune ... complete
    model.forward = forward_orig

or you can write a custom forward if the forward_dummy cannot be used.

[qingzhouzhen]

for many mmdetection3d models, there is a forward_dummy function in the model, so you can try to prune it by:

    forward_orig = model.forward
    model.forward = model.forward_dummy
    # prune ... complete
    model.forward = forward_orig

or you can write a custom forward if the forward_dummy cannot be used.

I follw your suggestion

    forward_orig = model.forward
    model.forward = model.forward_dummy
    # pruner
    config_list = [{
        'sparsity_per_layer': 0.1,
        'op_types': ['Linear', 'Conv2d']
      },
      {
        'exclude': True,
        'op_names': ['blocks.0.9', 'blocks.1.15', 'blocks.2.15']
      }]
    from nni.compression.pytorch.pruning import L1NormPruner
    pruner = L1NormPruner(model, config_list)
    _, masks = pruner.compress()
    for name, mask in masks.items():
        print(name, ' sparsity : ', '{:.2}'.format(mask['weight'].sum() / mask['weight'].numel()))
    pruner._unwrap_model()
        # speedup the model
    from nni.compression.pytorch.speedup import ModelSpeedup
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    model.cuda()
    for key in masks.keys():
        masks[key]['weight'] = masks[key]['weight'].cuda()
    print("begin speedup")
    ModelSpeedup(model, torch.rand(4,123926, 4).cuda(), masks).speedup_model()
    print("after speedup: ", model.backbone)
    model.forward = forward_orig
    model.cuda()

but I got an error in speedup ModelSpeedup(model, torch.rand(4,123926, 4).cuda(), masks).speedup_model():

in the future. This means that the trace might not generalize to other inputs!
  batch_size = coors[-1, 0].item() + 1
Traceback (most recent call last):
  File "tools/pruner_train_dummpy2.py", line 296, in <module>
    main()
  File "tools/pruner_train_dummpy2.py", line 255, in main
    forward_orig = model.forward
  File "/opt/conda/lib/python3.7/site-packages/nni/compression/pytorch/speedup/compressor.py", line 73, in __init__
    self.torch_graph = build_module_graph(model, self.dummy_input)
  File "/opt/conda/lib/python3.7/site-packages/nni/common/graph_utils.py", line 25, in build_module_graph
    return TorchModuleGraph(model, dummy_input)
  File "/opt/conda/lib/python3.7/site-packages/nni/common/graph_utils.py", line 265, in __init__
    super().__init__(model, dummy_input, traced_model)
  File "/opt/conda/lib/python3.7/site-packages/nni/common/graph_utils.py", line 67, in __init__
    self._trace(model, dummy_input)
  File "/opt/conda/lib/python3.7/site-packages/nni/common/graph_utils.py", line 91, in _trace
    self.trace = torch.jit.trace(model, dummy_input, **kw_args)
  File "/opt/conda/lib/python3.7/site-packages/torch/jit/__init__.py", line 955, in trace
    check_tolerance, strict, _force_outplace, _module_class)
  File "/opt/conda/lib/python3.7/site-packages/torch/jit/__init__.py", line 1109, in trace_module
    module._c._create_method_from_trace(method_name, func, example_inputs, var_lookup_fn, strict, _force_outplace)
RuntimeError: 0 INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1595629403081/work/torch/csrc/jit/ir/alias_analysis.cpp":465, please report a bug to PyTorch. We don't have an op for aten::to but it isn't a special case.  Argument types: Tensor, None, int, Device, bool, bool, bool, None,

it seems op not support?

[Louis-J]

It seems that this model cannot be traced by torch.jit.trace to get a torchscript model. We can only do speedup on torchscript models.
there is a page https://mmclassification.readthedocs.io/en/latest/tools/pytorch2torchscript.html but I don't know whether it works.
In this case only, you can try another pytorch version or another python version. It may help.