Can't Free GPU Memory in SGLang?

[jamesdborin]

Hi, I have been working on use cases where I might want to delete weights or move them to the CPU after they are initialized (use cases like pruning or offloading for example).

However I find that I cannot do this as I expect.

For example if I delete all of the expert weights from a Qwen3-30B-A3B with a snippet like this in load_weights in loader.py:

def load_weights(...):
    ...
    self.load_weights_and_postprocess(...)
    
    with torch.no_grad():
        for idx, layer in enumerate(model.model.layers):
            del layer.mlp.experts.w13_weight
            del layer.mlp.experts.w2_weight

            torch.cuda.synchronize()
            import gc; gc.collect()
            torch.cuda.empty_cache()

            alloc = torch.cuda.memory_allocated() / (1024**3)
            resv  = torch.cuda.memory_reserved() / (1024**3)
            logger.info(f"after layer {idx} allocated={alloc:.4f} GB, reserved={resv:.4f} GB")

    return model.eval()

I find that the available memory doesn't change at all:

[2025-12-31 16:52:45] after layer 0 allocated=5.8228 GB, reserved=5.8242 GB
[2025-12-31 16:52:45] after layer 1 allocated=5.8228 GB, reserved=5.8242 GB
[2025-12-31 16:52:46] after layer 2 allocated=5.8228 GB, reserved=5.8242 GB
...

Usually this is because there is another reference to the tensor somewhere, but I can't find where that reference might be.

Any suggestions would be really appreciated!