is there a maximum sparsity for a specific shape?

[yyfcc17]

for a shape of 768 x 1280 and 69 frames, I tuned some values of the decay factor. but it seems there is a maximum sparsity regardless of the decay factor?

in the case of 768 x 1280 and 69 frames, the maximum sparsity showed is

mask sparsity: 0.7366323471069336

after a specifc value of decay factor, the sparsity can not be increased no matter how small i tune the decay factor to.

[Radioheading]

I think there is, because for every decay factor, the attention between frame $i$ and frame $j$ would be full attention if $|i-j|\le 1$. But you can manually change this part of the code if you want higher sparsity. I recommend that you also apply decay on parts where $ |i-j|=1 $, and keep the main diagonal in full attention.

[yyfcc17]

thanks for your reply.

well, my purpose is to integrate radial attention into training or distillation stage. so i think sparsity should be much higher, e.x. 90% or 95% in some papers.

i've followed your advice, modified get_window_width function for wan model type:

def get_window_width(i, j, token_per_frame, sparse_type, num_frame, decay_factor=1, block_size=128, model_type=None):
    assert(sparse_type in ["radial"])
    dist = abs(i - j)
    if model_type == "wan":
        # if dist < 1:
        #     return token_per_frame
        # if dist == 1:
        #     return token_per_frame // 2
        pass
    elif model_type == "hunyuan":
        if dist <= 1:
            return token_per_frame
    else:
        raise ValueError(f"Unknown model type: {model_type}")
    group = dist.bit_length()
    decay_length = 2 ** token_per_frame.bit_length() / 2 ** group * decay_factor
    threshold = block_size
    if decay_length >= threshold:
        return decay_length
    else:
        return threshold

and i even set decay factor to 0, but the sparsity is still:
mask sparsity: 0.8520987629890442

how could i increase sparsity to 95%? do you have any advices? thank you.

[Radioheading]

I think you can also remove the attention sink if you really want super high sparsity lol. Currently, I think the rest density comes from the attention sink or some corner cases. You can plot the attention map to see the visualization.

[yyfcc17]

i managed to increase sparsity to 90% with 720P, but now i realised a question:

if i train the model under 90% sparsity with 480P, the model will work with 480P generation, but will it also work with 720P or 1080P? because the mask will change when it switch to other resolutions.