Implementation of closed-form solution

[YuyangXueEd]

Hi Shilin-Lu,

Thank you very much for your work and MACE is brilliant work!

I have a question when going through your code. In eq. 2, you use W_k to multiply the first term, but in your implementation here, you only add W_k at first (mat1), but not use matrix multiplication. I wonder whether I have some misunderstanding here or the W_k is ignored.

Looking forward to your reply. Thank you very much!

Best!

[Shilin-LU]

Hi, thank you so much for your kind words.

The value already represents the result of W multiplied by the text embedding, so there’s no need to apply W again.

[YuyangXueEd]

Hello,

Thanks for your quick reply!

I missed the layer in values.append(layer(old_emb).detach()), thanks for pointing out!

I will close the issue.

Best!

[YuyangXueEd]

Hello again,

May I ask why do you add or initialise mat1 and mat2 for closed_form_refinement function? From the formula 2 I don't see any other items instead of the sum, which is in the for loop for contexts and values.

mat1 = lamb * projection_matrices[layer_num].weight   # W_k
mat2 = lamb * torch.eye(projection_matrices[layer_num].weight.shape[1],  # I
device=projection_matrices[layer_num].weight.device)

...

total_for_mat1 += mat1
total_for_mat2 += mat2

Looking forward to your reply, thank you very much!

[Shilin-LU]

Apologies for the delayed response, it's been quite busy lately.

Typically, the lambda (lamb) parameter in the configuration is set to 0, which means mat1 and mat2 can be disregarded. However, if lamb is not equal to 0, this scenario is equivalent to adding an additional regularization term ||W_k' - W_k|| to Equation (1).