Assignment3 advice: Scale the preactivation of the output layer!

[el-hult]

Hi!

In the assignment, we have this given.

In the article, they use

so they do not agree on the formula for the output layer (called \alpha^{L+1} in the paper and f_\theta in the assignment). I think one must scale the output layyer by 1/\sqrt(n_L) and multiply the bias by \beta. In the reference code of Antonio, this is how it is implemented. When I implemented myself without that scaling, the learning rate of 1 was too large, and the training did not converge.

So hopefully you don't have to fall into that trap. :)

[antonior92]

Nice point. You are right!

In part 1 it should not make a difference. But in part 2 you should indeed use the correct scaling.
Thanks for spotting this inconsistency

[el-hult]

yeah. I shot myself in the foot by naïvely putting in the scaling for part 2 also in the code for part 1. then it broke again. 😅

You have to think twice if you share code between part 1 and part 2.