Hyperparameter optimization

[danieleewww]

Hello Mathis:

I ran through your solution using Intel® Core™ i5 CPU 760 @ 2.80GHz × 4 and GeForce GTX 970/PCIe/SSE2 under Ubuntu 14.04 LTS in about 18days, got kappa value shown in the attached which is similar as you did in the competition. I am planning either to go further with hyperparameter optimization or adjusting network depth or width. Any suggestion on this?

Cheers,

Daniel

[sveitser]

I would probably try the following things, but I don't think there is much more to be gained this way either.

• Make the networks deeper where possible.
• Add another convnet to the ensemble.
• Try using different rectifier units. Notably the elu unit that is now available in lasagne sounds promising.
• You might be able to boost the score slightly if you try to compute optimal thresholds at the end rather than using 0.5, 1.5, 2.5, 3.5.
• I think you can at least save some time during training if you remove the fully connected layers and/or add 'nonlinearity': None to the parameters for the output layer. The default linearity is rectify so the minimum the convnets can output at the moment is 0. You might have to tune the learning rate schedule again for this to work well though.