Fisheye grads not working for camera pose optimization

Hi,
I am trying cam pose optimization within existing map, when I use fisheye camera model the grads are not working but the pinhole seems to work ok. 
My fisheye is very wide FOV (195deg) so I would like to be able to use fisheye model. 