FFT version of Celeste misestimates brightness

[jeff-regier]

With #505, the Celeste FFT point estimates look very good except for on mag_r, the brightness in the r band. (The number of galaxies misclassified as stars increases too, but primary makes most of these mistakes too, so it's not necessarily a bad sign.)

[jeff-regier]

Setting max_iters=200 seems to help with the simple_star galsim test case, but not with validate_on_stripe82.jl --fft. Not that I'm working on this issue, just figured I'd mention it.

[rgiordan]

Looking at the galsim cases today, it appears that the FFT version is not finding a good optimum.

It would be useful if the omitted_ids functionality were restored (#480).

[rgiordan]

With the mysterious fix mentioned in #534, the FFT is now doing pretty well on GalSim again. I'm running stripe 82 and will check it tomorrow.

So far, I've seen big gains with the FFT by optimizing for a star first and then the star and galaxy parameters together -- sometimes 2-3x speedups in numbers of iterations, comparable ELBO values, and occasionally fixing misclassifications. These gains could be even greater if the ELBO were written in a way that for active sources you skip the galaxy computations entirely, because then all those iterations would be extremely fast.

Optimizing in two steps for the MOG doesn't help, though. My guess is the gains with the FFT may be due to the problem #463 is trying to solve.

[jeff-regier]

What's happening here? This object is very bright -- 16 star magnitude is about right. It's a galaxy but it has a small radius. Therefore star_mag_r and gal_mag_r should be about the same. Yet gal_mag_r is 22.6--off by a lot.

│ Row │ objid                 │ star_mag_r │ gal_mag_r │ ra       │ dec      │ gal_scale │ gal_ab   │
├─────┼───────────────────────┼────────────┼───────────┼──────────┼──────────┼───────────┼──────────┤
│ 1   │ "1237663784734490980" │ 16.1016    │ 22.6502   │ 0.525752 │ 0.503731 │ 0.127751  │ 0.784822 │

Seems like it must be a bug. Hard to imagine that it's a local minimum---any increase in galaxy brightness should improve the likelihood term by a lot.

[jeff-regier]

Looks like a pretty typical bright light source.

http://legacysurvey.org/viewer?ra=0.5257&dec=0.5037&zoom=16&layer=sdssco

[jeff-regier]

SDSS has it as a star. Coadd has it as a galaxy. DeCALS has it is multiple light sources.

http://skyserver.sdss.org/dr10/en/tools/explore/summary.aspx?id=0x112d10a7a0770164&spec=&apid=

Kind of interesting, but regardless, seems like 22.6 gal mag can only be because of a bug.

[rgiordan]

Yes, I agree it's doing something strange. If you increase the function convergence tolerance, the FFT code drives the galaxy brightness even further to zero. Let's keep this issue open.

Could you explain again why you think you need to use the FFT version for GB?

[jeff-regier]

Long term everyone thinks FFT is the way to go. We want to spend our time between now and GB optimizing code that we can use for the long term---not optimizing the MOG code. Secondarily, I'm hopeful that the FFT will improve how well we discriminate btw stars and galaxies once the bugs are fixed.

[rgiordan]

On this source, at least, changing the kernel parameter to the (more principled) -0.5, together with setting f_tol_rel to 1e-10 and increasing max_iters to 200, fixes this problem. (The advantage of having the kernel parameter at 0.0 was that it was strictly positive, the advantage of -0.5 is that is matches to higher order.)

Looking at an animation of the fitting process, it's pretty clear that it is starting from a dim galaxy and takes a while to get to the correct number. It appears that, in this case, the starting parameters from the catalog are quite dim. When the object is so likely a star, the galaxy improvements don't change the ELBO much so based on the ELBO values it terminates optimization early. (I'll check in my animation code you so you can do this kind of debugging, too.)

I'm not sure how to best solve this kind of problem at scale. I remain an advocate for making the objective separable (with a Normal model and variance estimated separately from the mean), and treating the star and galaxy optimization as separate problems.

[jeff-regier]

If the problem is that the probability of "galaxy" is too low for it to make progress, would it work to fix vs[ids.a] = 0.5 for the first 10 iterations? It's surprising though that that's necessary since we already don't let the galaxy probability fall below 0.5%. (0.5% of a lot is still a big number compared to machine precision). Is the change in the ELBO for when the brightness parameters is set to 16 magnitude rather than 22.6 magnitude really infinitesimally small? I wouldn't expect it would be so small that f_tol would need to be so low, well, unless f is really large in magnitude. Is setting f_tol_abs as well an option?

[rgiordan]

Recall that as soon as one TR step is below the threshold, the algorithm terminates. At the beginning, the Newton steps don't appear to be principally in the brightness direction. In any case, this is what is happening with this source.

Optim.jl only supports relative tolerance for functions last I looked.

I'll let you experiment with tweaking the optimization if you like. It is easy to play with now by modifying the behavior of maximize_f_two_steps. My preference is to fix it in the model. I'll have to do the Gaussian model to work with DECam data anyway, so I'll be probably doing that at some point soonish.