Sampler stalls during run

[dadams42]

** Dynesty version **
2.1.5

Describe the bug
The sampler gets stalled, the run proceeds without giving errors but progress is very slow. This did not happen at earlier stages of the run or with other comparable runs before.

Setup
I attach (in this order) a screenshot of the run for initialising the sampler and running it (done in a computing cluster) and also a version I ran in my laptop which yields similar results. The sampler has 34 dimensions and 12,000 live points. While this number is high, I found shorter runs of ~8,000 live points or less, despite converging well without the issue I'm reporting, did not fully sample the posterior and got stuck in certain 'modes', justifying a higher nlive parameter.

intialising:

running:

running on laptop:

** Dynesty output **
I attach the progress bar when running in my laptop, and the 'standard diagnostic test' featuring the likelihood, importance weight, and evidence:

Bug
The sampler makes very little progress at the current stage. Intially it reached a size of 1.2 GB over ~ 2 weeks when run in the cluster, now it does ~ 1 MB / day. Note that the 8,000 live point runs did not have this issue, so this is unlikely to be a simple case of more live points -> longer running time. I have run many similar runs before, the only difference here is the high (12,000) number of live points, so I suspect the issue is somehow connected to this.

Additional context
Would like to know if this is a known bug that was fixed in the 3.0.0 version and if not what temporary solution is recommended (e.g. running smaller nlive runs and merging together etc.). I'm also using the Dynamic Nested sampler (I'm interested in the posterior), but if by any chance the Nested sampler is better tested for this high-dimensional, nlive cases that would be good to know.

[segasai]

Hi,

Thanks for the report.
It is hard to say based on this information alone, whether there is a bug or not.

First some remarks:

1. 3.0 does not have any bug-fixes per-se ( 2.1.5 version does not have recorded bugs).
2. Given that the numbers you provided it seems most likely that the rsclice sampler is used. It is very stable, but if you have a pathological posterior shape, it may be slow.
3. The status line you show seems to reflect the beginning of the resumed run, not the actual slow part, so it hard to know what is going on. One needs to see the numbers of it/s, and the numbers like nc (number of function calls). As 'being stuck somewhere' would show very large 'nc'
4. 12000 live points is a lot and if you are saying that the posterior is poorly sampled with smaller number of points, I'd be concerned about posterior shape/potential issues with it.

Questions:

• Do you only see the issue when resuming the run or you can see it when running from scratch ?
• How long does your function takes to evaluate ? Are you certain that it is not your likelihood function that is slow for some values of parameters ? I often debug this by printing from likelihood the function values and arguments -- this way you get an idea what the sampling is doing.
• Is your likelihood function fully deterministic ?

[joshspeagle]

Echoing @segasai here on all points. And to follow up: How many dimensions are you trying to probe here? From the screenshot, it looks like the likelihood is evaluated almost instantaneously, given the huge number of iterations occurring per second, and the sampling efficiency of ~0.5% is hard to interpret without more information (this could be very concerning if the problem is simple and low-D, or totally expected if there's a huge number of dimensions, etc.).

[dadams42]

Thanks @segasai and @joshspeagle for the replies. I'm attaching two snaphsots of the progress bar, both at the beginning and at the end of a run of 12 hours in the computing cluster, in case this is more informative.

beginning:

end:

The likelihood is deterministic and to my knowledge doesn't have any pathologies, but it is pretty complex, since it consists of a dynamical model fitting some ~ 60,000 PDFs of stellar velocities (with 34 free parameters / dimensions). There are some 'physicality priors' which return -np.inf in the log likelihood if the model has negative velocity moments (this can happen from the Jeans equations), but I haven't had issues with that before. And the posterior doesn't seem odd or anything, it's just that sometimes for lower live points there are values at the tails of the distribution that can produce similar likelihoods to the maximum likelihood model and when increasing nlive the model shows a broader range and those good-fitting models no longer look like outliers.

The issue only appeared at a relatively advanced stage of the run and was not a problem when running from scratch and the multiple resumed runs afterwards before it did start to give problems (in the resumed runs).

I don't think that the likelihood should be much affected by the parameters in evaluation time. The posterior is consistent with previous runs that had worked before (i.e. exploring similar parameters).

Edit: I've updated snapshots to include a later run that was more inefficient

[segasai]

Thanks @dadams42 !

-- seeing the screenshots, it seems to me that you have a function that takes much longer to evaluate at that point because you can see roughly constant nc\sim 200 per iteration, while 1 iteration takes 1 minute. I don't believe there is any explanation, other than the likelihood taking longer to evaluate. I'd suggest checking that (maybe print the time per call as a test).

[dadams42]

Thanks @segasai. I checked that the likelihood evaluation time is consistently around 0.4 seconds per likelihood call, which seems consistent with your comment. I also checked that the runs that were running well had a similar evaluation time, so this doesn't seem to be what's causing problems. I am not sure if it is related to parallelization. The thing is that in the cluster run report, the number of CPU hours used was much less than the number of CPUs times the time of the run. So it would seem that either the parallelization has some issue and / or that the sampler is getting stuck doing something that is not being parallelized. For now I'll have to accept that for some reason efficiency seems to dramatically drop for high enough nlive, so I'll try to cap that, but would be nice to get to the bottom of the issue.

[segasai]

I think something is not right, but I don't think there is enough information say what.

If you say 0.4sec per eval, with ~ 190 function calls per iteration that will give you ~ 1 new sample per minute on 1 CPU.
However, when new samples are requested, they are requested in queue_size bulk, so if the longest new sample takes ~ 60 sec, you would expect to get at least queue_size other samples in the same time.

The only thing I can suspect is that if you had started the run on a laptop single threaded, and then continued the run on the cluster, because in that case the queue_size is not updated (that probably should be changed BTW)

[dadams42]

I see. The sampler was, however, initialised using the code of the first snapshot and in that case the number of CPUs in the machine matching the queue_size (97). Also, the funny thing is that the sampler worked fine for most of the run using the same code in the same machine before stalling. Perhaps it would be interesting to know if anyone has run before the sampler with nlive = 12,000+ and 34+ dimensions, especially if parallelizing like I did? If it's not something specific to the likelihood I use then I would expect it could be reproduced.

[segasai]

• There is nothing particularly problematic with 12,000pts, 34-dim and 100 threads. I've just ran 12000 livepoints on my laptop sampling of 34-d gaussian with n=97 pool, and I get nice consistent sampling speed. (this is with 3.0 that I have installed, but it should be the same with dyn 2.4)

0.1001487906it [59:01, 438.91it/s, batch: 0 | bound: 624 | nc: 201 | ncall: 280400894 | eff(%):  0.531 | loglstar:   -inf < -7.765 <    inf | logz: -125.315 +/-  0.095 | dlogz:  0.104 >  0.100
1488002it [59:01, 505.78it/s, batch: 0 | bound: 624 | nc: 180 | ncall: 280419578 | eff(%):  0.531 | loglstar:   -inf < -7.761 <    inf | logz: -125.315 +/-  0.095 | dlogz:  0.104 >  0.100
1488099it [59:02, 245.20it/s, batch: 0 | bound: 624 | nc: 171 | ncall: 280438254 | eff(%):  0.531 | loglstar:   -inf < -7.757 <    inf | logz: -125.315 +/-  0.095 | dlogz:  0.103 >  0.100

• I looked again at the screenshots of your runs. The large it/s shown is just a display issue (with the tqdm module) when you resume a run it counts all the past iterations as recently done, so you see the "high" it/s in the beginning, but in fact for the screenshots that you show you get consistent ~ 60s per iteration for the whole 12hr. I am convinced (assuming 0.4 sec per function evaluation you gave) the only explanation is 100% sequential execution (and most likely queue_size=1, but maybe there is some other issue with restoring and running sequentially).
• You mention earlier stalling, but since there are no diagnostics from that I can't comment on that.

[dadams42]

Thanks @segasai for letting me know. The queueing issue sounds reasonable, but what I can say is that the code I've used is the one shown and there is nothing there that should set an incorrect queue_size. Perhaps there is some obscure reason not related to dynesty for this. For now I guess I'll treat it as a fluke and hope it doesn't repeat. It is unfortunate, however, that I had invested many CPU hours in building the sampler to an advanced stage and it remains stuck (which, as I said, didn't happen before, both with the same sampler and others I've run with the same code).

[segasai]

Can you check the queue_size ? if you just pickle load the save file, and then print the sampler.queue_size , you should see the value. Also, if you just run it in environment where you can actually check if multiple cpus are being used would be useful

[dadams42]

So sampler.queue_size does indeed return 97, like it is supposed to. I remember running the sampler in my laptop too and got similar results (about ~ 1 MB increase in the checkpoint file after a day), though I ran it with the code shown in the third snapshot. The strange thing is that the exact same code in the same computing cluster was working fine until a certain stage in the run where the 'stalling' happened (if you're interested, I could privately share with you the ~1.2 GB sampler and the (lighter) files for defining the likelihood in case you find anything odd).

[dadams42]

P.S. Is there any chance that I may have accidentally run the same code twice at once (one with lower CPUs) and this somehow 'broke' the sampler and caused the issue? If not the only other thing I can think of is if something got updated or a file slightly modified in the cluster without my knowledge, but the main code and everything I have checked so far haven't changed (and indeed work fine in new runs and other samplers).