MCMC chain termination condition

[tgvaughan]

A recent issue #554 suggested switching to 64 bit integers to specify MCMC chain lengths. This reminded me of something (only tangentially related) that has been bugging me for a while: are chain lengths really the best way for users to specify how long the algorithm should run for? To me there appear to be two alternatives:

1. Specify run length in wall time
2. Specify run length in ESS of a chosen parameter

The advantage of (1) is that we can allocate precisely the amount of time we have available for the calculation. The drawbacks are that (a) the actual amount of calculation performed will be system-dependent, meaning that exact reproducibility is no longer possible, and (b) the wall time on its own says nothing about the quality of results the algorithm will produce.

The advantages of (2) are precisely the drawbacks of (1): requiring that the ESS of a (set of?) well-chosen parameter(s) exceeds some threshold goes some way toward ensuring we only stop when we have some useful results. It also maintains reproducibility. The only drawback is that we can't say precisely how long the algorithm will run for.

In contrast, the only advantage of the existing approach of stopping after a particular number of iterations seems to be reproducibility, which it shares with the ESS approach. Neither the ESS approach nor the current approach allow users to say exactly how long a run will take, but I think this should be a fairly low priority given that (a) beast 2 allows chain resumption following interruption and that (b) there is little point stopping if your ESS doesn't meet some threshold.

I therefore think it would be better to use an ESS-based stopping condition (or at least offer one). What do you guys think?

[alexeid]

I like running until an ESS is reached (if we do that we should fix resume to just complete the chain, rather than run the full length again)

The MrBayes approach is to always run two parallel chains and stop when they are close enough to each other. I like that option a lot as well. Again, resume should just complete when reaching the original criterion as judged by the full concatenated run.

On 18/05/2016, at 1:45 PM, Tim Vaughan notifications@github.com wrote:

A recent issue #554 #554 suggested switching to 64 bit integers to specify MCMC chain lengths. This reminded me of something (only tangentially related) that has been bugging me for a while: are chain lengths really the best way for users to specify how long the algorithm should run for? To me there appear to be two alternatives:

Specify run length in wall time
Specify run length in ESS of a chosen parameter
The advantage of (1) is that we can allocate precisely the amount of time we have available for the calculation. The drawbacks are that (a) the actual amount of calculation performed will be system-dependent, meaning that exact reproducibility is no longer possible, and (b) the wall time on its own says nothing about the quality of results the algorithm will produce.

The advantages of (2) are precisely the drawbacks of (1): requiring that the ESS of a (set of?) well-chosen parameter(s) exceeds some threshold goes some way toward ensuring we only stop when we have some useful results. It also maintains reproducibility. The only drawback is that we can't say precisely how long the algorithm will run for.

In contrast, the only advantage of the existing approach of stopping after a particular number of iterations seems to be reproducibility, which it shares with the ESS approach. Neither the ESS approach nor the current approach allow users to say exactly how long a run will take, but I think this should be a fairly low priority given that (a) beast 2 allows chain resumption following interruption and that (b) there is little point stopping if your ESS doesn't meet some threshold.

I therefore think it would be better to use an ESS-based stopping condition (or at least offer one). What do you guys think?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub #555

[tgvaughan]

A related idea is to allow scientific notation in setting chain lengths. Or even just log10 units. Seriously, how often do you run chains for lengths that are not powers of 10? Every time I do this I wind up going cross-eyed trying to count the zeros...

[jheled]

My personal .1 cents is that this is an issue for BEAST1, not BEAST2. When I pitched BEAST2 to Remco a bazillion years ago I stressed the importance of resuming, as the main reason for users to switch.

I can't say it happened, as there are still too many BEAST1 users, but being able to resume mitigates most of this issue.

[tgvaughan]

@jheled sure, it doesn't really matter, but am I the only one who regularly sets chains to run for like 10^9 iterations with no intention of ever running it that long, just because I don't yet know how many iterations it will take? I've never found much use at all for the current termination condition, apart from ensuring reproducibility once I've already done the analysis and know exactly how long I want to run it for.

[alexeid]

Many users don’t know how long to run a chain for, even if they can choose to resume. We should take the decision out of their hands by using a convergence-based diagnostic.

Even people in our own group regularly do simulation studies that make no sense because they have not run the chains long enough. Sometimes there are too many analyses and people don’t bother visually inspecting the trace of every analysis to see if it has been run long enough.

On 18/05/2016, at 1:51 PM, Joseph Heled notifications@github.com wrote:

My personal .1 cents is that this is an issue for BEAST1, not BEAST2. When I pitched BEAST2 to Remco a bazillion years ago I stressed the importance of resuming, as the main reason for users to switch.

I can't say it happened, as there are still too many BEAST1 users, but being able to resume mitigates most of this issue.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub #555 (comment)

[jheled]

Obviously I am not against the idea, but for me the real question is not implementing this kind of convenience, but finding out how you define a sane "ESS" for convergence of the tree in tree space. This is (to me) the real issue. How confident we are RE sampling the tree space ....

[alexeid]

Again I think the measure that MrBayes uses is the best around at the moment, but it requires two parallel chains. I prefer the diagnostic statistic to be the MAXIMUM difference in clade probability between the two chains, over all clades. The stopping criterion should be when this diagnostic statistic < 0.05 or something similar. Hoehna and Drummond (2012) used this criterion. MrBayes uses the average difference between clade probabilities averaged over all clades that appear in one or other of the chains. This is weak, since the average is dominated by clades that are either almost always present, or almost always absent in both chains. It is exactly the clades with intermediate support that can differ the most between two runs.

On 18/05/2016, at 2:05 PM, Joseph Heled notifications@github.com wrote:

Obviously I am not against the idea, but for me the real question is not implementing this kind of convenience, but finding out how you define a sane "ESS" for convergence of the tree in tree space. This is (to me) the real issue. How confident we are RE sampling the tree space ....

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub #555 (comment)

[jheled]

Then a mode where the "single" run is internally "implemented" as two parallel independent runs seem natural and important, right? Then implementing those kinds of between-chain criteria become trivial. Nowadays I run at least 8 chains with different seeds as my first move ....

Perhaps this is already implemented ???

[tgvaughan]

I suppose you can extend this idea to the rest of the parameter space using something like K-L divergence?

[rbouckaert]

FYI, MultiMCMC in BEASTlabs already implements the Rubin-Gelman statistic, and though it does not use it as stopping criterion, it does not take a lot of effort to make it do so.

[alexeid]

I think we need to focus on the standard templates. Experts can do whatever they like (including programming and exotic plugins) but the defaults of the standard templates should be more bullet-proof.

On 19/05/2016, at 3:18 PM, Remco Bouckaert notifications@github.com wrote:

FYI, MultiMCMC in BEASTlabs already implements the Rubin-Gelman statistic, and though it does not use it as stopping criterion, it does not take a lot of effort to make it do so.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub