No delta G value produced eventually

[boningyang47-bit]

Hi,

'An online_analysis_interval that is not a multiple of the checkpoint_interval can lead to redundant information in the real time yaml file after recovering from checkpoints.
Please cite the following:

    Friedrichs MS, Eastman P, Vaidyanathan V, Houston M, LeGrand S, Beberg AL, Ensign DL, Bruns CM, and Pande VS. Accelerating molecular dynamic simulations on graphics processing unit. J. Comput. Chem. 30:864, 2009. DOI: 10.1002/jcc.21209
    Eastman P and Pande VS. OpenMM: A hardware-independent framework for molecular simulations. Comput. Sci. Eng. 12:34, 2010. DOI: 10.1109/MCSE.2010.27
    Eastman P and Pande VS. Efficient nonbonded interactions for molecular dynamics on a graphics processing unit. J. Comput. Chem. 31:1268, 2010. DOI: 10.1002/jcc.21413
    Eastman P and Pande VS. Constant constraint matrix approximation: A robust, parallelizable constraint method for molecular simulations. J. Chem. Theor. Comput. 6:434, 2010. DOI: 10.1021/ct900463w
    Chodera JD and Shirts MR. Replica exchange and expanded ensemble simulations as Gibbs multistate: Simple improvements for enhanced mixing. J. Chem. Phys., 135:194110, 2011. DOI:10.1063/1.3660669

WARNING: Did not converge to within specified tolerance.
max_delta = 5.199868e-05, tol = 1.000000e-12, maximum_iterations = 10000, iterations completed = 9999
Failed to reach a solution to within tolerance with adaptive: trying next method
WARNING: Did not converge to within specified tolerance.
max_delta = 1.409672e-05, tol = 1.000000e-12, maximum_iterations = 10000, iterations completed = 9999
Failed to reach a solution to within tolerance with adaptive: trying next method
WARNING: Did not converge to within specified tolerance.
max_delta = 8.630401e-07, tol = 1.000000e-12, maximum_iterations = 10000, iterations completed = 9999
Failed to reach a solution to within tolerance with adaptive: trying next method
XIO: fatal IO error 22 (Invalid argument) on X server "10.10.0.123:38.0"^M
after 11673 requests (11673 known processed) with 7 events remaining.^M'

These are the last few lines of the slurm output file. I have seen similar warnings previously, but the calculations still eventually produced the ΔG values. However, this time the job terminated with an XIO error instead.

I also checked the files in the results folder, and both 'shared_ABFEComplexAnalysisUnit-fddf33229de14b3098c71fee384e52d5_attempt_0' and 'scratch_ABFEComplexAnalysisUnit-fddf33229de14b3098c71fee384e52d5_attempt_0' are empty.

Since the university allows a maximum of 4 days of GPU time per job, and my ABFE calculations typically take around 3 days and 20 hours to complete, I am wondering whether the issue could be that the adaptive solver is exceeding the allocated time limit. If this is the case, what should I do to avoid this problem?

Thank you very much for your help.

[IAlibay]

Thanks for ropening this issue @boningyang47-bit

Could you give us some more information about the system & environment you are running?

1. Which version of openfe, and gufe arer you running?
2. How big a system are you urunning?
3. Have you tried resuming the simulation? e.g. https://docs.openfree.energy/en/latest/guide/execution/quickrun_execution.html#resuming-a-halted-job

I also checked the files in the results folder, and both 'shared_ABFEComplexAnalysisUnit-fddf33229de14b3098c71fee384e52d5_attempt_0' and 'scratch_ABFEComplexAnalysisUnit-fddf33229de14b3098c71fee384e52d5_attempt_0' are empty.

Do the simulation unit folders have anything in them?

[boningyang47-bit]

For points 1 and 2: I am using openfe 1.9.1 for ABFE calculations of a biocomplex containing a receptor, cofactor, and ligand. The entire system contains 49,249 atoms, including the water molecules, ions, and the complex.

For point 3: No, I haven't tried resuming the simulation because I am not so sure about the openfe process. I am running a number of calculations at the same time, and only these two have encountered this issue. From my investigation, both calculations stopped before reaching the time limit, and the computational time consumed was similar to that of the other successful calculations (around 3 days and 20 hours). So I guess the issue may be related to post-processing the simulation data rather than the MD simulation itself?

For point 4: Only these two files are empty. The other files, including the solvent simulation unit, solvent setup unit, solvent analysis unit, complex simulation unit, and complex setup unit, all contain data.

[IAlibay]

For point 3: No, I haven't tried resuming the simulation because I am not so sure about the openfe process. I am running a number of calculations at the same time, and only these two have encountered this issue. From my investigation, both calculations stopped before reaching the time limit, and the computational time consumed was similar to that of the other successful calculations (around 3 days and 20 hours). So I guess the issue may be related to post-processing the simulation data rather than the MD simulation itself?

Resuming also works for the post-processing step - essentially it will allow you to just restart that particular bit of the transformation. In this instance, I suspect that this is what you would need to do to complete things. However resuming support is only available as of openfe v1.10, so you would need to rerun this simulation with a newer version of openfe.

Since the university allows a maximum of 4 days of GPU time per job, and my ABFE calculations typically take around 3 days and 20 hours to complete, I am wondering whether the issue could be that the adaptive solver is exceeding the allocated time limit. If this is the case, what should I do to avoid this problem?

The issue is that pymbar is taking a while to process things, most likely it's failing to converge to a solution during the forward/reverse analysis. It's a known issue that we don't have a solution for right now. The best you could do for now, beyond running a longer simulations so you have more data to pass through to pymbar, is to just give the analysis step more time to run.

[boningyang47-bit]

Thanks a lot. I will give it a try. By the way, could you tell me where openfe 1.10 is available?

[IAlibay]

Thanks a lot. I will give it a try. By the way, could you tell me where openfe 1.10 is available?

You can mamba/conda install or use our docker images: https://docs.openfree.energy/en/stable/installation.html
I would recommend using the latest release of openfe (v1.12) since it addresses a few bugs / issues and also offers new features.