Running topology search in parallel

[StephenStaklinski]

I noticed an issue that seems to be related to running the topology search in parallel. It does not happen when running sequentially. Please see the below error message and feel free to let me know if this issue is on my end:

Starting topology search
Running topology search in parallel...
Traceback (most recent call last):
  File "/grid/siepel/home/staklins/miniforge3/envs/laml/bin/run_laml", line 7, in <module>
    sys.exit(main())
             ~~~~^^
  File "/grid/siepel/home/staklins/miniforge3/envs/laml/bin/run_laml.py", line 290, in main
    opt_trees,max_score,opt_params = myTopoSearch.search(resolve_polytomies=resolve_polytomies,maxiter=args["maxIters"], verbose=args["verbose"], strategy=my_strategy, nreps=args['randomreps'],checkpoint_file=checkpoint_file)
                                     ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/grid/siepel/home/staklins/miniforge3/envs/laml/lib/python3.13/site-packages/laml_libs/Topology_search.py", line 145, in search                                                          
    trees,score,params = self.__search_one__(strategy,maxiter=maxiter,verbose=verbose,only_marked=False,checkpoint_file=checkpoint_file)                                                         
                         ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                         
  File "/grid/siepel/home/staklins/miniforge3/envs/laml/lib/python3.13/site-packages/laml_libs/Topology_search.py", line 196, in __search_one__                                                  
    new_score,n_attempts,success = self.single_nni(curr_score,nni_iter,strategy,only_marked=only_marked)                                                                                         
                                   ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                         
  File "/grid/siepel/home/staklins/miniforge3/envs/laml/lib/python3.13/site-packages/laml_libs/Topology_search_parallel.py", line 56, in single_nni                                              
    return new_score,curr_end_idx,took
           ^^^^^^^^^
UnboundLocalError: cannot access local variable 'new_score' where it is not associated with a value  

[StephenStaklinski]

I believe this was related to having an expired Mosek license, since updating the license eliminated the error. It might be nice to implement a check for if the license is valid and print a more clear message if not. I received a generic warning about EM having an issue on the M step, but it took a bit of work to track down the license issue.

I'll also add here that laml v1.0.5 does not seem to output ultrametric trees as previously done in v0.0.4. I wonder if this is a desired or undesired change?

[gillichu]

Hi Stephen,

Thanks for your interest in LAML and sorry for the belated reply!

We currently do have a test implemented in laml_tests.py which will check the validity of the Mosek license by solving a small problem (here). As is run_laml.py only checks that the Mosek license file exists in the license file. My apologies for the extra hurdles in tracking down the licensing issue. You make a good suggestion and I will incorporate a check for the expiration date in the plaintext Mosek license file.

re: your second point. laml v1.0.5 should still output ultrametric trees. This was true as recently as v1.0.0 (see here). Could you give the exact command and inputs you're using? If convenient you can use one of our test examples.

With this command run_laml -c character_matrix.csv -t starting.tree -p priors.csv -o example1 --nInitials 1 --timescale 10 in the example1/ folder, I get the following logging:

Launching LAML version 1.0.5
LAML was called as follows: /Users/gc3045/miniconda3/envs/test_laml/bin/run_laml -c character_matrix.csv -t starting.tree -p priors.csv -o example1 --nInitials 1 --timescale 10
No duplicates, 250 unique sequences remain.
Optimizing branch lengths, phi, and nu without topology search
Optimization by EM algorithm
Initial point 1. Random seed: 5900
Numerical optimization started with ultrametric constraint (default)
Initial phi: 0.24198859249347954. Initial nu: 0.2673964827238777. Initial nllh: 24909.34286406755
EM finished after 50 iterations.
Optimal phi: 0.14373612720559714. Optimal nu: 0.1337720068102268. Optimal nllh: 6987.3524788278
Optimal point found for initial point 1
Numerical optimization finished successfully
Tree height pre-scaling: 1.0339927482953675, input timescale: 10
Tree height after scaling: 10.000000000000002, mutation rate: 0.10339927482953674
Runtime (s): 20.078005417002714

I'll bundle this with some other updates and leave this issue open till I push the newer version.

[StephenStaklinski]

Thanks for looking into this! I'll admit I ran the tests when first installing an old version last year, but I must have not ran them again after upgrading the version. I also just checked and you're right that v1.0.5 is giving ultrametric trees for me now too. There must have been something odd on my end last week, maybe all related to the Mosek license issue. Thanks again.