Running Rome in multiple julia instances freezes solving

[lemauee]

Hi,

This is a issue very specific to experiments for my thesis/application:

As I want to perform a lot of individual solves for my thesis, I run multiple julia instances (each using multiple workers and threads) on a pretty beefy machine. This seems to be faster than running them sequentially, because parallelism can't be exploited everywhere. It apparently worked fine until the RoME v0.13 release (I think). Now the solver just "hangs up" in all but one instance (this happens at a different pose for every time I try).

This is the last thing that gets written to stdout (aka my logs ;)):

Solve Progress: approx max 1752, at iter 711 	 Time: 0:01:50�[K[ Info: CSM-5 Clique 47 finished

Solve Progress: approx max 1752, at iter 716 	 Time: 0:01:51�[K

I run the multiple julia instances each in their own screen session like

screen -S <sessionname>

And then fire up my my evaluation script that reads my dataset from a matfile and takes some other arguments (command gets auto-generated by my matlab frontend)

julia -t auto -p 8 --project=Masterarbeit/svn/julia -J ~/.julia/sysimage_RoME.so -- Masterarbeit/svn/julia/mmiSAM/evaluation/solve2DIncremental.jl --path tmp --file sigmaTrajTf-0-001-0-000-0-005_sigmaLmBearingAndRanging-0-004-0-001_wrongRatio-1-000_minDist-0-200_maxDist-2-000.mat --trajKeys tf --lmKeys bearingAndRanging --startPoseIdx 1 --endPoseIdx 150 --startPoseVal "[7.15793412168699;3.38794983135837;-1.36840501375598]" --plotSaveFinal 1 --plotSaveIter 1 --nRuns 1 --suffix variables --useMsgLikelihoods 0 --nullhypo 0.000000 --tukey 15.000000 --nKernels 100 --spreadNH 3.000000 --inflation 3.000000

I will resort to running things sequentally (as any online application would), but especcialy for tuning out parameters like spreadNH or inflation running things in parallel was very useful, as I don't have access to an infinite amount of machines ;) As multiprocess/multithread performance increased in the last releases, I suspect that something (maybe not even directly RoME-related, but in Julia general) gets in each others way, some lock does not get released.

Best,
Leo

[dehann]

Hi @lemauee ,

Oops that is odd. This might be something upstream. We'd have to look a bit more at what's happening. In the mean time, perhaps we can help compute results on local servers. Do you perhaps want to send tar.gz files of graphs you want solved and I can run on compute power here. Will return the result in new tar.gz. Perhaps in the process we start getting the same error here. I'm quite surprised a multi-Julia issue might be happening. If not there then something in Distributed.jl perhaps. Either way, we'd likely have to fix locally in the short term.

Sure you already using as such, but just in case: You can generate with fg = initfg(); getSolverParams(fg).graphinit=false and then the generation of graphs should be pretty quick. Solving can then use either graphinit (might not follow exact sequence) or treeinit (our desired but still experimental) method at later solve.

Lastly, I just merged (what I expect) the fix for your issue #378. Will tag IIF v0.21.3 in a few hours.

cc, might be worth @jim-hill-r (hi!) to look out for something like this multiple Julia's issue too.

Best,
Dehann

[lemauee]

Hi @dehann,

I don't know if sending you the tar.gz would help to reproduce the issue as expected, as I'm still doing a solve for every pose (I know that this is slow, but consistency to the online-approach we use in classical parametric fgs was important to us).

However, I'd happily send you the Julia-Project needed to run my stuff, the matfile and the commands to invoke the solve. Otherwise I could also set up the stuff through ssh on your server and then see if i can reproduce it there.

Best,
Leo

[lemauee]

The same hangup just happened when running only one julia instance, so this might not even be multiinstance-related (but the test yesterday suggested so...). Also, it occured on one of my private and not the universitys pcs. I recently did a fresh ubuntu install there, so nothing strange in the environment should hinder things there. Maybe its also worth to give it a try without a precompiled sysimage as I once had problems with RoMEPlotting as seen in JuliaRobotics/RoMEPlotting.jl#140 as you know.

Best, Leo

[lemauee]

Whats also notable is that there is absolutely nothing executed (no computational load) present when the hangup occurs (no CPU load in top or something similar can be seen).

[Affie]

Just a sanity check question, did you update the precompiled sysimage with the latest tags?

Other than that, do you know what clique is stalling?
One way to find out is to draw the tree (drawtree=true, showtree=true) and see at what clique it is waiting on.
Then you can try turning on debug messages for that clique CSM, say for 2:
ENV["JULIA_DEBUG"] = :csm_2
And post results here please.

[lemauee]

I just compiled the sysimage this morning using RoME v0.13.0 and IIF v0.21.2 .
The problem is that the clique it stopped on was another one everytime I think, but still need to confirm that. Some examples even ran through that just differed by the amount of noise or even just the setting of the spreadNH parameter and still "made it differently far". I just started a run without the compiled sysimage to roule out that source of failure.
To draw the tree I'll need a machine with x-server to start some pdf viewer, right?

[Affie]

To draw the tree I'll need a machine with x-server to start some pdf viewer, right?

You should be able to set drawtree=true and then open the tree over SSH, I haven't done it before, so just guessing.

Edit: The tree should be at the logger file location or /tmp/caesar

[Affie]

You could also try setting a timeout and then seeing if there is some more information on where it timed out, eg:
solveTree!(fg, timeout=70)

[lemauee]

timeout is in seconds I suppose? Then I should give it half an hour (1800s) or something for my example to be safe I'd say ;)
I will start an example with and without sysimage inclding drawtree, showtree and timeout now and report back.

[Affie]

Seconds, yes. It is per step for every clique individually, so should be fine lower.

[lemauee]

So lets make it 5 minutes, 300s to be save not to kill something prematurely