Simulation becomes slow due to Legion warning1097 #5
Description
Hello, I am a user of HTR-solver using an HPC system (unfortunately, a CPU-only system).
I have a problem when I run the test case named "Franko".
While the simulation runs, the output returns the message as below.
[21 - 2aab1f891880] 116.356580 {4}{runtime}: [warning 1097] LEGION WARNING: WARNING: The runtime has failed to memoize the trace more than 5 times, due to the absence of a replayable template. It is highly likely that trace 0 will not be memoized for the rest of execution. The most recent template was not replayable for the following reason: Remote shard not replyable. Please change the mapper to stop making memoization requests. (from file /home01/x2242a06/legion/runtime/legion/legion_trace.cc:2019)
For more information see: http://legion.stanford.edu/messages/warning_code.html#warning_code_1097
The warning does not stop the simulation. However, the simulation becomes slower and slower. The initial wall time for the one-step is 3sec, and the wall time increases continuously.
As shown in the figure, the iterations are even only 5000 steps, the wall-time increases almost 10-15esc. If I do the simulation until 50000 iterations, it would be 100-150 sec in this tendency.
Here, I have the first question. Is this tendency normal in the Franko case? In my guess, the periodic wall-time can be normal while the continuous increment of the wall-time is unusual behavior.
The second one is that if not, have you had this experience? I would like to how can I solve this problem.