Add paramters for controlling CArena defragmentation (#4479)

The defragmentation in #4451 has caused performance regression for some
applications on Frontier. For example, a Castro test had a 20%
performance hit. The issue appears to be that the GPU aware MPI on
Frontier does not work well with defragmenting the comms arena. In this
PR, we introduce a number of runtime parameters and disable
defragmentation for the comms arena by default.

Author: WeiqunZhang