lsvec Shared Memory Race Condition

[coreyostrove]

This patches another shared memory bug in lsvec, this time a race condition. The bug is easiest explained by pointing at the relevant lines. Below is the implementation of lsvec for TimeIndependentMDCObjectiveFunction:

  lsvec = self.terms(paramvec, oob_check, "LS OBJECTIVE")
  if _np.any(lsvec < 0):
      bad_locs = _np.where(lsvec < 0)[0]
      msg = f"""
      lsvec is only defined when terms is elementwise nonnegative.
      We encountered negative values for terms[i] for indices i
      in {bad_locs}.
      """
      raise RuntimeError(msg)
  unit_ralloc = self.layout.resource_alloc('atom-processing')
  shared_mem_leader = unit_ralloc.is_host_leader
  if shared_mem_leader:
      lsvec **= 0.5
  if raw_objfn_lsvec_signs:
      if shared_mem_leader:
          raw_lsvec = self.raw_objfn.lsvec(self.probs, self.counts, self.total_counts, self.freqs)
          lsvec[:self.nelements][raw_lsvec < 0] *= -1
  unit_ralloc.host_comm_barrier()

The bug arises due to the interation of the positivity check if _np.any(lsvec < 0) and later on the update of lsvec using the raw_objfn signs lsvec[:self.nelements][raw_lsvec < 0] *= -1. Since the positivity check is happening on all ranks, but is with respect to shared memory, if a rank that is the shared memory leader hits and runs that sign update line before another rank looking at the same memory hits the positivity check this would raise an error.

After staring at this code for a bit I couldn't see any reason why this positivity check (or anything else in this function) should be happening on anything other the shared memory leader, so the patch is to just add a corresponding if statement which enforces that.

Thanks to @juangmendoza19 for reporting this error.

God I hate shared memory sometimes...

[rileyjmurray]

LGTM. Thanks for fixing!