Better way to notify host code of an error in a ROCm kernel ?

[fwyzard]

Currently the ways we have to let a ROCm kernel notify the host of problems are

• assert(<false condition>)
• abort()

Unfortunately they behave exactly like their host counterpart, causing the application to terminate immediately.

At the very least this makes it quite hard to test that error conditions are handled correctly:

• testRocmSoALayoutAndView_t tests that an out-of-bound access is properly detected; it is, but it results in

  :0:rocdevice.cpp            :3020: 6791392435440d us:  Callback: Queue 0x14e920e00000 aborting with error : HSA_STATUS_ERROR_EXCEPTION: An HSAIL operation resulted in a hardware exception. code: 0x1016
  

  which results in a unit test failure.

• alpakaTestBufferROCmAsync tests that a device-side assertion can be triggered and detected; this results in

  src/HeterogeneousCore/AlpakaInterface/test/alpaka/testBuffer.dev.cc:38: auto (anonymous namespace)::testDeviceSideError(const Device &)::(anonymous class)::operator()(const Acc1D &, int *, size_t) const: Device-side assertion `data[index] != 0' failed.
  :0:rocdevice.cpp            :3020: 6791407414776d us:  Callback: Queue 0x1523e1400000 aborting with error : HSA_STATUS_ERROR_EXCEPTION: An HSAIL operation resulted in a hardware exception. code: 0x1016
  

GPU core dump created: gpucore.5065
````

Going through to the HIP / ROCm documentation, I didn't find any method to notify the host of a device-side error condition without aborting the whole application.

Can we design something better using only client code ?

[fwyzard]

assign heterogeneous

[cmsbuild]

New categories assigned: heterogeneous

@fwyzard,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @fwyzard.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[fwyzard]

A simple approach could be to

• allocate a status variable in pinned host memory (possibly one per device)
• write the address of the status variable to a device global variable
• device-side error conditions cause a write to the status variable
• a mostly-sleeping thread monitors the value of the status variable; when it detects an error condition, it may throw an exception with as much information about the error as pssible

Upsides:

• no hard abort
• possibility of printing additional information before stopping the host application

Downsides:

• there is no way to abort the whole kernel that caused the error, and the other running kernels. Assuming that the expected reaction to the error condition is to print an informative error message and exit, this might no be a big issue;
• the exception will be generated in the polling thread, not in the thread that submitted the kernel with the error condition; while it may be true in general that the thread to reat to an asynchronous error is not the one submitting the work, we may need a way to notify the submitting thread of the error condition; maybe this could be implemented if the error handling is done in the alpaka library ?
• we should not submit more kernels or asynchronous work once an error has been detected; maybe this could be implemented if the error handling is done in the alpaka library ?
• the information that can be provided by the failing kernel seems to be limited to the block and thread id - the kernel name itself is not available in device code; maybe this could be implemented if the error handling is done in the alpaka library ?

It looks like most of the downsides could be avoided if this functionality is implemented directly inside the alpaka library.

[fwyzard]

It looks like most of the downsides could be avoided if this functionality is implemented directly inside the alpaka library.

Thinking about it some more, if the error checking is implemented inside alpaka, we could avoid a polling thread altogether: all alpaka library functions could simply check the status flag and raise an error accordingly.