ReLU function works weird with NaN input

[T-H-ash]

Hi,

It seems F.relu(NaN) returns 0, where NaN is expected. Here's a code sample:

console.log(`debug-cpu: ${(await (await F.relu(new K.nn.Variable(T.fromArray([NaN])))).data.to('cpu')).toArray()}`);
console.log(`debug-webgl: ${(await (await F.relu(new K.nn.Variable(await T.fromArray([NaN]).to('webgl')))).data.to('cpu')).toArray()}`);

And I got:

debug-cpu: 0
debug-webgl: 0

It seems the grad also returns 0 for NaN input (I did't confirm it well), where NaN is expected.

[milhidaka]

https://stackoverflow.com/questions/37532640/making-a-nan-on-purpose-in-webgl
Whether or not NaN is implemented depends on the model, and it is difficult to handle well.