Add option to throw error on passing wrong precision floats to layers

[lassepe]

Motivation and description

The warning about wrong precision is very helpful to point at potential performance issues

Flux.jl/src/layers/stateless.jl

Line 60 in 2f19e68

@warn "Layer with Float32 parameters got Float64 input.

I think that this is the correct default behavior. However, in order to find out where the problem is coming from throwing an error to produce a stacktrace would be very helpful.

Possible Implementation

There could be a Preference or global flag that allows switching errors instead of warning for wrong precision inputs. This would also be made consistent with CUDA.allowscalar.

[mcabbott]

This seems like a good idea.

Maybe it should literally be the same switch as CUDA.allowscalar, to avoid introducing ever more functions you have to know about? It's not exactly the same meaning, but it's likely to be used at the same time.

That function belongs to GPUArraysCore, which Flux doesn't directly load right now, but NNlib does.

[darsnack]

When you say "same switch" do you mean defining something like Flux.throw_sanity_check_errors() which flips the requested precision switch in the OP as well as turning off CUDA.allowscalar? Or do you mean if CUDA.allowscalar is false, then we throw errors for precision? If it's the latter, then that seems wrong to me since scalar indexing and floating point precision are totally unrelated.

[lassepe]

I also think it makes sense to a function for enabling each of these checks separately. But having a Flux.enable_all_sanity_check_errors() on top of that is good for discoverability (also since the docstring of that can also point to the relevant sub-functions).