Zygote.refresh() gone without a trace!

[mchitre]

Zygote.refresh() is necessary to clear the gradient cache and reload rules when new rules are defined. This is documented in the Zygote documentation (e.g. Custom adjoints).

This worked until v0.7.4 and was removed in v0.7.5 in PR #1560. Given that this was documented, it should be considered a public API. So this change is BREAKING and isn't compliant with semver. More importantly, it was not even documented in the release notes!

The claim in #1560 is that Zygote.refresh() is no longer required. Clearly this isn't true. See:

julia> using Zygote, ChainRulesCore

julia> function f5(x::AbstractArray)
         y = similar(x)
         for i ∈ eachindex(x)
           y[i] = x[i] > 0 ? x[i]^2 : zero(eltype(x))
         end
         sum(y)
       end
f5 (generic function with 1 method)

julia> Zygote.gradient(f5, [1.7, 2.1, -1.5, -0.1, 3.1])
ERROR: Mutating arrays is not supported -- called setindex!(Vector{Float64}, ...)
This error occurs when you ask Zygote to differentiate operations that change
the elements of arrays in place (e.g. setting values with x .= ...)

Possible fixes:
- avoid mutating operations (preferred)
- or read the documentation and solutions for this error
  https://fluxml.ai/Zygote.jl/latest/limitations

Stacktrace:
   :

julia> function ChainRulesCore.rrule(::typeof(f5), x)
         J = [xᵢ > 0 ? 2xᵢ : zero(xᵢ) for xᵢ ∈ x]
         f5_pullback(v) = NoTangent(), J * unthunk(v)
         f5(x), f5_pullback
       end

julia> Zygote.gradient(f5, [1.7, 2.1, -1.5, -0.1, 3.1])
ERROR: Mutating arrays is not supported -- called setindex!(Vector{Float64}, ...)
This error occurs when you ask Zygote to differentiate operations that change
the elements of arrays in place (e.g. setting values with x .= ...)

Possible fixes:
- avoid mutating operations (preferred)
- or read the documentation and solutions for this error
  https://fluxml.ai/Zygote.jl/latest/limitations

Stacktrace:
  :

In previous versions (v0.7.4), a Zygote.refresh() after defining the rule worked:

julia> using Zygote, ChainRulesCore

julia> function f5(x::AbstractArray)
         y = similar(x)
         for i ∈ eachindex(x)
           y[i] = x[i] > 0 ? x[i]^2 : zero(eltype(x))
         end
         sum(y)
       end
f5 (generic function with 1 method)

julia> Zygote.gradient(f5, [1.7, 2.1, -1.5, -0.1, 3.1])
ERROR: Mutating arrays is not supported -- called setindex!(Vector{Float64}, ...)
This error occurs when you ask Zygote to differentiate operations that change
the elements of arrays in place (e.g. setting values with x .= ...)

Possible fixes:
- avoid mutating operations (preferred)
- or read the documentation and solutions for this error
  https://fluxml.ai/Zygote.jl/latest/limitations

Stacktrace:
  :

julia> function ChainRulesCore.rrule(::typeof(f5), x)
         J = [xᵢ > 0 ? 2xᵢ : zero(xᵢ) for xᵢ ∈ x]
         f5_pullback(v) = NoTangent(), J * unthunk(v)
         f5(x), f5_pullback
       end

julia> Zygote.refresh()

julia> Zygote.gradient(f5, [1.7, 2.1, -1.5, -0.1, 3.1])
([3.4, 4.2, 0.0, 0.0, 6.2],)