[BUG] NextJournal Tutorial Failing; Does not Work with MLJFlux

[TheCedarPrince]

Hi @ashryaagr and maintainers,

I was trying to follow along with the NextJournal Tutorial for Fairness.jl - great tutorial by the way! - but it seems that there is some sort of error now with MLJFlux. Here are the steps to reproduce what I am seeing:

1. Create tmp environment: pkg> activate --temp
2. Add Fairness, MLJ, MLJFlux to environment
3. Should see ArrayInterface and MLJFlux error in precompiling

Here is the full stack trace when I try to load a NeuralNetworkClassifier:

julia> @load NeuralNetworkClassifier
[ Info: For silent loading, specify `verbosity=0`.
import MLJFlux[ Info: Precompiling MLJFlux [094fc8d1-fd35-5302-93ea-dabda2abf845]
ERROR: LoadError: UndefVarError: @aggressive_constprop not defined
Stacktrace:
 [1] top-level scope
   @ ~/.julia/packages/ArrayInterface/gMtB5/src/ArrayInterface.jl:12
 [2] include
   @ ./Base.jl:419 [inlined]
 [3] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{
String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Ve
ctor{Pair{Base.PkgId, UInt64}}, source::String)
   @ Base ./loading.jl:1554
 [4] top-level scope
   @ stdin:1
in expression starting at /home/cedarprince/.julia/packages/ArrayInterface/gMtB5/sr
c/ArrayInterface.jl:1
in expression starting at stdin:1
ERROR: LoadError: Failed to precompile ArrayInterface [4fba245c-0d91-5ea0-9b3e-6abc
04ee57a9] to /home/cedarprince/.julia/compiled/v1.8/ArrayInterface/jl_nxMy3K.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_std
out::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:1705
  [3] compilecache
    @ ./loading.jl:1649 [inlined]
  [4] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1337
  [5] _require_prelocked(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1200
  [6] macro expansion
    @ ./loading.jl:1180 [inlined]
  [7] macro expansion
    @ ./lock.jl:223 [inlined]
  [8] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1144
  [9] include(mod::Module, _path::String)
    @ Base ./Base.jl:419
 [10] include(x::String)
    @ Flux ~/.julia/packages/Flux/KkC79/src/Flux.jl:1
 [11] top-level scope
    @ ~/.julia/packages/Flux/KkC79/src/Flux.jl:28
 [12] include
    @ ./Base.jl:419 [inlined]
 [13] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector
{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::V
ector{Pair{Base.PkgId, UInt64}}, source::String)
    @ Base ./loading.jl:1554
 [14] top-level scope
    @ stdin:1
in expression starting at /home/cedarprince/.julia/packages/Flux/KkC79/src/optimise
/Optimise.jl:1
in expression starting at /home/cedarprince/.julia/packages/Flux/KkC79/src/Flux.jl:
1
in expression starting at stdin:1
ERROR: LoadError: Failed to precompile Flux [587475ba-b771-5e3f-ad9e-33799f191a9c]
to /home/cedarprince/.julia/compiled/v1.8/Flux/jl_qD0gAY.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_std
out::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:1705
  [3] compilecache
    @ ./loading.jl:1649 [inlined]
  [4] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1337
  [5] _require_prelocked(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1200
  [6] macro expansion
    @ ./loading.jl:1180 [inlined]
  [7] macro expansion
    @ ./lock.jl:223 [inlined]
  [8] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1144
  [9] include
    @ ./Base.jl:419 [inlined]
 [10] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector
{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::V
ector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base ./loading.jl:1554
 [11] top-level scope
    @ stdin:1
in expression starting at /home/cedarprince/.julia/packages/MLJFlux/6XVNm/src/MLJFl
ux.jl:1
in expression starting at stdin:1
ERROR: Failed to precompile MLJFlux [094fc8d1-fd35-5302-93ea-dabda2abf845] to /home
/cedarprince/.julia/compiled/v1.8/MLJFlux/jl_Gyt3nF.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_std
out::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:1705
  [3] compilecache
    @ ./loading.jl:1649 [inlined]
  [4] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1337
  [5] _require_prelocked(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1200
  [6] macro expansion
    @ ./loading.jl:1180 [inlined]
  [7] macro expansion
    @ ./lock.jl:223 [inlined]
  [8] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1144
  [9] eval
    @ ./boot.jl:368 [inlined]
 [10] eval(x::Expr)
    @ Base.MainInclude ./client.jl:478
 [11] _import(modl::Module, api_pkg::Symbol, pkg::Symbol, doprint::Bool)
    @ MLJModels ~/.julia/packages/MLJModels/GKDnU/src/loading.jl:34
 [12] top-level scope
    @ ~/.julia/packages/MLJModels/GKDnU/src/loading.jl:206

I noticed that my package for MLJ was quite old so I also ran an update via pkg> up after this error. After updating, trying to load the NeuralNetworkClassifier results in this error:

julia> @load NeuralNetworkClassifier
[ Info: For silent loading, specify `verbosity=0`.
import MLJFlux[ Info: Precompiling MLJFlux [094fc8d1-fd35-5302-93ea-dabda2abf845]
ERROR: LoadError: UndefVarError: doc_header not defined
Stacktrace:
 [1] getproperty(x::Module, f::Symbol)
   @ Base ./Base.jl:31
 [2] top-level scope
   @ ~/.julia/packages/MLJFlux/6XVNm/src/types.jl:134
 [3] include(mod::Module, _path::String)
   @ Base ./Base.jl:419
 [4] include(x::String)
   @ MLJFlux ~/.julia/packages/MLJFlux/6XVNm/src/MLJFlux.jl:1
 [5] top-level scope
   @ ~/.julia/packages/MLJFlux/6XVNm/src/MLJFlux.jl:25
 [6] include
   @ ./Base.jl:419 [inlined]
 [7] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{
String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Ve
ctor{Pair{Base.PkgId, UInt64}}, source::Nothing)
   @ Base ./loading.jl:1554
 [8] top-level scope
   @ stdin:1
in expression starting at /home/cedarprince/.julia/packages/MLJFlux/6XVNm/src/types
.jl:134
in expression starting at /home/cedarprince/.julia/packages/MLJFlux/6XVNm/src/MLJFl
ux.jl:1
in expression starting at stdin:1
ERROR: Failed to precompile MLJFlux [094fc8d1-fd35-5302-93ea-dabda2abf845] to /home
/cedarprince/.julia/compiled/v1.8/MLJFlux/jl_MwDCnk.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_std
out::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:1705
  [3] compilecache
    @ ./loading.jl:1649 [inlined]
  [4] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1337
  [5] _require_prelocked(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1200
  [6] macro expansion
    @ ./loading.jl:1180 [inlined]
  [7] macro expansion
    @ ./lock.jl:223 [inlined]
  [8] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1144
  [9] eval
    @ ./boot.jl:368 [inlined]
 [10] eval(x::Expr)
    @ Base.MainInclude ./client.jl:478
 [11] _import(modl::Module, api_pkg::Symbol, pkg::Symbol, doprint::Bool)
    @ MLJModels ~/.julia/packages/MLJModels/GKDnU/src/loading.jl:34
 [12] top-level scope
    @ ~/.julia/packages/MLJModels/GKDnU/src/loading.jl:206

Additionally, here is my current set-up and version info:

Julia Version: 1.8
[7c232609] Fairness v0.3.2
⌃ [add582a8] MLJ v0.16.7
[094fc8d1] MLJFlux v0.2.9
Info Packages marked with ⌃ have new versions available

What am I doing incorrectly here? Really looking forward to working with this package - thanks! 😄

[ashryaagr]

Hi. It seems that there is no issue on your end. But as pointed out on #78 , this might be a compatibility error.
I will try to get this to work. If I can't get this to work, then I will try to send you the Project.toml i.e. exact versions of the setup I had (old list though), which worked.

[TheCedarPrince]

Hi @ashryaagr - ah I thought that it was related! If you don't have time right now, if you send me the Project.toml of the older version I could create a PR that adds compat entries to the Project.toml of Fairness. Then, as you go through trying to see what can be updated with CompatHelper, we could update the compat entries incrementally until things are largely up to date. Would that work/be helpful? Feel free to drop the toml here and I could pick up on things from there.

[ashryaagr]

Hi @TheCedarPrince - Here is the Project.toml . This is basically the same as that currently on repository.

My best guess for the reason for this issue is that MLJFlux has been updated and Fairness.jl is behind on updates (most probably only with compat updates). So, can you try by restricting MLJFLux to be of version v0.1.6. Pkg.add("MLJFlux@v0.1.6").

And once Fairness.jl gets back on track with the updates, it should be usable with the latest MLJ and MLJFlux version

Big thanks for taking the initiative to help with the compat entries!!

[ashryaagr]

Update - turns out that restricting to v1.6 leaves no satisfiable version of Flux to install due to some CUDA issue. It might also be because I opened my dual-booted linux system after a long time and lag on some update.

[ashryaagr]

### So here is a solution - not a standard solution, but a workaround for the time-being. You can download the repository and delete all the compat requirements from Fairness. Then you can install that package. Then you can install latest version of MLJ and MLJFlux. I have tried this on my system and this fix does work. I do not get any errors with @load NeuralNetworkClassifier

And in case you still run into an error like this (I did, but was able to solve this):

ERROR: LoadError: UndefVarError: @aggressive_constprop not defined

then, delete the packages in your ~/.julia/packages . Then re-install the downloaded repository, MLJ and MLJFlux. This solved this problem and everything ran smoothly.

The only issue is that LinProgWrapper and PenaltyAlgorithm give errors during tests on my system - perhaps because they require changes with these latest packages. But anyways these two algorithms are our own proposed algorithms. The other standard algorithms like reweighing, equalized odds and calibrated equalized odds work perfectly.

[TheCedarPrince]

Hey! Thanks for the temporary workaround! I can confirm I was able to get this working but instead of destroying everything in .julia/packages, I was able to do the following:

1. Clone Fairness.jl repository
2. Delete compat entries from Project.toml
3. Create temporary environment (I created one for myself by running pkg> activate @DEV)
4. Dev Fairness into the environment (i.e. @DEV> dev /path/to/Fairness/.)
5. Install MLJ and MLJFlux via @DEV> add MLJ, MLJFlux
6. Update MLJ and MLJFlux to latest versions and dependencies @DEV> up

And that works.
I ran the test suite for Fairness and can confirm the only failing tests are for LinProgWrapper and PenaltyWrapper -- that may need to be its own issue perhaps.

I'll open a PR on compat entries that we can start experimenting with.
Thanks for all the help so far!

P.S. For completeness, here is the stack trace of the failed tests:

     Testing Running tests...
Test Summary: | Pass  Total   Time
fair_tensor   |    3      3  16.3s
Evaluating over 6 folds: 100%[=========================] Time: 0:00:06
Evaluating over 6 folds: 100%[=========================] Time: 0:00:00
Evaluating over 6 folds: 100%[=========================] Time: 0:00:00
Evaluating over 6 folds: 100%[=========================] Time: 0:00:00
Test Summary: | Pass  Total   Time
measures      |   56     56  24.6s
Test Summary:  | Pass  Total     Time
dataset_macros |   24     24  1m10.9s
[ Info: Training machine(ReweighingWrapper(grp = Sex, …), …).
[ Info: Training machine(ConstantClassifier(), …).
[ Info: Training machine(ReweighingSamplingWrapper(grp = Sex, …), …).
[ Info: Training machine(ConstantClassifier(), …).
[ Info: Training machine(EqOddsWrapper(grp = Sex, …), …).
[ Info: Training machine(ConstantClassifier(), …).
[ Info: Training machine(CalEqOddsWrapper(grp = Sex, …), …).
[ Info: Training machine(ConstantClassifier(), …).
[ Info: Training machine(LinProgWrapper(grp = Sex, …), …).
[ Info: Training machine(ConstantClassifier(), …).
┌ Error: Problem fitting the machine machine(LinProgWrapper(grp = Sex, …), …
).
└ @ MLJBase ~/.julia/packages/MLJBase/CtxrQ/src/machines.jl:627
[ Info: Running type checks...
[ Info: Type checks okay.
LinProgWrapper Postprocessing: Error During Test at /home/src/Projects/open_
source_contribs/Fairness.jl/test/algorithms/postprocessing.jl:26
  Got exception outside of a @test
  UndefVarError: add_NL_constraint not defined
  Stacktrace:
    [1] getproperty
      @ ./Base.jl:31 [inlined]
    [2] fit(model::LinProgWrapper{ConstantClassifier}, verbosity::Int64, X::
DataFrames.DataFrame, y::CategoricalArrays.CategoricalVector{Int64, UInt32,
Int64, CategoricalArrays.CategoricalValue{Int64, UInt32}, Union{}})
      @ Fairness /home/src/Projects/open_source_contribs/Fairness.jl/src/alg
orithms/postprocessing/linprog.jl:106
    [3] fit_only!(mach::Machine{LinProgWrapper{ConstantClassifier}, true}; r
ows::Nothing, verbosity::Int64, force::Bool)
      @ MLJBase ~/.julia/packages/MLJBase/CtxrQ/src/machines.jl:625
    [4] fit_only!
      @ ~/.julia/packages/MLJBase/CtxrQ/src/machines.jl:572 [inlined]
    [5] #fit!#61
      @ ~/.julia/packages/MLJBase/CtxrQ/src/machines.jl:693 [inlined]
    [6] fit!(mach::Machine{LinProgWrapper{ConstantClassifier}, true})
      @ MLJBase ~/.julia/packages/MLJBase/CtxrQ/src/machines.jl:690
    [7] macro expansion
      @ /home/src/Projects/open_source_contribs/Fairness.jl/test/algorithms/
postprocessing.jl:31 [inlined]
    [8] macro expansion
      @ /home/src/Programs/Julia/julia-1.8.0/share/julia/stdlib/v1.8/Test/sr
c/Test.jl:1357 [inlined]
    [9] top-level scope
      @ /home/src/Projects/open_source_contribs/Fairness.jl/test/algorithms/
postprocessing.jl:27
   [10] include(fname::String)
      @ Base.MainInclude ./client.jl:476
   [11] top-level scope
      @ /home/src/Projects/open_source_contribs/Fairness.jl/test/algorithms/
algorithms.jl:2
   [12] include(fname::String)
      @ Base.MainInclude ./client.jl:476
   [13] macro expansion
      @ /home/src/Projects/open_source_contribs/Fairness.jl/test/runtests.jl
:19 [inlined]
   [14] macro expansion
      @ /home/src/Programs/Julia/julia-1.8.0/share/julia/stdlib/v1.8/Test/sr
c/Test.jl:1357 [inlined]
   [15] top-level scope
      @ /home/src/Projects/open_source_contribs/Fairness.jl/test/runtests.jl
:19
   [16] include(fname::String)
      @ Base.MainInclude ./client.jl:476
   [17] top-level scope
      @ none:6
   [18] eval
      @ ./boot.jl:368 [inlined]
   [19] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:276
   [20] _start()
      @ Base ./client.jl:522
┌ Warning: The `@pipeline` macro is deprecated. For pipelines without target
 transformations use pipe syntax, as in `ContinuousEncoder() |> Standardizer
() |> my_classifier`. For details, query the `Pipeline` docstring. To wrap a
 supervised model in a target transformation, use `TransformedTargetModel`,
as in `TransformedTargetModel(my_regressor, target=Standardizer())`
│   caller = ip:0x0
└ @ Core :-1
[ Info: For silent loading, specify `verbosity=0`.
import MLJDecisionTreeInterface ✔
[ Info: For silent loading, specify `verbosity=0`.
import MLJDecisionTreeInterface ✔
[ Info: Training machine(PenaltyWrapper(grp = sex, …), …).
┌ Error: Problem fitting the machine machine(PenaltyWrapper(grp = sex, …), …
).
└ @ MLJBase ~/.julia/packages/MLJBase/CtxrQ/src/machines.jl:627
[ Info: Running type checks...
[ Info: Type checks okay.
PenaltyWrapper Inprocessing: Error During Test at /home/src/Projects/open_so
urce_contribs/Fairness.jl/test/algorithms/inprocessing.jl:1
  Got exception outside of a @test
  MethodError: no method matching Fairness.FairTensor(::Array{Any, 3}, ::Vec
tor{InlineStrings.String7})
  Closest candidates are:
    Fairness.FairTensor(::Any, ::Vector{String}) at /home/src/Projects/open_
source_contribs/Fairness.jl/src/fair_tensor.jl:17
  Stacktrace:
    [1] fit(model::PenaltyWrapper{Pipeline316}, verbosity::Int64, X::DataFra
mes.DataFrame, y::CategoricalArrays.CategoricalVector{Int64, UInt32, Int64,
CategoricalArrays.CategoricalValue{Int64, UInt32}, Union{}})
      @ Fairness /home/src/Projects/open_source_contribs/Fairness.jl/src/alg
orithms/inprocessing/penalty.jl:64
    [2] fit_only!(mach::Machine{PenaltyWrapper{Pipeline316}, true}; rows::No
thing, verbosity::Int64, force::Bool)
      @ MLJBase ~/.julia/packages/MLJBase/CtxrQ/src/machines.jl:625
    [3] fit_only!
      @ ~/.julia/packages/MLJBase/CtxrQ/src/machines.jl:572 [inlined]
    [4] #fit!#61
      @ ~/.julia/packages/MLJBase/CtxrQ/src/machines.jl:693 [inlined]
    [5] fit!(mach::Machine{PenaltyWrapper{Pipeline316}, true})
      @ MLJBase ~/.julia/packages/MLJBase/CtxrQ/src/machines.jl:690
    [6] macro expansion
      @ /home/src/Projects/open_source_contribs/Fairness.jl/test/algorithms/
inprocessing.jl:6 [inlined]
    [7] macro expansion
      @ /home/src/Programs/Julia/julia-1.8.0/share/julia/stdlib/v1.8/Test/sr
c/Test.jl:1357 [inlined]
    [8] top-level scope
      @ /home/src/Projects/open_source_contribs/Fairness.jl/test/algorithms/
inprocessing.jl:2
    [9] include(fname::String)
      @ Base.MainInclude ./client.jl:476
   [10] top-level scope
      @ /home/src/Projects/open_source_contribs/Fairness.jl/test/algorithms/
algorithms.jl:3
   [11] include(fname::String)
      @ Base.MainInclude ./client.jl:476
   [12] macro expansion
      @ /home/src/Projects/open_source_contribs/Fairness.jl/test/runtests.jl
:19 [inlined]
   [13] macro expansion
      @ /home/src/Programs/Julia/julia-1.8.0/share/julia/stdlib/v1.8/Test/sr
c/Test.jl:1357 [inlined]
   [14] top-level scope
      @ /home/src/Projects/open_source_contribs/Fairness.jl/test/runtests.jl
:19
   [15] include(fname::String)
      @ Base.MainInclude ./client.jl:476
   [16] top-level scope
      @ none:6
   [17] eval
      @ ./boot.jl:368 [inlined]
   [18] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:276
   [19] _start()
      @ Base ./client.jl:522
Test Summary:                              | Pass  Error  Total   Time
algorithms                                 |    8      2     10  33.2s
  Reweighing                               |    2             2   4.8s
  ReweighingSampling                       |    2             2   2.0s
  Equalized Odds Postprocessing            |    2             2  10.1s
  Calibrated Equalized Odds Postprocessing |    2             2   2.1s
  LinProgWrapper Postprocessing            |           1      1   7.0s
  PenaltyWrapper Inprocessing              |           1      1   3.1s
ERROR: LoadError: Some tests did not pass: 8 passed, 0 failed, 2 errored, 0
broken.
in expression starting at /home/src/Projects/open_source_contribs/Fairness.j
l/test/runtests.jl:18
ERROR: Package Fairness errored during testing