Running with multiple nodes on a compute cluster

[MilesCranmer]

Latest way to do this:

using ClusterManagers, SymbolicRegression

# Create a custom operator (will be passed to the processes):
inv(x) = 1/x

# Create dataset (locally)
X = rand(Float32, 5, 100) .+ 1
y = 1.2f0 .+ 2 ./ X[3, :]

# Create options (locally)
options = SymbolicRegression.Options(;
    binary_operators=(+, *),
    unary_operators=(inv,),
    npopulations=1000,
    ncyclesperiteration=10000,
)

# Run with these procs (it will automatically distribute them!)
hall_of_fame = EquationSearch(X, y, niterations=8, options=options, 
                            numprocs=40*4, addprocs_function=addprocs_slurm,
                            parallelism=:multiprocessing)

dominating = calculate_pareto_frontier(X, y, hall_of_fame, options)
best = dominating[end]

---

Old example:

Here's an example of running SymbolicRegression.jl on a cluster.

using Distributed, ClusterManagers

# Create processes across cluster (here we use Slurm, but others exist - see ClusterManagers.jl)
procs = addprocs_slurm(40*4; lazy=true)

# Activate the current project on all processes:
using Pkg
project_path = splitdir(Pkg.project().path)[1]
@everywhere begin
    Base.MainInclude.eval(quote
        using Pkg
        Pkg.activate($$project_path)
    end)
end

# import this package everywhere
@everywhere using SymbolicRegression

# Create a custom operator (will be passed to the processes):
inv(x) = 1/x

# Create dataset (locally)
X = rand(Float32, 5, 100) .+ 1
y = 1.2f0 .+ 2 ./ X[3, :]

# Create options (locally)
options = SymbolicRegression.Options(;
    binary_operators=(+, *),
    unary_operators=(inv,),
    npopulations=1000,
    ncyclesperiteration=10000,
)

# Run with these procs (it will automatically distribute them!)
hallOfFame = EquationSearch(X, y, niterations=8, options=options, procs=procs)

# Close down processes:
rmprocs(procs)

dominating = calculateParetoFrontier(X, y, hallOfFame, options)
best = dominating[end]

[MilesCranmer]

Update: on v0.8.5, this is even simpler - just pass the addprocs_slurm (or addprocs_pbs, etc.) to EquationSearch:

using ClusterManagers, SymbolicRegression

# Create a custom operator (will be passed to the processes):
inv(x) = 1/x

# Create dataset (locally)
X = rand(Float32, 5, 100) .+ 1
y = 1.2f0 .+ 2 ./ X[3, :]

# Create options (locally)
options = SymbolicRegression.Options(;
    binary_operators=(+, *),
    unary_operators=(inv,),
    npopulations=1000,
    ncyclesperiteration=10000,
)

# Run with these procs (it will automatically distribute them!)
hallOfFame = EquationSearch(X, y, niterations=8, options=options, 
                            numprocs=40*4, addprocs_function=addprocs_slurm,
                            parallelism=:multiprocessing)

dominating = calculate_pareto_frontier(X, y, hallOfFame, options)
best = dominating[end]