[BUG]: Cannot load guesses in Template Expression if function contains parameters

[gm89uk]

What happened?

For TemplateExpressions, I cannot load guesses that contain a parameter:
guesses = [(f="(((#1 - #2) * -0.16528662) + #4) - #3")],

In this instance #4 is p1[n]:

    expression_spec = @template_spec(expressions = (f,),parameters=(p1=n,)) do x1,x2,x3,x4 #x4 is param, #n is number of categories in x4
        LF = p1[x4]
	return f(x1,x2,x3,LF)
 end

The search runs no problem, but I cannot load the model again. Thank you in advance.

Version

v2.0.0-alpha.8

Operating System

Windows

Interface

Julia REPL

Relevant log output

     
ERROR: syntax: misplaced assignment statement in "[f = "(((#1 - #2) * -0.16528662) + #4) - #3"]" around REPL[42]:1
Stacktrace:
 [1] top-level scope
   @ REPL[42]:1

julia> mach = machine(model, X, y)
ERROR: UndefVarError: `model` not defined in `Main`
Suggestion: check for spelling errors or missing imports.
Stacktrace:
 [1] top-level scope
   @ REPL[43]:1

Extra Info

[MilesCranmer]

This specific error is from a missing comma. The fix is:

- [(f="(((#1 - #2) * -0.16528662) + #4) - #3")]
+ [(f="(((#1 - #2) * -0.16528662) + #4) - #3",)]

[MilesCranmer]

I think it should otherwise work though. Parameters will get optimised similar to constants at the start of the search.

(It's a bit lower level, but to provide exact constant guesses, one can also pass a full TemplateExpression as input. That being said, you shouldn't really need this as it will optimize before adding to the population)

Feel free to reopen if I'm missing something though!

[gm89uk]

Thanks Miles, that fixed the syntax issue, but the guess does not load (optimisation does not seem to run at the start, I have not tried the 'lower level' loading).

I will try to set up some simple test code to demonstrate and request to re-open if it is indeed a persistent issue.

[gm89uk]

Here you are. Despite giving a perfect solution as a guess, it always rediscovers the entire solution.

using SymbolicRegression, Random, DataFrames, MLJ

n = 500                         
p1 = [5, 10, 0.8, 3.1]       
noise_level = 0.1               
rng = MersenneTwister(123)      
x_data = rand(rng, n) * 10.0
cat_data = rand(rng, 1:4, n)
y_true = @. p1[cat_data]*x_data^3 + sqrt(x_data) / p1[cat_data]
noise = randn(rng, n) * noise_level
y_data = y_true .+ noise
X = (x=x_data, cat=cat_data)
y = y_data
df = DataFrame(X)
df.y = y
expression_spec = @template_spec(expressions = (f,), parameters=(p1=4,)) do x, cat
    return f(x, p1[cat])
end
model = SRRegressor(
    elementwise_loss = L1DistLoss(),
    niterations      = 100000,
    binary_operators = [+, -, *, /],
    unary_operators  = [sqrt],
    maxsize          = 30,
    guesses = [(f="(#2 * ((#1 * #1) * #1)) + (sqrt(#1) / #2)",)],
    populations      = 32,
    population_size  = 200,
    parsimony        = 0.0,
    expression_spec  = expression_spec,
)

mach = machine(model, X, y)
fit!(mach)

[MilesCranmer]

Thanks. It's weird, I wonder if it just means the optimizer is failing or something. Here is where it triggers the optimization of a guess:

https://github.com/MilesCranmer/SymbolicRegression.jl/blob/137db25c2af3d3a7fa221d0b03656af618402698/src/SearchUtils.jl#L804

[gm89uk]

Without parameters, it does load guesses successfully. Not sure how much help I can be to you as to why the underlying code isn't working.

Could it be because in template_spec the parameters are a variable rather than literal constants? When it first loads the expression, perhaps there is no node with constants to optimise, they aren't part of the tree yet? So the guess is parsed into an expression tree then the optimizer traverse the tree looking for constant nodes to tune, that do not yet exist?

[MilesCranmer]

Weird. So there is a unittest for parametric template expression guess parsing: https://github.com/MilesCranmer/SymbolicRegression.jl/blob/137db25c2af3d3a7fa221d0b03656af618402698/test/test_guesses.jl#L80-L108

But it doesn't actually check if the constant optimization is run. However, a parametric template expression treats the parameters like any other constants, so there shouldn't (in principle) be any reason for these to not get optimized too. Very weird.

[gm89uk]

The main difference I can see in the test example is that the parameters are not within f(..).
It has been so incredibly useful being able to load previous HoF, hopefully you can figure it out! Thanks Miles.

[gm89uk]

Hi Miles,

I think I have figured out why this doesn't work. I changed the equation to make it harder for SR to guess it quickly to differentiate between lucky runs and genuine differences:

I think the issue is in parameter optimisation; the initial parameters used, if they are not close to the true parameters, results in a high loss and the guess gets discarded.

Therefore, to test the theory, I set: fraction_replaced_guesses=1.0 and compared against fraction_replaced_guesses=0.0, and found that setting this to 1.0 actually finds the solution much quicker, because I suppose it keeps forcing the guesses functional form back in and better chance for the parameters optimiser to improve leading to lower loss, which permits it to enter HoF.

Changing the parameters to something easy like in the below code (from p1 = [5, 10, 0.8, 3.1] to p1 = [1, 1.1, 0.9, 1.01]) made the guess load much more successfully, because the parameter optimiser got there much quicker, and therefore the guess was not discarded due to high loss from poor initial parameter optimisation.

using SymbolicRegression, Random, DataFrames, MLJ

n = 500                         
#p1 = [5, 10, 0.8, 3.1]      
p1 = [1, 1.1, 0.9, 1.01]  #easy, close parameters works much better      
noise_level = 0.001               
rng = MersenneTwister(123)      
x_data = rand(rng, n) * 10.0
cat_data = rand(rng, 1:4, n)
y_true = @. (p1[cat_data]*x_data^3 + sqrt(x_data)) / (p1[cat_data] + sqrt(x_data))
noise = randn(rng, n) * noise_level
y_data = y_true .+ noise
X = (x=x_data, cat=cat_data)
y = y_data
df = DataFrame(X)
df.y = y
expression_spec = @template_spec(expressions = (f,), parameters=(p1=4,)) do x, cat
    return f(x, p1[cat])
end
model = SRRegressor(
    elementwise_loss = L1DistLoss(),
    niterations      = 100000,
    binary_operators = [+, -, *, /],
    unary_operators  = [sqrt],
    maxsize          = 30,
    guesses = [(f="((#2 * #1 * #1 * #1) + sqrt(#1)) / (#2 + sqrt(#1))",)],
    fraction_replaced_guesses=1.0, #toggle between 0.0 and 1.0
    populations      = 32,
    population_size  = 200,
    parsimony        = 0.0,
    expression_spec  = expression_spec,
)

mach = machine(model, X, y)
fit!(mach)

Here is another example:

Initially the parameter guesses were way off although the equation was in the HoF, however, I took a screenshot too late

Later the parameters further optimised and the loss dropped:

So same functional form, but the parameters changes the loss. So even though the parameters were very close in the first screenshot, the difference in loss is quite large, meaning it's relatively easy for slightly off parameters, to lead to high loss and the equation not entering the HoF.

Therefore, to conclude, poor parameter optimisation for the initial guess, leads to high loss and discarding of the guess. A high fraction_replaced_guesses leads to higher chance of the guess being used and the success of it loading. Wide ranging difficult parameters, are unlikely to be found by the optimiser initially, therefore, even the true form of the equation will be discarded due to high loss.

Hope that makes sense!

[MilesCranmer]

I guess the issue is that there's not an easy way to provide the parameters in the guess at the moment? Let me see.

[gm89uk]

Loading the parameters is the definitive solution. Maybe letting the parameter optimiser 'optomise' for a bit longer at the initial load for each guess, before discarding from the HoF would also work. Although I'm not sure how faithfully the optimiser can find the initial parameters especially if there are a large number of parameters to get the same loss that the guess initially had.