Revisiting Symbolic regression with "conditional constants" (#521)

[debraaudus]

Thank you for developing pySR! I am facing the problem as described in #521 asked by @mkellerressel . Specifically, the problem is to learn an equation that has two types of inputs, numeric (x) and categorical (i). I want a single equation where the constants ($\theta$) depend on the categorical variable, $y \approx f(x,\theta_i)$

In the responses there where three suggestions:
(1) The approach of the paper Rediscovering orbital mechanics with machine learning. I don't think this approach will work for my more general case. Please correct me if I'm wrong.
(2) Using custom trees via an update Program synthesis/graph expressions #271. At the time of the previous post, this wasn't implemented. But, it is now. Any pointers on how to do this would be greatly appreciated. For example, do I need to modify the source code?
(3) Using a custom loss function as suggested by @gm89uk . This was really interesting approach, but it wasn't obvious to me how to extend for non-additive cases. For example, two non-linear constants.

Any thoughts or pointers would be appreciated.

[MilesCranmer]

The latest PySR which came out some time after those previous responses has a feature dedicated for this! Check out TemplateExpressionSpec, specifically with the parameters option: https://ai.damtp.cam.ac.uk/pysr/examples/#11-expression-specifications:

spec = TemplateExpressionSpec(
    expressions=["f", "g"],
    variable_names=["x1", "x2", "class"],
    parameters={"p1": 3, "p2": 3},
    combine="p1[class] * sin(f(x1, x2)) + p2[class]",
)

model = PySRRegressor(expression_spec=spec)
model.fit(X, y)

In this example, the third column of X is the categorical feature, and the first two are regular features.

We also prescribe the specific form they are used. But you can also just write down

    combine="f(x1, x2, p1[class], p2[class])"

and it is like your expression f has two conditional constants.

[debraaudus]

Thank you so much! After remembering that Julia indices start at 1, I was able to successfully find the answer for a toy problem with a known answer. It does seem like one needs to know how many constants a priori. Presumably the number should be small, so it isn't too much work to iterate over possibilities, say 1 to 4 constants. However, if you have ideas of how to get around that, I'd be interested.

[MilesCranmer]

I think iterating over possibilities is honestly the best bet. Start with 1 conditional constant, and increase from there. This gives you another "axis" of the Pareto frontier. You could define some kind of metric to select the optimal number of conditional constants based on some regularisation penalty, but it is somewhat subjective. I think in research you may want to list multiple options – e.g., "here's the best expression for 1 tunable constant, and here's the best expression for 2."

[debraaudus]

I agree. That's a very reasonable approach.

[mkellerressel]

ParametricExpressionSpec and TemplateExpressionSpec are great additions to pySR, thanks @MilesCranmer for including them! My master student who is working on the problem that sparked the question (#521) has used them with success to tackle the 'conditional regression' problem.

[MilesCranmer]

Great to hear!