How can I speed up the search?

[skadoosh-MC]

Hi,

I usually run PySR on an HPC and let it run for 4 days at a time but I find that the loss is still quite large and the equations haven't evolved as much as I wanted them to. I've been reading and implementing the tips from Astroautomata, like turning on turbo, bumper and batching, keeping the operators simple, keeping a large ncycles_per_iteration, etc. One thing I would like to mention here is that I've kept a large maxsize since I'd like more complex equations (I know this can slow the process down a little bit). I've even downsampled the number of points in my dataset. Do you have any other suggestions that could help speed up the search?

My second question is, is there a way to save a model and then continue training from where it stopped?

Below is what my hyperparameters look like -

# loss is defined before the regressor object

model_pk = PySRRegressor(
            binary_operators=["+", "-", "*", "/", "^"],        
            constraints={'^': (-1, 1)},         
            unary_operators=["square"],
            elementwise_loss=elementwise_loss,     
            niterations=1000,
            populations=200,
            population_size=300,             
            maxsize=70,            	
            maxdepth=20,   		
            timeout_in_seconds=345600,     # 4 days
            ncycles_per_iteration=3000,
            weight_optimize=0.001,
            weight_add_node=0.004522835200528927,
            weight_insert_node=27.524411192119594,
            weight_mutate_constant=3.4908752644974177,
            weight_mutate_operator=0.005418386628230309,
            weight_do_nothing=0.001374964776098089,
            fraction_replaced=0.013374993832223066,
            fraction_replaced_hof=0.030981822809764206,
            procs=40,    	  # cores in a node 
            batching=True,
            turbo=True,
	    bumper=True   
)

Any help will be appreciated!! Thank you in advance : D

[MilesCranmer]

A few tips:

1. Don't use square as an operator if you are also using ^ since they are degenerate with eachother.
2. I would impose stricter constraints on operators such as / and ^.
3. Since you are using a larger maxsize you could experiment with providing a parsimony. As I say on the tuning page, I like to set this to 0.1x the top loss I would expect for a particular problem.
4. Experiment with batch size.
5. @gm89uk has found that adaptive_parsimony_scaling set to 20 seems to produce better results sometimes.

I also find that warmup_maxsize_by can sometimes help. But note that it is set relative to niterations rather than timeout_in_seconds so you need to select the specific fraction based on niterations.

You can also try template expressions. If you know the functional form, this can help a lot. https://ai.damtp.cam.ac.uk/pysr/examples/#template-expressions

My second question is, is there a way to save a model and then continue training from where it stopped?

Yes you can do this with warm_start=True and then just call .fit again. It should start where it left off. You can even use this as a sort of adaptive runs – perhaps gradually turning down the parsimony over time might help, so that the model only gradually explores more complex solutions.