Meaning/definition of « steps » in Tensorboard

[erwanlrx]

Hello!

Thank you for the integration of the tensorboard logger. However, on most graphs in Tensorboard I find it difficult to understand the X-axis, which is named « steps », and does not seem to be defined in the PySR documentation.

Intuitively, I would have imagined that this number of steps would match « niterations » or « niterations * ncycles_per_iteration », but it does not seem to the case.

Hence my question, what is the intuitive definition/meaning of these « steps » ? Or more concretely, in the Julia code how often/when are logged the loss of the different complexities ?

Thanks in advance,
Erwan

[MilesCranmer]

Here is where the logger callback sits: https://github.com/MilesCranmer/SymbolicRegression.jl/blob/725876517e5ce2a50b5085532532e9238b3bf217/src/SymbolicRegression.jl#L993. So the total number of logger "steps" is equal to niterations * populations * num_outputs, where num_outputs is the number of target features (i.e., y.shape[1], if it is a 2D array). Otherwise num_outputs=1.

Indeed it might be nicer to store it in terms of fractional niterations or something, but I'm not sure how to do that in tensorboard. Let me know if you know of a way.

[erwanlrx]

Thanks a lot for this answer,

I am not familiar with Julia code, but from the code I understand that the logger callback is in the while loop w.r.t. to niterations. So I guess, {1, .... populations * num_outputs} steps correspond to the first iteration, and {1+populations * num_outputs, ...., 2* populations * num_outputs} steps correspond to the second iteration and so on...

Therefore, is it correct to assume that if I instantiate the logger as follows:

"TensorBoardLoggerSpec(log_dir, log_interval= populations * num_outputs)"

then it will record only one step for each iteration ? or am I mistaken ?

The X-axis on the graph will still be in "steps" but at least it will be easier to interpret as each point will corresponds to an iteration.

[erwanlrx]

More generally, there seems to exist a related closed issue for Pytorch, https://github.com/Lightning-AI/pytorch-lightning/issues/2110 where they log by epoch instead of step.

From the explanations, in the tensorboard logs they seem to set the value of 'step' for each log.
In Python, in their example, it gives for instance "tensorboard_logs['step'] = self.current_epoch"
Maybe in Julia, I could be possible to do something similar, where you set somewhere the step as equal to 'niterations'

This issue is not urgent at all, but in the future I guess such visualization would help to better understand the convergence of PySR and to tune the hyperparameter niterations.