The training dataset is extremely large - how should it be handled in CAI?

[achinastone]

[joaopauloschuler]

Training with Volume Pairs - TNeuralDataLoadingFit

The TNeuralFit implementation has a limitation: your dataset needs to be placed into RAM. In the case that your dataset is too large for RAM, you can call TNeuralDataLoadingFit:

TNNetGetPairFn = function(Idx: integer; ThreadId: integer): TNNetVolumePair of object;
TNNetGet2VolumesProc = procedure(Idx: integer; ThreadId: integer; pInput, pOutput: TNNetVolume) of object;
  
TNeuralDataLoadingFit = class(TNeuralFitBase)
...
    procedure FitLoading(pNN: TNNet;
      TrainingCnt, ValidationCnt, TestCnt, pBatchSize, Epochs: integer;
      pGetTrainingPair, pGetValidationPair, pGetTestPair: TNNetGetPairFn); overload;
    procedure FitLoading(pNN: TNNet;
      TrainingCnt, ValidationCnt, TestCnt, pBatchSize, Epochs: integer;
      pGetTrainingProc, pGetValidationProc, pGetTestProc: TNNetGet2VolumesProc); overload;

The Hypotenuse with FitLoading example uses TNeuralDataLoadingFit so it creates training pairs on the fly.

[achinastone]

I'm using this exact approach right now. But when it runs for an extended period, can we save the training state and pick up where we left off later?

[joaopauloschuler]

If you have a validation accuracy or loss metric, it will automatically save it for you:

        if (FSaveBest = SaveBestAccuracy) then
        begin
          if (ValidationRate > ValidationRecord) then
          begin
            ValidationRecord := ValidationRate;
            FMessageProc('VALIDATION RECORD! Saving NN at '+fileName);
            FAvgWeight.SaveToFile(fileName);
          end;
        end;

        if (FSaveBest = SaveBestLoss) then
        begin
          if (FValidationLoss < ValidationRecord) then
          begin
            ValidationRecord := FValidationLoss;
            FMessageProc('VALIDATION RECORD! Saving NN at '+fileName);
            FAvgWeight.SaveToFile(fileName);
          end;
        end;

Use this (from uneuralfit):

  TNeuralSaveBest = (SaveBestAccuracy, SaveBestLoss);  
...
  TNeuralFitBase = class(TMObject)
    protected
      FSaveBest: TNeuralSaveBest;
...
    public
...
      property SaveBest: TNeuralSaveBest read FSaveBest write FSaveBest;

[joaopauloschuler]

To load, use this:

[achinastone]

Thanks.