Question : Use inputs in different sections of the network

[HuguesDug]

I read the code rather intensivelly, as the examples, and was still not able to find how to do the following task.

I have an input, sizex = 20, sizey=2, depth= 1

I want to have a network that will process the inputs depth=1, x from 1 to 10, y=1, another that will process depth=1 y=1, x from 11 to 20, ... same with y=2.
This gives 4 blocks that would pass through several layers.
Then, I want to join the results together so the net continues.

I found that you can keep in memory the created layer with assigning the result of the create to a variable, so you can add layers to it. Fine. This is done with using : NN.AddLayerAfter.

I found you can join the result with Concat, fine. This is done using : TNNetDeepConcat

But how do you force the network to use only some of the input data for a specific portion of the net. I read the "resize" and "split", ... does not seems to be the right answers. The split seems to use only the depth, and nothing to specify the neurons to use.

[joaopauloschuler]

Hi,
In the case that you could have your input as SizeX=10, SizeY=1 and Depth=4,
I could see 2 steps:

1. Separate the input according to the channel (or depth). For this, you can use TNNetSplitChannels. As an example, in an RGB image, you can use TNNetSplitChannels to get the R channel with TNNetSplitChannels.Create([0]) or TNNetSplitChannels.Create(0,1). In your case, your network will have 4 branches:

branch0 := NN.AddLayerAfter(TNNetSplitChannels.Create([0]), InputLayer);
branch1 := NN.AddLayerAfter(TNNetSplitChannels.Create([1]), InputLayer);
branch2 := NN.AddLayerAfter(TNNetSplitChannels.Create([2]), InputLayer);
branch3 := NN.AddLayerAfter(TNNetSplitChannels.Create([3]), InputLayer);

2. Concatenate branches with either TNNetDeepConcat or TNNetConcat. TNNetDeepConcat requires SizeX and SizeY to be the same.

There are some complex examples here:
https://sourceforge.net/p/cai/svncode/HEAD/tree/trunk/lazarus/experiments/visualCifar10BatchUpdate/uvisualcifar10learningbatchupdate.pas

Are you able to separate your input according to the Depth so you could use TNNetSplitChannels?

[HuguesDug]

Thanks for the comprehensive answer.

With the TNNetSplitChanenels, I will be able to do a first portion.

The data I have will be fed in the net TVolume by program. I can shape the data more or less as I want. But on the other hand, I need to take a portion of the layer data to feed a branch (only a section of a layer).

I also need to process the data in the with same x channel together as inputs. So, I would need to "flip" the data.

[joaopauloschuler]

Hi, I probably miss understood you. I thought that you had 4 portions of 10 values each. In this understanding, you could create 4 channels (depth = 4). If this is not the case, I could create a layer that copies an arbitrary region of the input layer. What do you think?

[HuguesDug]

I over simplified my example to make it clear. And your last answer is exactly what I inted to do : get a layer that copy a arbitrary region of the input... but I could not find in the librairy where this possibility is given.
This would be a "TNNETCustomLayerInputCopy" ?
Any idea on how it should work ?

[joaopauloschuler]

Hi,
I'm wondering if you could shape your input to something like this: SizeX=1, SizeY=1, Depth=40 (or any number). Then, you could copy portions of the input with TNNetSplitChannels.Create(ChannelStart, ChannelLen: integer).

If the above doesn't work for you, I could create a layer that copies cubic regions from the previous layer.

What do you think?

[HuguesDug]

Would work... but I am going to have something like 200 inputs... I think getting a strip along x or along y would be better.

A layer type : TNnetLayercopy(inputx,inputy, dx, dy): would be exactly what would help.

If you can help creating this, I think it would be a good add on for the lib.

[HuguesDug]

Hello,

After reviewing the WEB, the solution to the problem would be something like this.
It is the way "Keras/Tensorflow" operates.

define two sets of inputs

inputA = Input(shape=(32,))
inputB = Input(shape=(128,))

the first branch operates on the first input

x = Dense(8, activation="relu")(inputA)
x = Dense(4, activation="relu")(x)
x = Model(inputs=inputA, outputs=x)

the second branch opreates on the second input

y = Dense(64, activation="relu")(inputB)
y = Dense(32, activation="relu")(y)
y = Dense(4, activation="relu")(y)
y = Model(inputs=inputB, outputs=y)

combine the output of the two branches

combined = concatenate([x.output, y.output])
....
....

Several inputs (InputA and InputB) that you use as a previous layer in the tree, results being concat later before output layer.

The problem is in the code of TNNet.Compute that accepts only one input. It the case of several input layers, it should then iterate though all the input layers in the NET and calculate next layers only when all the previous layers have been calculated. With the use of Compute from Index, I think it would not be that much of a rewrite of code.

[joaopauloschuler]

New compute methods have been added allowing multiple inputs:

procedure Compute(pInput: array of TNeuralFloatDynArr); overload;
procedure Compute(pInput: array of TNeuralFloat; FromLayerIdx:integer = 0); overload;

It's assumed that first layers are inputs. Follows an example:

type TBackInput  = array[0..3] of array[0..1] of TNeuralFloat;
type TBackOutput = array[0..3] of array[0..2] of TNeuralFloat;

const inputs : TBackInput =
  ( // x1,   x2
    ( 0.1,  0.1), // False, False
    ( 0.1,  0.9), // False, True
    ( 0.9,  0.1), // True,  False
    ( 0.9,  0.9)  // True,  True
  );

const reluoutputs : TBackOutput =
  (// XOR, AND,   OR
    ( 0.1, 0.1, 0.1),
    ( 0.8, 0.1, 0.8),
    ( 0.8, 0.1, 0.8),
    ( 0.1, 0.8, 0.8)
  );

  procedure RunAlgo();
  var
    NN: TNNet;
    EpochCnt: integer;
    Cnt: integer;
    pOutPut: TNNetVolume;
    vInputs: TBackInput;
    vOutput: TBackOutput;
    FirstInput, SecondInput,x,y: TNNetLayer;
  begin
    NN := TNNet.Create();
    FirstInput := NN.AddLayer( TNNetInput.Create(2) );
    SecondInput := NN.AddLayer( TNNetInput.Create(2) );
    
    // first branch
    NN.AddLayerAfter( TNNetFullConnectReLU.Create(3),  FirstInput);
    x := NN.AddLayer( TNNetFullConnectReLU.Create(3) );
    
    // second branch
    NN.AddLayerAfter( TNNetFullConnectReLU.Create(3),  SecondInput);
    y := NN.AddLayer( TNNetFullConnectReLU.Create(3) );
    
    NN.AddLayer( TNNetConcat.Create([x, y]));
    NN.AddLayer( TNNetFullConnectReLU.Create(3) );
    NN.SetLearningRate(0.01, 0.9);

    vInputs := inputs;
    vOutput := reluoutputs;
    pOutPut := TNNetVolume.Create(3,1,1,1);

    for EpochCnt := 1 to 3000 do
    begin
      for Cnt := Low(inputs) to High(inputs) do
      begin
        NN.Compute([vInputs[Cnt],vInputs[Cnt]]);
        NN.GetOutput(pOutPut);
        NN.Backpropagate(vOutput[Cnt]);
        if EpochCnt mod 300 = 0 then
        WriteLn
        (
          EpochCnt:7,'x',Cnt,
          ' Output:',
          pOutPut.Raw[0]:5:2,' ',
          pOutPut.Raw[1]:5:2,' ',
          pOutPut.Raw[2]:5:2,
          ' - Training/Desired Output:',
          vOutput[cnt][0]:5:2,' ',
          vOutput[cnt][1]:5:2,' ' ,
          vOutput[cnt][2]:5:2,' '
        );
      end;
      if EpochCnt mod 300 = 0 then WriteLn();
    end;
    NN.DebugWeights();
    NN.DebugErrors();
    pOutPut.Free;
    NN.Free;
    Write('Press ENTER to exit.');
    ReadLn;
  end;                                          

[HuguesDug]

Excellent !

However, I am lost with your example
I guess it should be as follow, as voutput does not have same number of cnt as inputs ?

for EpochCnt := 1 to 3000 do
begin
for Cnt := Low(inputs) to High(inputs) do
begin
NN.Compute([vInputs[Cnt],vInputs[Cnt]]);
end;
NN.GetOutput(pOutPut);
NN.Backpropagate(vOutput);

[joaopauloschuler]

I updated the example. Have a look please if it makes more sense. I'm submitting both inputs with the same data just to show the syntax (it doesn't make sense having 2 inputs with the same data).

[HuguesDug]

OK with the example.
I will test on my dataset and will report if I see a bug or a problem. But it sounds all good.

[HuguesDug]

Hello,
I tried to compile the new release with Delphi 10.3.3, release mode.

3 errors are raised :

• neuralnetwork.pas, procedure TNNet.Compute(pInput: array of TNeuralFloatDynArr);
  line 8063 : Compute(pInput[0]);
  Error E2251, Ambiguous overloaded call to 'Compute'.
  Cause, compiler cannot chose witch version of the compute procedure to call, due to default value FromLayerIdx:integer in procedure TNNet.Compute(pInput: array of TNeuralFloat; FromLayerIdx:integer = 0);
  Fix --> Compute(pInput[0]) must be replaced by Compute(pInput[0],0)

• neuralnetwork.pas, procedure TNNetLayer.ComputeErrorDeriv();
  line 9263 : FallbackComputeErrorDeriv();
  Error E2501, Inline function cannot call nested routine 'FallbackComputeErrorDeriv'
  Cause, nested procedure call inside an inline funtion
  Fix --> Declare the procedure FallbackComputeErrorDeriv out as a methode of object TNNetLayer or copy the code.
  /!\ the procedure does not perform the same code if FPC and if Delphi. Very strange. For me, potential bug.

• neuralthread, function fNTL: TNeuralThreadList;
  line 125, function fNTL: TNeuralThreadList;
  Error 2441, Inline function declared in interface section must not use local symbol 'vNTL'
  Cause, the vNTL function is declared in the implementation section, where inline is in interface section. Different scope do not allow inline.
  Fix --> move the vNTL var statement (line 90) to the implementation section (line 69)

[joaopauloschuler]

In regards to "FallbackComputeErrorDeriv", this portion of code works only on FPC:

if FActivationFn = @Identity then
  begin
    FOutputErrorDeriv.Copy(FOutputError);
  end 

Both codes do the same thing (except that the FPC version is faster).