I was toying around with the tiny xor example and I changed the training loop from the current loop:
for idx in range(ITER):
pred = model(x)
loss = tt.mse_loss(pred, y)
loss.backward()
optimizer.step()
optimizer.zero_grad()
print(loss.item())to:
for idx in range(ITER):
loss = Tensor([0.0])
for x1, y1 in zip(x, y):
pred = model(x1)
loss += tt.mse_loss(pred, y1)
loss.backward()
optimizer.step()
optimizer.zero_grad()
print(loss.item())Semantically its pretty much the same code, but on running the backward pass it gives the following error:
Traceback (most recent call last):
File "/Users/hedwig/Tinytorch/tiny_xor_net.py", line 52, in <module>
loss.backward()
File "/Users/hedwig/Tinytorch/tinytorch.py", line 262, in backward
grads = node._ctx.op.backward(node._ctx, node.grad)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/hedwig/Tinytorch/tinytorch.py", line 382, in backward
grad_y = transpose_last_axis(x.data) @ grad.data
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~
ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 1 is different from 2)
The error, I can hazard a guess is due to not broadcasting the grad and the parent data arrays before the backward pass for the MatMul Function. The shapes obtained in this case (before the matmul fails) is as follows:
Shape of grad.data: 1,)
Shape of x.data.T: (2,)
Shape of y.data.T: (1, 2)
I was toying around with the tiny xor example and I changed the training loop from the current loop:
to:
Semantically its pretty much the same code, but on running the backward pass it gives the following error:
The error, I can hazard a guess is due to not broadcasting the grad and the parent data arrays before the backward pass for the MatMul Function. The shapes obtained in this case (before the matmul fails) is as follows: