Investigate accuracy issues in ViLT (87.8)

[amalbasaTT]

https://github.com/tenstorrent/pytorch2.0_ttnn/blob/main/docs/models/ViLT/input_variations.md

[npetrovic-tenstorrent]

After passing four test_accuracy() calls,
1.0
1.0
0.9999971531701141
0.9999286433209008

and fail with error on convolution:
convolution = aten.convolution.default(arg3_1, arg1_1, arg2_1, [32, 32], [0, 0], [1, 1], False, [0, 0], 1, )

Error:
File "/home/ubuntu/pytorch2.0_ttnn/tests/autogen_accuracy_tests/ViLT_code.py", line 344, in
forward(*inputs)
File "/home/ubuntu/pytorch2.0_ttnn/tests/autogen_accuracy_tests/ViLT_code.py", line 162, in forward
convolution = aten.convolution.default(arg3_1, arg1_1, arg2_1, [32, 32], [0, 0], [1, 1], False, [0, 0], 1, )
File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_ops.py", line 513, in call
return self._op(*args, **(kwargs or {}))
RuntimeError: weight should have at least three dimensions

[npetrovic-tenstorrent]

After passing four test_accuracy() calls, 1.0 1.0 0.9999971531701141 0.9999286433209008

and fail with error on convolution: convolution = aten.convolution.default(arg3_1, arg1_1, arg2_1, [32, 32], [0, 0], [1, 1], False, [0, 0], 1, )

Error: File "/home/ubuntu/pytorch2.0_ttnn/tests/autogen_accuracy_tests/ViLT_code.py", line 344, in forward(*inputs) File "/home/ubuntu/pytorch2.0_ttnn/tests/autogen_accuracy_tests/ViLT_code.py", line 162, in forward convolution = aten.convolution.default(arg3_1, arg1_1, arg2_1, [32, 32], [0, 0], [1, 1], False, [0, 0], 1, ) File "/home/ubuntu/tt-metal/python_env/lib/python3.8/site-packages/torch/_ops.py", line 513, in call return self._op(*args, **(kwargs or {})) RuntimeError: weight should have at least three dimensions

@kevinwuTT Please take a look on this. When we run without generator script, it seems to work with accuracy reaching the end:
INFO - Accuracy calculated: 0.878645550989681.

I run it as pytest tests/models/vilt/test_vilt.py -s &> /tmp/log.log

[kevinwuTT]

The issue is that the inputs to the 2nd forward function is not the same as the first.

In the graph generated by the compiler:

arg1_1: {'val': FakeTensor(..., size=(768, 3, 32, 32)

convolution_default = torch.ops.aten.convolution.default(arg3_1, arg1_1, arg2_1, [32, 32], [0, 0], [1, 1], False, [0, 0], 1)

But if I print the shape of arg1_1 in the generated ViLT_code.py,

torch.Size([2, 768])

I'll have to rethink if splitting the forward graphs is the better solution.

[npetrovic-tenstorrent]

After running on branch with fixed problem with multiple forward functions - kw/gen_acc_tests_for_profiling it returns low PCC due to stack and expand operations:

Traceback (most recent call last):
  File "/home/ubuntu/pytorch2.0_ttnn/tests/export_code/accuracy/ViLT_code.py", line 349, in <module>
    stack = forward_4(arg0_1, device)
  File "/home/ubuntu/pytorch2.0_ttnn/tests/export_code/accuracy/ViLT_code.py", line 304, in forward_4
    test_accuracy(expand, ttnn_prefix_repeat)
  File "/home/ubuntu/pytorch2.0_ttnn/tests/export_code/accuracy/ViLT_code.py", line 117, in test_accuracy
    return mre, assert_with_pcc(expected, actual, pcc = 0.90)
  File "/home/ubuntu/pytorch2.0_ttnn/tests/export_code/accuracy/ViLT_code.py", line 96, in assert_with_pcc
    assert pcc_passed, construct_pcc_assert_message(pcc_message, expected_pytorch_result, actual_pytorch_result)
AssertionError: 0.4694027561011089

After fallback to aten for both stack and expand, PCC reaches 1.0.

To reproduce in pytorch2.0_ttnn/torch_ttnn/passes/lowering/to_tt_pass.py add return None
after:

1)  if node.target == torch.ops.aten.stack.default:
2)  if node.target == torch.ops.aten.expand.default:

The issue is that the inputs to the 2nd forward function is not the same as the first.

In the graph generated by the compiler:

arg1_1: {'val': FakeTensor(..., size=(768, 3, 32, 32)

convolution_default = torch.ops.aten.convolution.default(arg3_1, arg1_1, arg2_1, [32, 32], [0, 0], [1, 1], False, [0, 0], 1)

But if I print the shape of arg1_1 in the generated ViLT_code.py,

torch.Size([2, 768])

I'll have to rethink if splitting the forward graphs is the better solution.

[npetrovic-tenstorrent]

After further investigation, we have identified that the issue causing low accuracy is related to the same problem as described here, affecting other models such as CLiP, resulting from problem of ttnn.repeat that reflects expand operation:
tenstorrent/tt-metal#19776