Investigate low accuracy in vgg19_bn (96.97)

[nemanjagrujic]

This model now crashes so its N/A

[nemanjagrujic]

This model throws runtime error:

RuntimeError: TT_THROW @ /home/ubuntu/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception
info:
Out of Memory: Not enough space to allocate 8257536 B L1 buffer across 64 banks, where each bank needs to store 129024 B

Problem is in convolution lowering where this method is called:

def insert_sharded_nxc_to_ncx(g, output_tensor, output_shape):
    output_tensor = g.call_function(ttnn.sharded_to_interleaved, (output_tensor, TtnnL1MemoryConfig()))
    target_shape = (output_shape[0], *output_shape[2:], output_shape[1])
    output_tensor = g.call_function(ttnn.reshape, (output_tensor, target_shape))
    target_permute = [0, len(output_shape) - 1] + list(range(1, len(output_shape) - 1))
    return g.call_function(ttnn.permute, (output_tensor, target_permute))

Getting large tensor to L1 later causes the crash.

[nemanjagrujic]

Here is the ticket #846

After this is fixed by manually switching from L1 to DRAM. This model still show low PCC of 0.9697369527295003.

[nemanjagrujic]

The cause of low PCC (0.9697) is conv2d. When we fall back convolution to torch implementation PCC goes up to 0.9978.

But it seams that there is no apparent error in some specific convolution. PCC just gradually, slowly, drops down as computation approaches end of the model.

Each specific run of convolution has high PCC when fed the same input.

@amalbasaTT @ayerofieiev-tt @npetrovic-tenstorrent