Issue84/aksh cross chunk attention

[akshsabherwal]

Relevant issues

• This closes [ChatLSE] Explore Cross-Chunk attention mechanisms  #84

Description

• Create src/python/chatlse/cross_chunk_attention.py where the ShiftedCrossChunkAttention (SCCA) class is defined
• Edit src/python/chatlse/crawler.py to ensure that embed_function()incorporates the additional text processing

The basic mechanism of SCCA is as follows:

1. Calculate embeddings of each chunk for each document (using existing utilities from embeddings.py)
2. Shape embeddings into a tensor of required dimension
3. Shift embeddings across chunks (e.g. if chunk 1 has embedding A, chunk 2 has embedding B, and chunk 3 has embedding C, then the shift results in chunk 1 having embedding C, chunk 2 having embedding A, and chunk 3 having embedding B)
4. Break up each embedding into 8 different embeddings (called 'heads') and perform attention calculations on each of these heads
5. Reshape the tensor back to its original dimension, resulting in each chunk now having an attended embedding

How to test

1. Run the crawler
2. Run the app
3. Ask it any question you asked it in v0.1 that you had issues with, and see if there is any improvement at all.