Benchmarking the query adapter

[sinopeus]

[joachim-Heirbrant-SL]

Decided to use legalbenchrag for our ablation study. More specifically, we will start by obtaining metrics related to the chunk retrieval and reranking using the 'cuad' subset.

[StijnGoossens]

Options
A) No query adapter
B) Trained on synthetic dataset

• Change to standard vector search (insert_evals)
• Change LLM to gpt-4o (for question generation)

C) Trained on actual data (Split cuad dataset in train and test. Training happens on questions with positive chunks and negative chunks)

Experiments

• baseline vector search (evaluate on test set only)
• vector search + query adapter (option B) (evaluate on test set only)