AutoRag evaluator example help

[amantzanis]

Hello everyone! I have been reading up on AutoRAG and following the evaluator colab notebook as an example.

I have copied the first node from the config.yaml, I run the evaluator and after it generates embeddings the task fails.

I get no error message, it just fails.

This is my code, I hope it helps!

import os
path = './content/eli5_data'
os.makedirs(path, exist_ok=True)

import pandas as pd

qa_df = pd.read_parquet(path + '/qa_train.parquet')
sample_qa_df = qa_df.sample(5, random_state=42)
sample_qa_df = sample_qa_df.reset_index()
sample_qa_df.to_parquet(path + '/qa_sample.parquet')

%%writefile config.yaml

node_lines:

• node_line_name: node_line_1
  nodes:
  • node_type: retrieval
    top_k: 10
    strategy:
    metrics: [bleu, meteor, rouge]
    speed_threshold: 10
    modules:
    • module_type: bm25
    • module_type: vectordb
      embedding_model: huggingface_cointegrated_rubert_tiny2
    • module_type: hybrid_rrf
      target_modules: ('bm25', 'vectordb')
      rrf_k: [3, 5, 10]

make project folder

os.makedirs('./content/project_dir', exist_ok=True)

from autorag.evaluator import Evaluator
evaluator = Evaluator(qa_data_path='./content/eli5_data/qa_sample.parquet', corpus_data_path='./content/eli5_data/corpus.parquet',
project_dir='./content/project_dir')

evaluator.start_trial('./config.yaml')