Requirements

• make sure you install and configure Java and Tuffy.
• Please follow this instruction to setup the local postgreSQL server.
• Create an empty folder llm/ under tuffy-0.3-jun2014/samples/

Dataset

• Pleaser refer to /data for links to download the first-order logic rules and synthesized MLNs.
• Synthesized reasoning tasks:

python3 synthesize_tasks.py \ 
    --rules-df rules_data.pkl \
    --tuffy-jar tuffy-0.3-jun2014/tuffy.jar \
    --tuffy-workdir tuffy-0.3-jun2014/samples/llm \ 
    --nfact $NUM_FACT --nrule $NUM_RULE --rule-wght-schema uniform --task {qa | resolve}

Inference & Eval

• Donwload the LLM you want to evaluate (e.g. phi-4) to LLM_DIR.
• Run inference with the synthesized task:

python run_{qa | resolve}.py \
            --train_data_path $train_data_path \
            --test_data_path $test_data_path \
            --num_demo 1 --prompt_method {icl | cot} \
            --ckpt_dir $LLM_DIR