QUEST-BigText

This repository contain Code and Evaluation data of QUEST-BigText Peoject.

Quick Run & Benchmark

• Quick Run: We take QUEST-BigText(cosine:0.75, jaccard:0.25, question: "who directed braveheart and play ed in mad max?") as an example. All parameters could be adjusted in config.yml.

  Open the /data/QUEST-BigText/QUEST-BigText_ALL/0_QUEST_BigText_Final/3_QUEST_BigText/Code folder, and use the command:

    source /home/jingjing/my_virtualenv_dir/QUEST_Lucene_Bigtext/bin/activate
    
    export CLASSPATH=stanford-corenlp-full-2018-10-05/stanford-corenlp-3.9.2.jar:stanford-corenlp-full-2018-10-05/stanford-corenlp-3.9.2-models.jar
    		
    spark-submit \
    --conf spark.local.dir=/data/jingjing/tmp \
    process_single_question_pyspark.py "who directed braveheart and play ed in mad max?" 'demo' "/data/QUEST-BigText/QUEST-BigText_ALL/EvaluationData/CQ-W_QA/who directed braveheart and play ed in mad max?/demo"
    
    deactivate
  

  The results are shown in the results folder.

• Benchmark We take QUEST-BigText(cosine:0.5, jaccard:0.25) as an example.

  1. open the /data/QUEST-BigText/QUEST-BigText_ALL/EvaluationData/EvaluationCode folder,

  2. adjust the config.yml, we use parameters:

     testVersion: subgraph_HP_QUEST_Lucene_BigText_v1.3_n50_cos050_jac025_1 QuestionFilePath: ./0_questions/Subgraph_ContainAns_Question_111.txt

  3. run the code using the command: python2 Benchmarks.py

  4. using the command: cat ./Benchmarks/subgraph_HP_QUEST_Lucene_BigText_v1.3_n50_cos050_jac025_1_containAns_111.txt to find the result subgraph_HP_QUEST_Lucene_BigText_v1.3_n50_cos050_jac025_1_containAns_111.txt at Benchmarks folder.

Details

- Folders

0. 0_QUEST_BigText_Final contains all codes of our experiments.

    0_QUEST_Lucene: QUEST using Lucene txt only
    1_QUEST_Google: QUEST using google search
    2_QUEST_Wiki: QUEST using wiki website
    3_QUEST_BigText: QUEST-BigText 
   

1. EvaluationData

   CQ-W_QA.json: a file which contains all questions and corresponding golden answer.

   CQ-W_QA: a folder contains all questions results.

    0. 1Questions: a folder contains all questions
    1. question folder:
      for example, the "in what movie did nicolas cage and john travolta star in?" folder includes:
    	0. top 10 txt files: top 10 related documents from Lucene
    	1. demo folder: contains all prepared files of the question
    	2. subgraph folder: contains subgraph (vertices and edges) of the question
    	3. other folder: contains result-related files of the question.
   

   EvaluationCode: a folder contains all evaluation code

    0. AnsNo_QuasiGraphSize.py: check the total answer number, node size and edge size of questions.
    	eg. 399|1630|545406|which films were starred by julia roberts and richard gere?
    1. BenchmarkMulti.py: get benchmark(MRR, Precision@1, Hit@5) of multiple questions files(people, movie, place, others, language, music)
    2. Benchmarks.py: get benchmark(MRR, Precision@1, Hit@5) of only one question file
    3. BigText_AnsInSubGraph_new.py: Check if answer is in subgraph
    eg.
    	== ]Mention Node] ++ GoldenAns:]  runaway bride]  3|3  ]which films were starred by julia roberts and richard gere?   ]GoldenANS_ALL:]   | runaway bride | pretty woman]Mention/Entity_All]*pretty woman|pretty woman**runaway bride|runaway bride case**runaway bride|runaway bride (1999 film)*
    	
    	==: mention node or entity node contains golden answer.
    	++: mention node and entity node both contain golden answer
    	runaway bride: golden answer
    	3|3: 3 mention nides and 3 eneity nodes match to golden answer.
    	which films were starred by julia roberts and richard gere?: questions
    	GoldenANS_ALL:]   | runaway bride | pretty woman]: all of golden answers
    	Mention/Entity_All]*pretty woman|pretty woman**runaway bride|runaway bride case**runaway bride|runaway bride (1999 film)* : all of related mentions and entity nodes
    	
    4. CheckAnsLocation.py: get the location of answer
    eg.
    	corner_dist_wt  |  corner_dist  |  node_wt  |  tree_cost  |  tree_count  |  Question
    	45  |  52  |  45  |  59  |  48  |  which films were starred by julia roberts and richard gere?
    	
    5. CheckError.py: check if saved result is correct. "==" is correct, "xx" is wrong
    6. CheckQuestions.py: check if questions are answered using specific parameters
    7. Demo_subGraphSel_time.py: check the time of getting prepared files
    8. RunTime_v1.3_demo.py: check the runtime of QUEST-BigText
    9. subGraphSize.py: get the subgraph size of each question
   

   subGraphCode: a folder contain all subgraph-related codes

    using lauch_test_subgraph1.sh and test_subGraphGeneration1.py to generate subgraph
    using lauch_test_subgraph2_HP_NE_Mention_SPO.sh and test_subGraphGeneration2_HP_NE_Mention_SPO.py to generate prepared files
   

   0TestResult: a folder to receive all test result

2. 1_JavaCode a folder contains Java code for Lucene

3. LuceneData a folder contains all lucene related-data.

- Code

0. Setup (On BigText Platform)

Before running the QUEST-BigText, we need to create an virtual environment and install required libraries (This steps could be ignored if the environment is set already).

0. install all required libraries of pyspark

1. open .bashrc and add:

    export PYSPARK_PYTHON=python2
    export PYSPARK_DRIVER_PYTHON=python2
   

2. install virtual environment library pip2 install virtualenv

3. create a virtual environment virtualenv --python=python2 /home/jingjing/my_virtualenv_dir/QUEST_Lucene_Bigtext

4. active the virtual environment source /home/jingjing/my_virtualenv_dir/QUEST_Lucene_Bigtext/bin/activate

5. install libraries in this environment

    pip install requests
    pip install BeautifulSoup
    pip install networkx
    pip install numpy
    pip install stanfordcorenlp
    pip install gensim
    pip install nltk==3.4.5
   

6. deactivate could deactive the virtual environment

1. Run

The code of QUEST-BigText could be found at the /QUEST-BigText_ALL/0_QUEST_BigText_Final/3_QUEST_BigText/Code folder. We could run it by testing only one question or testing all questions.

• Test only one question

  We take QUEST-BigText(cosine:0.75, jaccard:0.25, question: "who directed braveheart and play ed in mad max?") as an example. All parameters could be adjusted in config.yml.

  Open the /data/QUEST-BigText/QUEST-BigText_ALL/0_QUEST_BigText_Final/3_QUEST_BigText/Code folder, and use the command:

    source /home/jingjing/my_virtualenv_dir/QUEST_Lucene_Bigtext/bin/activate
    
    export CLASSPATH=stanford-corenlp-full-2018-10-05/stanford-corenlp-3.9.2.jar:stanford-corenlp-full-2018-10-05/stanford-corenlp-3.9.2-models.jar
    		
    spark-submit \
    --conf spark.local.dir=/data/jingjing/tmp \
    process_single_question_pyspark.py "who directed braveheart and play ed in mad max?" 'demo' "/data/QUEST-BigText/QUEST-BigText_ALL/EvaluationData/CQ-W_QA/who directed braveheart and play ed in mad max?/demo"
    
    deactivate
  

  The results are shown in the results folder.

• Test all questions

  we take questions from "/QUEST-BigText_ALL/EvaluationData/CQ-W_QA/1Questions/0Test" as example 0. open /data/QUEST-BigText/QUEST-BigText_ALL/0_QUEST_BigText_Final/3_QUEST_BigText/Code folder

  1. adjust parameters in config.yml

      Cosine_threshold: 0.75
      Jaccard_threshold: 0.25
     

  2. adjust parameters in Run_QUEST_Lucene_BigText_V1.3.sh

      # the code version(version:QUEST-BigText, Parameter: cosine:0.75 | jaccard:0.25 | n = 50 | question file: 0Test)
      TestVersion="QUEST-BigText_n50_cos075_jac025_0Test"
      
      # "0Test" file contains given questions
      cat "${selectedFileDir}/1Questions/0Test" | while read LINE
     

  3. use the following command: nohup bash Run_QUEST_Lucene_BigText_V1.3.sh > ./testLog.txt &

     we could check the status by the command: top -c

  4. find results at "/data/QUEST-BigText/QUEST-BigText_ALL/EvaluationData/0TestResult" folder

2. Evaluation

1. adjust config.yml in QUEST-BigText_ALL/EvaluationData/EvaluationCode/ folder
2. use command: python2 Benchmarks.py
3. find the result in Benchmarks folder

Citation

If you find this repo useful, please cite our paper.

@inproceedings{xu2023bigtext,
  title={Bigtext-qa: question answering over a large-scale hybrid knowledge graph},
  author={Xu, Jingjing and Biryukov, Maria and Theobald, Martin and Venugopal, Vinu Ellampallil},
  booktitle={International Conference on Big Data Technologies and Applications},
  pages={33--48},
  year={2023},
  organization={Springer}
}

Acknowledgement

This project is supported by the Luxembourg National Research Fund (Fonds National de la Recherche - FNR), Grant ID: 15748747