Skip to content

README.md

Latest commit

 

History

History
54 lines (45 loc) · 2.46 KB

README.md

File metadata and controls

54 lines (45 loc) · 2.46 KB
Raw

Matching domain experts by training from scratch on domain knowledge

1. To work with the repo locally:

git clone git@github.com:braingpt-lovelab/matching_experts.git --recursive

2. Structure

Training related scripts and files are in model_training/; post-training analyses scripts and results are in analyses/.

3. Training

cd model_training

  1. To build the neuro-tokenizer: python tokenizer.py
  2. To train a GPT-2 using a specific configuration: bash launch_training.sh. Configurations are in configs/
  3. Make sure to supply wandb info in the config json.
  4. Accelerate config: accel_config.yaml

4. Training data

Domain-specific Neuroscience training data can be found here: https://huggingface.co/datasets/BrainGPT/train_valid_split_pmc_neuroscience_2002-2022_filtered_subset

5. Reproduce analyses from scratch:

cd analyses

  1. Run inference with GPT-2 variants on BrainBench test cases: python run_choice.py
  2. Produce token analysis intermediate results: python common_and_unique.py
  3. Call GPT-4 to identify neuroscience terms in GPT-2 pretrained tokenizer and neuro-tokenizer vocab: python neuro_term_tagging.py

6. Plot figures in the paper

cd analyses

  • Fig 1: python model_vs_human.py
  • Fig 2: python token_analyses.py
  • Fig 3: python tokenization_viz.py

7. Access raw BrainBench results of GPT-2 variants

cd analyses/model_results

Variant Training Data Tokenizer Raw Results Directory
Untrained - - pretrained gpt2_init/
Pretrained from scratch WebText pretrained gpt2/
Scratch from scratch neuroscience pretrained gpt2_scratch/
Finetuned (from pretrained) finetune neuroscience pretrained finetune_gpt2/
Scratch (Neuro tokenizer) from scratch neuroscience custom gpt2_scratch_neuro_tokenizer/

Attribution

@misc{luo2024matching,
      title={Matching domain experts by training from scratch on domain knowledge}, 
      author={Xiaoliang Luo and Guangzhi Sun and Bradley C. Love},
      year={2024},
      eprint={2405.09395},
      archivePrefix={arXiv},
      primaryClass={q-bio.NC}
}