Installation | The increase usage of LLMs in Scientific Papers | Validation | Data availability | Usage
A simple and effective method for estimating the fraction of text in a large corpus that has been substantially modified or generated by AI:
- Distributional GPT Detection. In contrast with instance-level detection, this framework focuses on population-level estimates. We demonstrate how to estimate the proportion of content in a given corpus that has been generated or significantly modified by AI, without the need to perform inference on any individual instance.
- Easy Deployment and Usage. Our code can quickly estimate the distribution of both AI- and human-generated text without an expensive model training procedure. Using these estimated text distributions, we can accurately predict the fraction of text in a large corpus that has been substantially modified or generated by AI.
This repository was built using Python 3.8.19, but should be backwards compatible with any Python >= 3.8. This repository was developed and has been thoroughly tested with pandas 2.0.3, numpy 1.24.4, pyarrow 15.0.2, fastparquet 2024.2.0, scipy 1.10.1, and ipykernel 6.29.4.
You can install this package locally via an editable installation or the provided yml file (the installation via yml file should be completed in 1-2 minutes):
git clone https://github.com/Weixin-Liang/Mapping-the-Increasing-Use-of-LLMs-in-Scientific-Papers.git
cd Mapping-the-Increasing-Use-of-LLMs-in-Scientific-Papers
conda env create -f environment.ymlIf you run into any problems during the installation process, please file a GitHub Issue.
We provide the code and data that could be used to reproduce the main Figure 1 of our paper (see increasing_temporal.ipynb).
This result reveals a steady increase in the fraction of LLM-modified content (
Once installed, estimating distributions and running inference is easy. We provide pre-mixed human written text and AI generated text in the data/validation folder for demo purposes, with alpha indicating the fraction of ai-generated text. For each inference call, the expected run time should be 1-2 minutes and the expected output should not deviate from the ground truth by more than 3%.
from src.MLE import MLE
# load the word occurrences frequency into our framework
model=MLE(f"distribution/CS.parquet")
# validate our method using mixed corpus with known ground truth alpha
for alpha in [0,0.025,0.05,0.075,0.1,0.125,0.15,0.175,0.2,0.225,0.25]:
estimated,ci=model.inference(f"data/validation_data/CS/ground_truth_alpha_{alpha}.parquet")
error=abs(estimated-alpha)
print(f"{'Ground Truth':>10},{'Prediction':>10},{'CI':>10},{'Error':>10}")
print(f"{alpha:10.3f},{estimated:10.3f},{ci:10.3f},{error:10.3f}")For a complete demonstration, check out validation.ipynb.
We make available the datasets used for our primary analysis of LLM adoption in scientific papers. For each month, up to 2000 papers are included. The data is structured by venue, with files named {year}_{month}.parquet (e.g., 2021_1.parquet for January 2021 through 2024_9.parquet for September 2024). We also provide the word frequency distribution file used to estimate the LLM usage in abstract writing (one for each venue) and our validation data for Computer Science, Mathematics, Physics, Statistics, and Electrical Engineering and Systems Science. If you are interested in the complete data, please check out the following links:
For arXiv, https://www.kaggle.com/datasets/Cornell-University/arxiv; For bioRxiv, https://github.com/nicholasmfraser/rbiorxiv; For Nature portfolio, https://www.nature.com/nature-portfolio. If you want the main text of arXiv papers, please checkout arxiv.py for latex source downloading and doc2json for latex parsing. The main text for Nature portfolio can be accessed online.
This repository includes the arXiv abstracts used for the analysis in our paper. However, our framework can easily be extended to other domains of your choice. It requires two datasets--one consisting of documents written entirely by humans, and another consisting of documents written entirely by AI--which are used to estimate the distribution of human- and AI-generated text in your chosen domain. Using these estimates, you can perform inference on a target dataset with an unknown fraction of AI-generated content.
The function estimate_text_distribution in src.estimation requires two file path as input to indicate where human- and AI-generated text are stored. The two input files should be .parquet format. For human-generated text, our provided function need the input parquet file to have a column named as human_sentence and required data to be organized as one tokenized sentence(a list of word) per row. Similarly, for ai-generated text, our provided function need a column named as ai_sentence and required data to be organized as one tokenized sentence(a list of word) per row.
example of human-generated data:
| human_sentence |
|---|
| ["This", "is", "an", "example"] |
| ["Another", "sentence", "for", "you"] |
example of ai-generated data:
| ai_sentence |
|---|
| ["This", "is", "an", "example"] |
| ["Another", "sentence", "for", "you"] |
For inference on target dataset, the function inference in class MLE also requires a file path as input. It also need the input parquet file to have a column named as inference_sentence and required data to be organized as one tokenized sentence(a list of word) per row.
example of inference data:
| inference_sentence |
|---|
| ["This", "is", "an", "example"] |
| ["Another", "sentence", "for", "you"] |
Note that we provide our tokenize function in tokenize_demo.ipynb for reference.
Below is a high-level overview of the repository/project file-tree:
data/- Data source consisting of arXiv abstract data across five main fields (Physics, Mathematics, Computer Science, Statistics, and Electrical Engineering and Systems Science). Theinference_datafolder contains corpora that could reproduce our main Figure 1 (the increasing usage of LLM in abstract writing). The data is organized such that for each venue, files are named in the format{year}_{month}.parquet, ranging from January 2021 (2021_1.parquet) to September 2024 (2024_9.parquet). Each parquet file contains up to 2000 papers. Thevalidation_datafolder contains corpora with mixed AI-generated and human-written data, whose ground truth portion is known. This is used to validate the effectiveness of our framework (Figure 3a-e). Details on the data can be found in our paper.distribution/- Folder to save the distribution parquet generated by theestimate_text_distributionfunction for demo purposes. We provide the word frequency distribution file used to estimate the LLM usage in abstract writing (one for each venue).src/- Package source providing core utilities for distribution estimation, framework loading, data inference, etc.LICENSE- All code is made available under the MIT License; happy hacking!increasing_temporal.ipynb- Reproduction of our main Figure 1 (the increasing temporal of LLM usage in abstract writing).validation.ipynb- Demonstration of our framework on arXiv abstracts across five main fields. This includes the validation process on manually mixed data with a known ground truth portion of AI-written text (See Figure 3).tokenize_demo.ipynb- Demonstration of how we tokenize arXiv abstracts, including the tokenize function. Here we use the spaCy(https://spacy.io/) library, but other tools like nltk are also feasible. You may need to modify the function in your own cases.environment.yml- Full project configuration details (including dependencies), as well as tool configurations.README.md- You are here!