Multi-hop Question Generation

This project generates and evaluates multi-hop questions from knowledge graphs stored in the current GraphRAG Parquet artifact format.

Input datasets

Run project commands from the repository root. Each dataset is a directory under ./input containing these four artifacts:

input/
├── wiki-1k/
│   ├── documents.parquet
│   ├── entities.parquet
│   ├── relationships.parquet
│   └── text_units.parquet
├── war_and_peace/
├── origin_of_the_species/
└── benjamin_franklin/

The default dataset is ./input/wiki-1k. Select another dataset with the global --input-dir option, for example:

python mhqg_cli.py --input-dir ./input/war_and_peace generate --count 10
python connected_subgraphs.py --input-dir ./input/benjamin_franklin --sample 5
python provenance_context.py \
  --input-dir ./input/origin_of_the_species \
  "CHARLES DARWIN" "H.M.S. BEAGLE"

The expected current-format columns are:

Artifact	Required columns
documents.parquet	id, title, text
entities.parquet	id, title, type, description, text_unit_ids
relationships.parquet	id, source, target, description, weight, text_unit_ids
text_units.parquet	id, text, document_id

entity_ids, relationship_ids, and other enrichment columns on text units are optional. Provenance lookup falls back to the text_unit_ids stored on entities and relationships when those reverse-link columns are absent.

Creating or importing an index

With a current GraphRAG release, initialize and index a workspace as usual:

python -m venv .venv-graphrag
source .venv-graphrag/bin/activate
pip install graphrag

graphrag init --root /path/to/graphrag-workspace
# Add source documents and configure .env/settings.yaml.
graphrag index --root /path/to/graphrag-workspace

Copy the four current artifacts from the GraphRAG output directory into one dataset directory:

mkdir -p ./input/my-dataset
cp /path/to/graphrag-output/{documents,entities,relationships,text_units}.parquet \
  ./input/my-dataset/

The built-in index command also writes the current filenames and schema:

python mhqg_cli.py --input-dir ./input/my-dataset index source-texts/*.txt

GraphRAG indexing and this project's index command can invoke model APIs and may be expensive, so start with a small corpus.

API key and generation checks

Put your OpenAI API key on the first line of api_key.txt in the repository root, replacing the placeholder already in that file. The key file is ignored by Git so it cannot be committed accidentally.

All OpenAI generation, evaluation, and indexing commands default to gpt-5.6-sol with reasoning effort set to none for structure extraction and single-hop generation. The final composition prompt uses low reasoning. Model names can still be overridden with the existing CLI model options.

Review (including catalogue approval and LLM self-review) and graph ambiguity analysis are disabled by default. Worked-example generation is also disabled; new structures use zero-shot question generation. Generate questions with:

python mhqg_cli.py generate --count 10 --output ./output/questions.json

Enable automatic worked-example generation for new structures explicitly:

python mhqg_cli.py generate --example-generation

Existing catalogue or general examples can still be used when automatic example generation is disabled. Review-enabled generation continues to require approved examples.

Provenance text units are omitted from the final composition prompt by default; single-hop generation still uses its relationship-linked evidence. Restore the full text-unit context for composition explicitly when needed:

python mhqg_cli.py generate --composition-text-units

Text units remain in the generated question record for provenance regardless of this option.

Add --usage to save token accounting separately from the generated questions:

python mhqg_cli.py generate \
  --count 1 \
  --output ./output/questions.json \
  --usage

This leaves questions.json unchanged and writes per-call and aggregate token counts to ./output/questions.usage.json. If --output is omitted, questions are printed to stdout and usage is written to ./output/usage.json.

When --output is provided, each accepted question is checkpointed atomically instead of waiting for the whole batch to finish. Resume an interrupted batch by repeating the command with --resume:

python mhqg_cli.py generate \
  --count 100 \
  --output ./output/questions.json \
  --resume

With --resume, --count is the desired total in the output file. For example, if the file already contains 37 questions, the command above generates the remaining 63. Resume requires --output; without --resume, an existing output file starts a new batch and is replaced.

Add --prompts to save every prompt sent to the generator and evaluator LLMs:

python mhqg_cli.py generate \
  --count 1 \
  --output ./output/questions.json \
  --prompts

This writes ./output/questions.prompts.json, or ./output/prompts.json when --output is omitted. Prompt files can be large and may contain source text, so handle them with the same access controls as the input corpus.

They can be enabled explicitly when needed:

python mhqg_cli.py generate --enable-review --enable-ambiguity-analysis