GeospatialHallucination

[EMNLP 2025 Findings] Mitigating Geospatial Knowledge Hallucination in Large Language Models: Benchmarking and Dynamic Factuality Aligning

This is the official codebase for our EMNLP 2025 Findings paper. It provides a knowledge-graph-inspired benchmark for geospatial knowledge hallucination and a Dynamic Factuality Aligning (DynamicKTO) method to mitigate it.

News

• 2025-11 Paper presented at EMNLP 2025 in Suzhou, China.
• 2025-07 Paper released on arXiv: 2507.19586.

A Series of Urban Foundation Models

This work is part of a series of urban foundation models from FIB Lab, Tsinghua University:

Work	Venue	Focus
CityGPT	KDD 2025	Text-based urban spatial cognition (CityInstruction + SWFT + CityEval)
This work (GeospatialHallucination)	EMNLP 2025 Findings	Benchmarking & mitigating the geospatial hallucinations of the above LLMs
UrbanLLaVA	ICCV 2025	Multi-modal extension based on CityGPT (spatial reasoning + understanding)

In short: CityGPT injects city-scale knowledge to empower LLMs' urban spatial cognition; this work systematically benchmarks and eliminates the geospatial hallucinations these models still suffer from; and UrbanLLaVA further extends the line into the multi-modal dimension.

Core Idea

Overview of the benchmark construction and the Dynamic Factuality Aligning method.

LLMs possess rich geospatial knowledge that has been applied to tasks such as mobility prediction and social indicator prediction. However, they frequently produce geospatial hallucinations — incorrect or inconsistent representations of geospatial information (e.g., fake POIs, wrong addresses, non-existent roads) — which compromise their reliability. This work addresses the problem from both the evaluation and the mitigation sides:

1. A comprehensive evaluation framework built on structured geospatial knowledge graphs, providing a controlled benchmark that reveals how hallucinations manifest across 20 advanced LLMs.
2. Dynamic Factuality Aligning (DynamicKTO), a learning method based on Kahneman-Tversky Optimization (KTO) that dynamically aligns the model with factual geospatial knowledge, improving benchmark performance by over 29.6%.

Example: LLMs fabricate non-existent urban entities and confuse spatial relations & attributes.

Benchmark composition: three levels (Entity / Relation / Attribute) across three elements (POI / AOI / Road).

Hallucination Taxonomy

Inspired by Knowledge Graphs, geospatial hallucinations are classified into three first-level categories:

Level	Subcategory	Description
1. Entity	Entity Fabrication / Entity Omission	The model claims a non-existent or denies an existing POI / AOI / Road
2. Relation	Relation Fabrication / Relation Omission	The model mistakes the locate-at / near / connect relations between entities
3. Attribute	Attribute Confusion	The model confuses the address / category / land use / area / length attributes of an entity

The benchmark evaluates the three levels across three geospatial elements (POI, AOI, Road) with 14 fine-grained subcategories.

Key Results

• Benchmark: 14 fine-grained subcategories over 3 levels (Entity / Relation / Attribute) and 3 elements (POI / AOI / Road), covering 13 cities worldwide.
• Evaluation: systematic evaluation across 20 advanced LLMs reveals prevalent geospatial hallucinations, especially on long-tail knowledge.
• Mitigation: Dynamic Factuality Aligning (KTO) improves benchmark accuracy by over 29.6%, enhancing the trustworthiness of LLMs in geospatial knowledge and reasoning tasks.

Pipeline Overview

City Map (pycitysim)
    │  1_KG_construct            →  factual urban KG (POI / AOI / Road + address + relations)
    ▼
Factual KG + 2_hallucination_gen  →  real vs. hallucinated entity names
    │
    ├─ 3_*/4_tag_*_train_*        →  DPO preference pairs (entity / attribute / relation)
    ├─ 5_test_* + 6_simplifier    →  multiple-choice test sets
    │
    ▼
7_dpo2kto + train/DynamicKTO      →  KTO training (LLaMA-Factory)
eval/inference + eval/stats       →  LLM responses + hallucination statistics

Repository Structure

.
├── config.py                 # Global configuration (regions, maps, models, paths)
├── assets/                   # Figures (framework / benchmark / example)
├── data/                     # Data pipeline
│   ├── 1_KG_construct.py     # Construct the factual urban KG (POI/AOI/Road layers)
│   ├── 2_hallucination_gen.py# Generate hallucinated entity names with an LLM
│   ├── 3_train_*_construct.py# Build DPO training data (entity / attribute / relation)
│   ├── 4_tag_*_construct.py  # Build tagged DPO training data
│   ├── 5_test_*_construct.py # Build multiple-choice test sets
│   ├── 6_simplifier.py       # Simplify test sets (drop refusal options)
│   ├── 7_dpo2kto.py          # Convert DPO format to KTO format
│   ├── stats.py              # Token statistics utility
│   └── train/ train_tagged/ test/   # Output directories (DPO / tagged DPO / test data)
├── eval/
│   ├── inference.py          # Query LLM APIs or local vLLM models
│   └── stats.py              # Accuracy + hallucination-type statistics
└── train/
    └── DynamicKTO/
        └── LLaMA-Factory/    # Submodule (KTO training framework)

Installation

git clone https://github.com/tsinghua-fib-lab/GeospatialHallucination.git
cd GeospatialHallucination
# initialize the LLaMA-Factory submodule
git submodule update --init --recursive

pip install -r requirements.txt

Key dependencies include pycitysim (map data), shapely, pandas, numpy, tenacity, openai, tqdm, and optionally vllm for local inference.

Configuration

All global settings live in config.py. Before running any script, fill in:

• SERVING_IP — server for OSM reverse geocoding (Nominatim).
• SERVER_IP, LOCAL_MODEL_KEY — local model server.
• MONGODB_URI — MongoDB that hosts the city maps (used by pycitysim).
• REGION_EXP — region for the experiment (default Beijing; see MAP_DICT and REGION_BOUNDARY for the 13 supported cities).
• MIN_ROAD_LENGTH — minimum road length (meters) to consider.

Also set the relevant environment variables for LLM APIs, e.g. SiliconFlow_API_KEY, DeepInfra_API_KEY or OpenAI_API_KEY.

Usage

1. Build the Knowledge Graph

Set REGION_EXP and MONGODB_URI in config.py, then:

cd data
python 1_KG_construct.py

This writes <Region>_roads_basic.csv, <Region>_pois_basic.csv, <Region>_aois_basic.csv and <Region>_{pois,aois,roads}.json under resource/.

2. Generate hallucinated entity names

Update the source file paths and the API_TYPE / API_KEY in the script:

cd data
python 2_hallucination_gen.py

Outputs hallucinated_{poi,aoi,road}_names.txt.

3. Construct training data

Adjust the source/hallucinated-name paths and CITY in each script:

cd data
python 3_train_entity_construct.py
python 3_train_attribute_construct.py
python 3_train_relation_construct.py
# or the tagged variants
python 4_tag_train_entity_construct.py
python 4_tag_train_attribute_construct.py
python 4_tag_train_relation_construct.py

DPO-format data is written to train/ (or train_tagged/) as dpo_{entity,attribute,relation}.json.

4. Construct test data

cd data
python 5_test_entity_construct.py   # 1.x entity-level questions
# ... plus relation / attribute construction scripts (see data/)
python 6_simplifier.py              # remove "Cannot determine"/"Refuse to answer" options

Test sets are JSON files of multiple-choice questions saved under data/test/.

5. Convert DPO to KTO

cd data
python 7_dpo2kto.py

Produces kto_{entity,attribute,relation}.json in train_tagged/.

6. Train with DynamicKTO

Use the LLaMA-Factory submodule under train/DynamicKTO/ to run KTO training with the generated KTO datasets.

7. Evaluation

Set the test set directory and the model list in eval/inference.py, then:

cd eval
# test models served via DeepInfra / Siliconflow / OpenAI APIs or local vLLM
python inference.py
python inference.py --sampled   # run on a small sample (first 10 instances)

Responses are saved under eval/output/. Compute accuracy and hallucination-type statistics:

python stats.py

This writes stats.json, testset_stats.json, hallucination_first_level.json, hallucination_second_level.json, hallucination_type.json and an output_stats.xlsx report.

Supported Cities

Beijing, Shanghai, Mumbai, Tokyo, London, Paris, Moscow, New York, San Francisco, São Paulo, Nairobi, Cape Town, Sydney.

Supported Model Providers

OpenAI, DeepInfra, Siliconflow, DeepBricks (via OpenAI-compatible API) and local models served through vLLM. The evaluated models include Qwen2, InternLM2.5, Mistral, Mixtral, Llama-3, Gemma-2, DeepSeek, GPT-3.5/4, etc. (see config.py).

Citation

If you find this work useful, please cite:

@inproceedings{wang-etal-2025-mitigating,
    title = "Mitigating Geospatial Knowledge Hallucination in Large Language Models: Benchmarking and Dynamic Factuality Aligning",
    author = "Wang, Shengyuan  and Feng, Jie  and Liu, Tianhui  and Pei, Dan  and Li, Yong",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.findings-emnlp.45/",
    doi = "10.18653/v1/2025.findings-emnlp.45",
    pages = "870--888"
}

License

This project is released under the MIT License. Copyright (c) 2025 FIB Lab, Tsinghua University.