Leiden University Paper Map

This repository contains the workflow used to build an interactive map of papers affiliated with Leiden University. It extracts Leiden papers from the CWTS OpenAlex BigQuery dataset, generates SPECTER2 embeddings in Google Colab, and displays UMAP coordinates in a static WebGL viewer.

This project demonstrates the power of science maps that visualize every individual paper. By moving from cluster-level to paper-level visualization, it becomes possible to capture subtleties that clusters can conceal, such as areas of high paper density.

Live demo

Explore the Leiden University Paper Map

1. Extract Leiden papers

Run get_leiden_papers.sql in BigQuery and save its result as a table. The query selects works affiliated with OpenAlex institution I121797337 (Leiden University) and produces:

id,title,abstract

2. Generate embeddings

Open leiden_embeddings.ipynb in Google Colab and edit the config cell:

BQ_TABLE = 'YOUR_PROJECT.YOUR_DATASET.leiden_papers'
BILLED_PROJECT = 'YOUR_PROJECT'
GCS_OUTPUT = 'gs://YOUR_BUCKET/leiden_embeddings/'

The notebook writes SPECTER2 embeddings as Parquet parts.

The notebook embeds papers that have both a title and abstract, saves resumable Parquet checkpoints to Google Drive, and uploads the parts to Cloud Storage. Load those Parquet parts into the BigQuery table configured as EMBEDDINGS_TABLE in the next notebook.

3. Generate UMAP coordinates

Open leiden_umap_coordinates.ipynb in Google Colab and configure the billing project, embeddings table, coordinates table, and Cloud Storage output folder. It generates the five viewer layouts (n_neighbors 15, 25, 50, 100, and 200), writes each result to Cloud Storage, and loads it into BigQuery.

4. Prepare viewer data

The cloud prototype expects these files:

google_cloud_prototype/data/paperdata.csv
google_cloud_prototype/data/coordinates_15.csv
google_cloud_prototype/data/coordinates_25.csv
google_cloud_prototype/data/coordinates_50.csv
google_cloud_prototype/data/coordinates_100.csv
google_cloud_prototype/data/coordinates_200.csv

paperdata.csv columns:

work_id,publication_year,title,abstract

Coordinate CSV columns:

work_id,x,y

The work_id values must match between metadata and coordinate files.

Run export_viewer_data.sql in BigQuery after replacing its project, dataset, and bucket placeholders. Download the resulting metadata and coordinate CSV shards, then prepare the viewer files:

python tools\prepare_viewer_data.py `
  --metadata .\exports\metadata-*.csv `
  --coordinates .\exports\coordinates-*.csv

The repository includes a ten-paper synthetic sample, so both viewers work immediately after cloning. Replace the sample files in google_cloud_prototype/data/ with the generated Leiden files for deployment.

5. Preview or publish

Preview either viewer with a local static server:

python -m http.server 8000

Then open http://localhost:8000/viewer/ for the single-file viewer or http://localhost:8000/google_cloud_prototype/ for the preloaded prototype.

To publish the prototype, upload google_cloud_prototype/ to a static host. The scripts and instructions in cloud/ cover Google Cloud Storage.

Implementation details and operational tradeoffs are documented in TECHNICAL_NOTES.md.

License

Released under the MIT License. See LICENSE.