What is the purpose of DAGSTER_HOME/history/runs/index.db?

[tspencerdata]

I'm working on some ops to clean up DAGSTER_HOME on a sqlite backend. One of the biggest storage hogs is DAGSTER_HOME/history/runs/index.db. I can't find anything about this in the documentation- does anyone know what is the purpose of this db?

[Urus1201]

DAGSTER_HOME/history/runs/index.db is a SQLite index database that Dagster maintains to make run-history queries fast without scanning individual run files.

What it stores

When Dagster uses the default local SQLite storage backend, each run's event log is stored as a separate SQLite file under DAGSTER_HOME/storage/{run_id}/. The index.db keeps a centralized, queryable index of run-level metadata across all of those per-run files:

• Run IDs, statuses, and run tags
• Timestamps (start, end)
• Job/pipeline names and snapshot hashes
• Which runs are in which step / asset partitions

Without this index, Dagster would have to open every individual run file to answer a query like "give me all failed runs from last week" — which is O(n) in the number of runs. The index makes this O(1).

Why it gets large

For long-lived Dagster deployments with many runs, index.db accumulates entries forever unless you explicitly clean them up. It can become the biggest file in DAGSTER_HOME if you have thousands of historical runs, because SQLite does not automatically reclaim space even after rows are deleted (you need to run VACUUM).

How to clean it up safely

Dagster exposes a cleanup command specifically for this:

# Purge run history older than 7 days (keeps events too by default)
dagster run delete --older-than 7 --all

# Or use the Python API for finer control:
from dagster import DagsterInstance
from datetime import datetime, timedelta

with DagsterInstance.get() as instance:
    cutoff = datetime.utcnow() - timedelta(days=30)
    runs = instance.get_runs(
        filters=RunsFilter(statuses=[DagsterRunStatus.SUCCESS],
                           updated_before=cutoff)
    )
    for run in runs:
        instance.delete_run(run.run_id)

After deleting runs, reclaim disk space by running VACUUM on the SQLite file:

import sqlite3
with sqlite3.connect("path/to/DAGSTER_HOME/history/runs/index.db") as conn:
    conn.execute("VACUUM")

Production note

In production deployments (PostgreSQL or MySQL backend), this file does not exist — the equivalent data lives in the database. If index.db is getting large, it's often a signal that it's time to switch to a proper database backend, or set up a dagster instance retention policy.