Skip to content

intro.md

Latest commit

 

History

History
258 lines (185 loc) · 11.4 KB

intro.md

File metadata and controls

258 lines (185 loc) · 11.4 KB

Get Started

Overview

The Demographic Microsimulator (DEMOS) is an agent-based simulation framework used to model the evolution of population demographic characteristics and lifecycle events, such as education attainment, marital status, and other key transitions. DEMOS modules are designed to capture the interdependencies between short-term and long-term lifecycle events, which are often influential in downstream transportation and land-use modeling.

A key feature of DEMOS is its ability to track changes in an agent’s demographic status from year t to year t + 1. This structure allows the model to evolve populations over any user-defined time horizon. As a result, DEMOS is well suited for analyzing medium- and long-term transportation-related decisions, including household vehicle transactions (e.g., purchasing, selling, or replacing vehicles) and work location choices. Core features of DEMOS include the modeling of more than ten lifecycle events, behaviorally realistic patterns informed by long-running panel data, explicit representation of interdependencies among lifecycle processes, and a flexible, modular simulation architecture.

A technical memorandum describing DEMOS is available here. The memorandum provides an overview of the framework’s functionality, model structure, input and output data, and its applications in transportation planning and broader policy analysis contexts. Interested readers are also encouraged to consult the paper listed below for additional details on the DEMOS methodology.

DEMOS operates on tabular data representing agents or entities (primarily persons and households), and is configured via a simple TOML file. DEMOS can be run from source or using Docker for reproducibility.

This document summarizes instructions to install, configure and run DEMOS. Sections 1-4 will help you correctly organize the data and configuration file, so we recommend reading them once before attempting to run DEMOS.

DEMOS requires at least 16gb of available RAM to execute. This greatly depends on the size and resolution of the input data, but from our internal testing, we recommend at least 16gb of RAM

:local:
:depth: 2
:backlinks: none
:class: this-will-duplicate-information-and-it-is-still-useful-here

1. Installation

Docker Compose (recommended)

The latest docker image for demos is stored in ghcr.io/nrel/demos:latest. The input data and configuration file are fed to the container through volumes. Alternatively, we provide a docker-compose workflow that can be used.

Prepare the configuration file and data folder

# Create a directory where to run DEMOS from
mkdir demos
cd demos

# Create the configuration folder and retrieve an example configuration
mkdir configuration
cd configuration
curl -L -o demos_config.toml https://raw.githubusercontent.com/nrel/DEMOS/main/configuration/demos_config_sfbay.toml

# Create the data folder for the output to be stored
cd ..
mkdir data
# Populate the data folder

# Finally, retrieve the docker-compose.yml file
curl -L -o docker-compose.yml https://raw.githubusercontent.com/nrel/DEMOS/main/docker-compose.yml

Now you can run docker as follows:

docker compose up

Note:
Make sure the Docker Daemon is running. This changes from system to system but Docker Desktop should have a status flag indicating if the daemon is live, if Desktop is available

By default, this assumes that your config file is located in ./configuration/demos_config.toml and the data folder is ./data, with ./ being the root of the project (See the file stucture section for details on how to organize the input data). If you need to specify a different location for them, you can run:

DEMOS_CONFIG_PATH=<path-to-config> DEMOS_DATA_DIR=<path-to-data-dir> docker compose up

Note for MacOS/Windows:
Increase Docker's memory allocation to at least 16–20 GB via Docker Desktop:
Preferences → Resources → Memory.

From Source (is you are not using Docker)

  1. Clone the repository:

    git clone https://github.com/NREL/DEMOS_NREL.git
cd DEMOS_NREL

  2. Create and activate a Python 3.10 environment:

    conda create -n demos-env python=3.10
conda activate demos-env
pip install .

  3. Run DEMOS:

    cd demos
python simulate.py -cfg ../configuration/demos_config.toml

2. Preparing Your Configuration

DEMOS is configured via a TOML file (see example configuration for a full example and configuration API for descriptions on each of the accepted parameters).
At minimum, you must define the persons and households tables:

[[tables]]
file_type = "h5"
table_name = "persons"
filepath = "../data/custom_mpo_06197001_model_data.h5"
h5_key = "persons"

[[tables]]
file_type = "h5"
table_name = "households"
filepath = "../data/custom_mpo_06197001_model_data.h5"
h5_key = "households"

Other tables and module configurations can be added as needed. The default configuration exposes all options with default values.

Examples of important configuration options are:

Selection of modules to run:

The modules parameter accepts a list of strings identifying the modules. By default all are included, but if you'd like to only run a selection of them you can change it. For instance to run only aging and education:

modules = [
    "aging",
    "education",
]

Configuration of calibration procedures:

Certain modules support calibration of the simulation output to observed values. Specific calibration parameters can be set for each module that supports it. If no configuration is provided, calibration is not performed. Additionally, some modules (namely employment and household_reorganization) implement simultaneous calibration. While the employment module implements both types of calibration, only one of the two can be used. An error will be raised if two types of calibration are defined.

Calibration configuration is defined at module-level-config.calibration_procedure (module-level-config is defined differently for every module. The options are displayed here and in each module's documentation). We will use the employment module as an example.

If you want to use simultaneous calibration for the employment module

[employment_module_config.simultaneous_calibration_config]
tolerance = 100
max_iter = 2
learning_rate = 2
momentum_weight = 0.3

Due to the complexity and nuances of simultaneous calibration, the required tables of observed values (observed_entering_workforce and observed_exiting_workforce) are hard-coded, and an error will be raised if they are not present.

If you instead want to use simple calibration

We need to define the following:

[employment_module_config.enter_model_calibration_procedure]
procedure_type = "rmse_error"
observed_values_table = "observed_entering_workforce"
tolerance_type = "relative"
tolerance = 0.01
max_iter = 1000

[employment_module_config.exit_model_calibration_procedure]
procedure_type = "rmse_error"
observed_values_table = "observed_exiting_workforce"
tolerance_type = "relative"
tolerance = 0.01
max_iter = 1000

This will allow DEMOS to execute calibration on each of the two modules in the employment module.

If you want to skip calibration, just delete these entrances from the configuration file.

(file-tree-structure-for-data-and-configuration)=

3. File Tree Structure for Data and Configuration

To run DEMOS, organize your files as follows:

DEMOS_NREL/
├── configuration/
│   └── demos_config.toml                   # Main configuration file (TOML)
├── data/
|   ├── custom_mpo_06197001_model_data.h5   # Example HDF5 data file
│   ├── relmap_06197001.csv                 # Example CSV data file
│   ├── income_rates_06197001.csv           # Example CSV data file
│   ├── hsize_ct_06197001.csv               # Example CSV data file 
|   └── calibrated_configs/                 # Here is were the parameters of the estimated models go
|       └── ...
├── demos/ # Source code 
|   ├── simulate.py                         # Main entry point for running DEMOS
|   ├── models                              # Logic of individual modules
|       └── ... 
|   └── ... 
├── docs/                                   # Documentation
└── ...
  • Place your TOML configuration file in the configuration/ directory.
  • Place all input data files (CSV, HDF5, etc.) in the data/ directory.
  • Make sure the paths in your demos_config.toml match the location of your data files.

Tip:
You can use different data files or directories, but make sure the paths in your configuration file are correct relative to the project root (DEMOS_NREL/).

4. Running DEMOS

After setting up your data and configuration, use Docker as described above.

The output of DEMOS will be stored in data/output/demos_output_{year}.h5.

5. Understanding the Workflow

  • Tables: Each row in the persons and households tables represents an agent or entity.
  • Modules: Simulation logic is organized into modules, each operating on the tables for each simulated year.
  • Orca: DEMOS uses the orca library to manage tables and define lazily-computed columns (see documentation for details).
  • Configuration: All simulation options, data sources, and module settings are controlled via the TOML config file.

6. Troubleshooting Common Errors

  • Missing Required Tables:
    If either persons or households is missing from your config, DEMOS will raise an error:

    ValueError: Both 'persons' and 'households' tables are required. Tables defined: [...]

  • Inconsistent Persons Table:
    DEMOS checks for data consistency (e.g., every household must have exactly one head).
    Behavior is controlled by inconsistent_persons_table_behavior in the config:

    • "error": Stop on inconsistency (default)
    • "fix": Attempt to fix common issues (e.g., assign head to single-person households)
    • "ignore": Proceed without checks (not recommended)

  • Households with Multiple Partners:
    If a household has more than two partners, it will be dropped or raise an error, depending on your config.

  • Memory Errors (Docker):
    DEMOS requires a lot of memory. If you see out-of-memory errors, increase Docker's memory allocation.

7. Next Steps

8. Need Help?

For more details on modules, orca columns, and advanced configuration, see the full documentation.