sumo-fpga-case-study

A complete FPGA acceleration pipeline for SUMO traffic simulation on AWS F2, and the honest engineering postmortem of why it doesn't beat the CPU.

This is the companion repository to the Medium article "Why FPGAs Can't Accelerate SUMO Traffic Simulation (And What I Learned Trying)".

TL;DR

Across about six months of side-project engineering I built a complete FPGA acceleration pipeline for SUMO (the open-source traffic micro-simulator) on AWS F2 instances:

• HLS kernel implementing MSVehicle::planMoveInternal
• Custom PCIM DMA engine with 8-beat bursts
• 2-instance parallel batch dispatcher and controller
• Full host runtime with MMIO and DMA transport
• End-to-end SUMO integration with bit-exact CPU/FPGA result match

It all works. The numbers look great in isolation:

Path	Per Vehicle
FPGA MMIO	968 µs
FPGA DMA (single)	88 µs
FPGA DMA (2-instance batch)	22 µs

But the CPU executes the same function in under 1 µs. Even at our best, the FPGA is ~22x slower per call. By Amdahl's Law, FPGA acceleration of SUMO is capped at ~1.18x (planMove alone) or ~1.50x (all four hot functions).

The full reasoning is in docs/FEASIBILITY_FULL_LOOP.md.

Repository Layout

docs/
  SUMO_FPGA_RETROSPECTIVE.md     The Medium article (companion to this repo)
  FEASIBILITY_FULL_LOOP.md       Amdahl's Law analysis + alternative approaches
  LESSONS_LEARNED.md             21 hard-won lessons from the project
  MEMORY_ARCHITECTURE.md         BRAM/AXI layout reference
  SDE_IMPLEMENTATION.md          PCIM DMA engine architecture
  IMPLEMENTATION_PLAN.md         Full Phase 1-3 design rationale
  *_ANALYSIS.md                  Per-function profiling and HLS analysis

hdk_cl/design/                   Custom Logic (RTL) shell
  cl_planmove_shell.sv           Top-level shell with OCL/PCIM/PCIS
  cl_bram_axi.sv                 Dual-port BRAM with AXI4
  cl_pcim_dma.sv                 Custom PCIM burst DMA engine
  cl_batch_dispatcher.sv         Scatter/gather DMA to N BRAMs
  cl_batch_controller.sv         Parallel kernel start + ap_done polling
  hls_rtl/                       HLS-generated kernel RTL

hls_core/
  src/planMove_HLS.cpp           The HLS kernel (HLS-friendly C++)
  src/fpga_runtime.cpp           Host runtime (MMIO + DMA APIs)
  include/                       Public headers
  test/                          Test harnesses (sim + hardware)
  synthesis/                     Vitis HLS TCL scripts

integration/
  sumo_patches/                  Patches to SUMO source for FPGA backend
  diagnostics/                   Standalone PCIM/DMA debug programs

scripts/                         Build and deployment automation
terraform/                       AWS infrastructure-as-code
dcps_archive/                    Reference DCPs (download from GitHub Releases)

Reproducing the Results

The three reference DCPs (Vivado Design Checkpoints) that produced the working AFIs are available as a GitHub Release attachment. To recreate any of them as a deployable AFI:

# 1. Download the DCP you want
wget https://github.com/<owner>/sumo-fpga-case-study/releases/download/v1.0-reference-builds/2026_03_29-041154.Developer_CL.tar

# 2. Upload to your own S3 bucket (AWS requires it)
aws s3 cp 2026_03_29-041154.Developer_CL.tar s3://<your-fpga-s3-bucket>/dcps/

# 3. Submit AFI creation (free, ~30-60 min)
aws ec2 create-fpga-image \
    --name "planMove_BRAM_recreated_$(date +%Y%m%d)" \
    --input-storage-location Bucket=<your-fpga-s3-bucket>,Key=dcps/2026_03_29-041154.Developer_CL.tar \
    --logs-storage-location  Bucket=<your-fpga-s3-bucket>,Key=logs/recreate

# 4. Once available, load on F2 and enable BME on both PFs
sudo fpga-load-local-image -S 0 -I <new-agfi-id>
sudo setpci -s 34:00.0 COMMAND=0x06:0x04
sudo setpci -s 34:00.1 COMMAND=0x06:0x04

See dcps_archive/README.md for full details on each DCP and what it produces.

Building From RTL

The full RTL is in hdk_cl/design/. Build flow (requires AWS FPGA SDK and Vivado 2024.1):

# HLS synthesis (regenerates planMove_HLS.v from C++)
cd hls_core/synthesis
vitis_hls -f hls_synthesis_lightweight.tcl

# Vivado P&R via AWS HDK (must be on an EC2 instance with Vivado)
./scripts/trigger_remote_build.sh --start <your-build-instance-name>

The build instance auto-stops when done. See scripts/teardown_aws.sh for tearing down resources when finished.

License

Apache License 2.0. See also NOTICE.md for attribution and the relationship to the Medium article.

Acknowledgments

• SUMO (Eclipse Public License 2.0) — the open-source traffic simulator this project attempted to accelerate.
• AWS FPGA SDK — the HDK and shell that made F2 development possible.
• Xilinx Vitis HLS 2024.1 — the high-level synthesis toolchain.