Intelligence at the Edge of Chaos

This repository contains the official implementation for our ICLR 2025 paper, Intelligence at the Edge of Chaos. In this work, we investigate how the complexity of rule-based systems, specifically elementary cellular automata (ECA), impacts the intelligence of large language models (LLMs) trained on sequences generated by these systems. We train LLMs on a spectrum of ECA patterns—ranging from uniform and periodic to highly chaotic—and evaluate their performance across various downstream tasks, including logical reasoning and chess move prediction. Our experiments demonstrate that LLMs exposed to intermediate, structured complexity significantly outperform those trained on overly simple or excessively chaotic patterns. These results highlight a critical insight: structured complexity serves as an essential catalyst for developing advanced artificial cognition. Our findings propose that intelligence emerges most effectively at the boundary between order and chaos.

Table of Contents

1. Overview
2. Setup
3. Pre-training
4. Downstream Tasks
   • ARC
   • Chess
   • NIM Game
5. Model Configurations
6. Citation
7. License

Overview

Explore how varying complexity levels influence model performance in both pretraining and subsequent evaluations. The implementations allow investigating whether and how increased complexity leads to better results across different tasks.

Setup

Create a new virtual environment:

conda create -n complexity python=3.10
conda activate complexity

Install the dependencies

pip install -r requirements.txt

Pre-training

Directory: Pretrain/

Key Files:

• pretrain.py: Main script for pre-training

Run Command:

python Pretrain/pretrain.py \
    --gpt2_size small \
    --rule 110 \
    --k 1 \
    --save_dir your_save_path \

See Pretrain/pretrain.py for more command line arguments.

Downstream Tasks

Abstract Reasoning Task

Directory: DownStream_Task/ARC/

Key Files:

• arc.py: The main script for the abstract reasoning downstream task
• build_dataset.ipynb: Prepares and processes the dataset

Run Command:

python DownStream_Task/ARC/arc.py \
    --gpt2_size small \
    --ckpt_dirs your_checkpoint_path \
    --save_dir your_save_path \

See DownStream_Task/ARC/arc.py for more command line arguments.

Chess

Directory: DownStream_Task/chess/

Key Files:

• chess.py: Primary script for the chess downstream task
• san-v2.zip: Dataset for the chess downstream task

Run Command:

cd DownStream_Task/chess; unzip san-v2.zip; cd -
python DownStream_Task/chess/chess.py \
    --ckpt_dirs your_checkpoint_path \
    --save_dir your_save_path

See DownStream_Task/chess/chess.py for more command line arguments.

NIM Game

Directory: DownStream_Task/NIM/

Key Files:

• nim_game.py: The main script for the NIM game downstream task
• data/: Contains the NIM game dataset

Run Command:

python DownStream_Task/NIM/nim_game.py \
    --ckpt_dirs your_checkpoint_path \
    --save_dir your_save_path 

See DownStream_Task/NIM/nim_game.py for more command line arguments.

Model Configurations

The supported model sizes are:

Model Size	Layers	Heads	Embedding Size
tiny	1	1	64
small	12	12	768
large	36	20	1280

Citation

If you find this work useful, please cite it:

@article{zhang2024intelligence,
  title={Intelligence at the Edge of Chaos},
  author={Zhang, Shiyang and Patel, Aakash and Rizvi, Syed A and Liu, Nianchen and He, Sizhuang and Karbasi, Amin and Zappala, Emanuele and van Dijk, David},
  journal={arXiv preprint arXiv:2410.02536},
  year={2024}
}

License

This project is licensed under the GPL-3.0 License. See the LICENSE file for details.