code release

Author: frankniujc

ABOUT_THIS_TEMPLATE.md

@@ -2,115 +2,54 @@
   <img src='logo.png' width='200'>
 </p>
 
-# arxiv2025_icl_investigation
-[![Arxiv](https://img.shields.io/badge/Arxiv-YYMM.NNNNN-red?style=flat-square&logo=arxiv&logoColor=white)](https://put-here-your-paper.com)
-[![License](https://img.shields.io/github/license/UKPLab/arxiv2025-icl-investigation)](https://opensource.org/licenses/Apache-2.0)
-[![Python Versions](https://img.shields.io/badge/Python-3.9-blue.svg?style=flat&logo=python&logoColor=white)](https://www.python.org/)
-[![CI](https://github.com/UKPLab/arxiv2025-icl-investigation/actions/workflows/main.yml/badge.svg)](https://github.com/UKPLab/arxiv2025-icl-investigation/actions/workflows/main.yml)
+# Illusion or Algorithm?
+[![Arxiv](https://img.shields.io/badge/Arxiv-2505.11004-red?style=flat-square&logo=arxiv&logoColor=white)](https://arxiv.org/abs/2505.11004)
 
-This is the official template for new Python projects at UKP Lab. It was adapted for the needs of UKP Lab from the excellent [python-project-template](https://github.com/rochacbruno/python-project-template/) by [rochacbruno](https://github.com/rochacbruno).
+This repository contains the code to generate the data and results to the paper: [Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning](https://arxiv.org/abs/2505.11004).
 
-It should help you start your project and give you continuous status updates on the development through [GitHub Actions](https://docs.github.com/en/actions).
+> **Abstract:** Large-scale Transformer language models (LMs) trained solely on next-token prediction with web-scale data can solve a wide range of tasks after seeing just a few examples. The mechanism behind this capability, known as in-context learning (ICL), remains both controversial and poorly understood. Some studies argue that it is merely the result of memorizing vast amounts of data, while others contend that it reflects a fundamental, symbolic algorithmic development in LMs. In this work, we introduce a suite of investigative tasks and a novel method to systematically investigate ICL by leveraging the full Pythia scaling suite, including interim checkpoints that capture progressively larger amount of training data. By carefully exploring ICL performance on downstream tasks and simultaneously conducting a mechanistic analysis of the residual stream's subspace, we demonstrate that ICL extends beyond mere "memorization" of the training corpus, yet does not amount to the implementation of an independent symbolic algorithm. Our results also clarify several aspects of ICL, including the influence of training dynamics, model capabilities, and elements of mechanistic interpretability. Overall, our work advances the understanding of ICL and its implications, offering model developers insights into potential improvements and providing AI security practitioners with a basis for more informed guidelines.
 
-> **Abstract:** The study of natural language processing (NLP) has gained increasing importance in recent years, with applications ranging from machine translation to sentiment analysis. Properly managing Python projects in this domain is of paramount importance to ensure reproducibility and facilitate collaboration. The template provides a structured starting point for projects and offers continuous status updates on development through GitHub Actions. Key features include a basic setup.py file for installation, packaging, and distribution, documentation structure using mkdocs, testing structure using pytest, code linting with pylint, and entry points for executing the program with basic CLI argument parsing. Additionally, the template incorporates continuous integration using GitHub Actions with jobs to check, lint, and test the project, ensuring robustness and reliability throughout the development process.
-
-Contact person: [Federico Tiblias](mailto:[email protected]) 
-
-[UKP Lab](https://www.ukp.tu-darmstadt.de/) | [TU Darmstadt](https://www.tu-darmstadt.de/
-)
+Contact: [Jingcheng Niu](mailto:[email protected]) and [Subhabrata Dutta](mailto:[email protected])
+@ [UKP Lab](https://www.ukp.tu-darmstadt.de/) | [TU Darmstadt](https://www.tu-darmstadt.de/)
 
 Don't hesitate to send us an e-mail or report an issue, if something is broken (and it shouldn't be) or if you have further questions.
 
+## ICL Tasks
 
-## Getting Started
-
-> **DO NOT CLONE OR FORK**
-
-If you want to set up this template:
-
-1. Request a repository on UKP Lab's GitHub by following the standard procedure on the wiki. It will install the template directly. Alternatively, set it up in your personal GitHub account by clicking **[Use this template](https://github.com/rochacbruno/python-project-template/generate)**.
-2. Wait until the first run of CI finishes. Github Actions will commit to your new repo with a "✅ Ready to clone and code" message.
-3. Delete optional files: 
-    - If you don't need automatic documentation generation, you can delete folder `docs`, file `.github\workflows\docs.yml` and `mkdocs.yml`
-    - If you don't want automatic testing, you can delete folder `tests` and file `.github\workflows\tests.yml`
-    - If you do not wish to have a project page, delete folder `static` and files `.nojekyll`, `index.html`
-4. Prepare a virtual environment:
-```bash
-python -m venv .venv
-source .venv/bin/activate
-pip install .
-pip install -r requirements-dev.txt # Only needed for development
-```
-5. Adapt anything else (for example this file) to your project. 
-
-6. Read the file [ABOUT_THIS_TEMPLATE.md](ABOUT_THIS_TEMPLATE.md)  for more information about development.
-
-## Usage
-
-### Using the classes
-
-To import classes/methods of `arxiv2025_icl_investigation` from inside the package itself you can use relative imports: 
-
-```py
-from .base import BaseClass # Notice how I omit the package name
-
-BaseClass().something()
-```
-
-To import classes/methods from outside the package (e.g. when you want to use the package in some other project) you can instead refer to the package name:
-
-```py
-from arxiv2025_icl_investigation import BaseClass # Notice how I omit the file name
-from arxiv2025_icl_investigation.subpackage import SubPackageClass # Here it's necessary because it's a subpackage
-
-BaseClass().something()
-SubPackageClass().something()
-```
+We produce the results in **Sections 4–6** of our paper using the ICL tasks implemented in this code base. These tasks are designed to probe different aspects of in-context learning, such as copying, symbolic substitution, and pattern induction.
 
-### Using scripts
-
-This is how you can use `arxiv2025_icl_investigation` from command line:
-
-```bash
-$ python -m arxiv2025_icl_investigation
-```
+See [`example.ipynb`](./example.ipynb) for runnable demos and usage instructions showing how we generate and evaluate these tasks across Pythia model checkpoints.
 
-### Expected results
+## Singular Residual Stream Direction Analysis (SUDA)
 
-After running the experiments, you should expect the following results:
+Finally, we present a mechanistic connection between the development of ICL competence and specialization in the residual stream’s subspace through Singular Unembedding Direction Analysis (SUDA).
 
-(Feel free to describe your expected results here...)
+SUDA is a diagnostic method that projects the residual stream onto the singular vectors of the unembedding matrix, allowing us to trace how models allocate subspace as they learn to perform in-context learning. Specifically, we:
 
-### Parameter description
+1. Compute a singular value decomposition of the unembedding matrix: $SVD(W_U) = UST^\top$.
+2. Project the residual stream activations onto individual directions in $V$.
+3. Measure the task-relevant signal captured by each direction by evaluating how well the model performs when using only a subset of directions.
 
-* `x, --xxxx`: This parameter does something nice
+Across a diverse set of ICL tasks, we observe that while the performance trajectories vary (some tasks improve steadily, others show late or abrupt development) the underlying trend of internal structural formation in the residual stream remains consistent. This suggests that models consistently learn to specialize subspaces in a task-relevant way as training progresses.
 
-* ...
-
-* `z, --zzzz`: This parameter does something even nicer
-
-## Development
-
-Read the FAQs in [ABOUT_THIS_TEMPLATE.md](ABOUT_THIS_TEMPLATE.md) to learn more about how this template works and where you should put your classes & methods. Make sure you've correctly installed `requirements-dev.txt` dependencies
+The implementation is available at [`icl_analysis/suda.py`](icl_analysis/suda.py).
 
 ## Cite
 
 Please use the following citation:
 
 ```
-@InProceedings{smith:20xx:CONFERENCE_TITLE,
-  author    = {Smith, John},
-  title     = {My Paper Title},
-  booktitle = {Proceedings of the 20XX Conference on XXXX},
-  month     = mmm,
-  year      = {20xx},
-  address   = {Gotham City, USA},
-  publisher = {Association for XXX},
-  pages     = {XXXX--XXXX},
-  url       = {http://xxxx.xxx}
+@misc{niu2025illusionalgorithminvestigatingmemorization,
+      title={Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning}, 
+      author={Jingcheng Niu and
+        Subhabrata Dutta and
+        Ahmed Elshabrawy and
+        Harish Tayyar Madabushi and
+        Iryna Gurevych},
+      year={2025},
+      eprint={2505.11004},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2505.11004}, 
 }
-```
-
-## Disclaimer
-
-> This repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication. 
+```