SymboLLM-FE: LLM-Accelerated Symbolic Regression for Automated Feature Engineering

SymboLLM-FE is a two-stage hybrid framework for automated feature engineering (AutoFE) on tabular data. It combines symbolic regression — which efficiently mines explicit, mathematically grounded formulas strongly correlated with the target — with LLM refinement — which injects rich domain priors to turn opaque formulas into semantically meaningful, executable feature-engineering code. This achieves both high predictive performance and human-interpretable transparency with only single-digit LLM calls.

This repository provides a minimal, self-contained, reproducible pipeline on the academic student dropout dataset (3-class classification, 4424 rows, 36 features).

📄 Paper	🌐 Project Page

---

Overview

Existing AutoFE approaches face a dichotomy:

• Traditional AutoFE (e.g., AutoFeat, OpenFE) rely on blind operator stacks and exhaustive search. They improve model scores but generate features that are opaque mathematical compositions with no semantic grounding, and their search spaces explode combinatorially.
• LLM-based AutoFE (e.g., CAAFE, OcTree, FEBP, LLM-FE) generate semantically coherent features via task background knowledge, but require costly multi-round iterations with downstream predictors (up to ~50 rounds) and remain susceptible to hallucinations and implicit biases.

SymboLLM-FE synergizes the two paradigms: symbolic regression provides the what — statistically grounded, performance-enhancing formulas — while the LLM provides the why — domain-aware semantic interpretation and code synthesis. This closed loop restricts the LLM to a deterministic feature integrator rather than an unconstrained generator, eliminating spurious features while keeping interpretability.

Stage	Component	Guarantee
1	Spearman-guided expanding-sliding window	Reduces subset search from exponential (O(2^n)) to polynomial (O(n^2))
1	Symbolic regression (gplearn, 14 protected operators)	Explicit formulas with parsimony-controlled complexity, 100% numerical validity
2	Statistical prior-grounded LLM refinement	Semantic interpretation of formulas; 100% of generated rules traceable to SR output
2	Downstream predictor validation loop	Feature utility validated by model performance; auto error-feedback self-correction
2	Compact feature merge	70 features on average vs. 1436 (OpenFE); ~4 API calls per task vs. 10–50 for LLM baselines

On six real-world datasets, SymboLLM-FE outperforms existing AutoFE (+1.23% over traditional, ~+1% over LLM-based, downstream TabPFN), and achieves an average +2.5pp score improvement over vanilla TabPFN across four Kaggle competitions.

---

Installation

git clone https://github.com/<your-org>/SymboLLM-FE.git
cd SymboLLM-FE
pip install numpy pandas scikit-learn gplearn catboost shap openai

Requirements: Python ≥ 3.9, gplearn ≥ 0.4.1, catboost, shap, openai ≥ 1.0.

⚠️ The pipeline uses the OpenAI-compatible chat API. No API key is hard-coded in this repository.

---

API Key

SymboLLM-FE calls an OpenAI-compatible endpoint. Set your key and endpoint in symbo_llm_fe.py (lines ~282–287):

client = OpenAI(
    api_key="sk-...",           # your API key
    base_url="https://...",      # your endpoint URL
)
completion = client.chat.completions.create(
    model="hunyuan-lite",        # your model name
    ...
)

---

Data Setup

The repository ships with the academic dataset (student dropout & academic success, sourced from Kaggle):

File	Description
academic.csv	4424 rows, 36 features, 3-class target (Dropout / Graduate / Enrolled)

The pipeline also runs on any CSV whose last column is the target (label-encoded internally to X0..Xn).

---

Reproducing the Pipeline

Single run

python symbo_llm_fe.py

The script executes the full 7-stage pipeline and writes all artifacts to the repository root:

1. Preprocessing — label-encode the target, cast features to float32, rename columns to X0..Xn
2. Feature-importance sorting (SHAP) — CatBoostClassifier SHAP on the training set only (no test leakage), reorder columns globally
3. SR sliding window — gplearn SymbolicClassifier fits on expanding feature subsets (189 fits for the academic dataset)
4. Formula output — all discovered formulas with Accuracy saved to results.csv
5. LLM prompt + API call — select top-K formulas by Accuracy, build prompt with dataset background, call the LLM
6. Code parsing & execution — extract ```python blocks from the LLM response, execute feature engineering to produce features_after_llm.csv
7. Downstream evaluation — CatBoostClassifier with 3-seed stratified cross-validation (Accuracy, F1, AUC)

Data-leakage prevention

SHAP-based feature sorting (Step 2) computes importance on the training set only; all subsequent train/test splits (Steps 3, 7) use independent stratified sampling, so no test information ever influences feature construction.

Note on the paper vs. this repository. The paper's full pipeline orders features with Spearman correlation; this minimal pipeline uses SHAP-based ordering, which the paper (Appendix D) shows to be highly consistent with Spearman ranking (Kendall's τ = 0.52). Everything else follows the paper's pipeline exactly.

---

Benchmarks

The paper evaluates SymboLLM-FE on six real-world datasets spanning binary / multi-class classification and regression (OpenML + Kaggle):

Dataset	Type	Samples	Features	Task
Credit-g	Binary classification	1,000	20	Credit risk prediction
Spaceship	Binary classification	2,000	13	Kaggle Spaceship Titanic
Cmc	Multi-class classification	1,473	9	Contraceptive method choice
Academic	Multi-class classification	4,424	36	Student dropout prediction
Ailerons	Regression	12,250	33	F16 aileron control
Tesla	Regression	6,906	8	Stock price forecasting

Comparison AutoFE: AutoFeat, OpenFE (traditional); CAAFE, OcTree, FEBP, LLM-FE, LLM-RANK (LLM-based). Downstream predictors: CatBoost, XGBoost, MLP, TabPFN. All results are averaged over three random seeds.

---

Repository Layout

SymboLLM-FE/
├── academic.csv              Dataset (4424 rows, 36 features, 3-class target)
├── symbo_llm_fe.py           Main pipeline script (complete, self-contained)
├── results.csv               Output: SR-generated formulas with Accuracy scores
├── llm_prompt.txt            Output: LLM prompt constructed from SR formulas
└── features_after_llm.csv    Output: augmented dataset after LLM feature engineering

---

Results

The table below is the paper's Table 2 (main results, downstream model TabPFN, ↑ higher is better, ↓ lower is better). SymboLLM-FE achieves the best or second-best result on every dataset.

FE Method	Credit-g↑	Spaceship↑	Cmc↑	Academic↑	Ailerons↓	Tesla↓
Baseline	77.03±0.47	80.79±1.27	57.85±0.89	77.33±0.67	5.10±0.48	2.39±0.09
AutoFeat	77.83±1.43	80.76±1.17	57.85±0.32	76.80±0.46	5.04±0.45	2.38±0.00
OpenFE	76.50±3.34	80.30±1.00	57.85±2.35	76.42±0.61	5.26±0.49	2.18±0.06
CAAFE	78.00±0.71	80.99±1.00	57.78±1.28	77.29±0.56	5.03±0.46	2.66±0.09
OcTree	76.50±0.82	80.79±1.15	57.85±1.52	77.33±0.70	5.09±0.47	2.43±0.10
FEBP	77.50±0.41	80.85±1.29	57.93±0.48	77.36±0.35	5.08±0.48	2.41±0.08
LLM-FE	76.67±0.62	80.22±0.96	57.29±2.20	76.42±0.61	5.05±0.46	2.39±0.09
LLM-RANK	77.50±0.82	80.70±1.15	57.74±0.85	77.25±0.19	5.10±0.48	5.87±0.19
SymboLLM-FE	77.00±1.63	81.27±1.31	57.97±0.73	77.89±0.23	5.02±0.46	2.16±0.06

Efficiency and cost comparison (paper's Table 1, classification averaged over datasets):

Method	Generated Features	Model Score	LLM API Calls
AutoFeat	2	79.75	–
OpenFE	1436	79.53	–
CAAFE	30	79.94	10
OcTree	50	79.52	50
FEBP	200	79.36	29
LLM-FE	30	79.59	35
LLM-RANK	–	78.86	7
SymboLLM-FE	70	80.02	4

SymboLLM-FE secures the highest model score (80.02) while generating a compact feature set (70) with only 4 API calls — versus 10–50 for LLM-based baselines — demonstrating a superior efficiency–performance balance. With stronger reasoning LLM backbones (GPT-o1, DeepSeek-R1), accuracy further improves (e.g., 79.49% on TabPFN with DeepSeek-R1).

Ablation study (paper's Figure 4) confirms each component is indispensable: removing Spearman pre-sorting, the expanding-sliding window, or LLM refinement all degrade performance.

---

Citation

@inproceedings{cheng2026symbollm-fe,
  title={SymboLLM-FE: LLM-Accelerated Symbolic Regression
for Automated Feature Engineering on Tabular Data},
  author={Zi-Jian Cheng1,2, Zi-Yi Jia1,2, Zhi Zhou1,3, Yu-Feng Li1,3, Lan-Zhe Guo1,2 †},
  booktitle={Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing},
  year={2026}
}

---

License

This project is released under the MIT License.