SymboLLM-FE: Symbolic Classification + LLM Feature Engineering

Minimal reproducible pipeline for automated feature engineering on the academic student dropout dataset (3-class classification), using symbolic classification and LLM-guided feature synthesis.

Files

File	Description
academic.csv	Dataset (4424 rows, 36 features, 3-class target: Dropout/Graduate/Enrolled)
symbo_llm_fe.py	Main pipeline script (complete, self-contained)
results.csv	Output: SR-generated formulas with Accuracy scores
llm_prompt.txt	Output: LLM prompt constructed from SR formulas
features_after_llm.csv	Output: augmented dataset after LLM feature engineering

Dependencies

pip install numpy pandas scikit-learn gplearn catboost shap openai

Quick Start

1. Edit symbo_llm_fe.py lines ~282–287: replace YOUR_API_KEY, YOUR_ENDPOINT, YOUR_MODEL with your credentials.
2. Run:

python symbo_llm_fe.py

Pipeline Overview

1. Preprocessing: label-encode categoricals, float32 cast, rename to X0..Xn
2. SHAP sorting: CatBoostClassifier SHAP on training set only (no test leakage), reorder columns
3. SR sliding window: gplearn SymbolicClassifier on expanding feature subsets (189 fits)
4. Formula output: save all discovered formulas with Accuracy to results.csv
5. LLM prompt + API call: select top-K formulas by Accuracy, build prompt, call LLM API
6. Code parsing & execution: extract ```python blocks from LLM response, execute feature engineering
7. Downstream evaluation: CatBoostClassifier with 3-seed cross-validation (Accuracy, F1, AUC)

Data Leakage Prevention

SHAP-based feature sorting (Step 2) computes importance on the training set only:

• A CatBoostClassifier is trained on an initial 80% training split
• SHAP TreeExplainer explains predictions on that same training data
• The derived column ordering is applied globally before any downstream splits
• All subsequent train/test splits (Steps 3, 7) use stratified sampling and are independent

API Configuration

Replace the following placeholders in symbo_llm_fe.py:

client = OpenAI(
    api_key="sk-...",           # your API key
    base_url="https://...",      # your endpoint URL
)
completion = client.chat.completions.create(
    model="hunyuan-lite",        # your model name
    ...
)