Six real notebooks doing genuine first-contact EDA on unfamiliar datasets. Source notebooks were read but not executed. The goal is to back out natural prompts and use them to test xorq expression codegen via the MCP tools.
Source: https://github.com/rajviishah/Exploratory-Data-Analysis/blob/master/EDA_NYC_Airbnb.ipynb Dataset: NYC Airbnb Open Data 2019 — 48,895 listings, 16 columns
| # | Operation | Natural prompt |
|---|---|---|
| 1 | isnull().sum() — 10,052 nulls in last_review and reviews_per_month |
"how many nulls are in each column?" |
| 2 | Fill nulls with domain defaults (0, "NoName", "NotReviewed") | "fill missing reviews_per_month with 0 and missing names with 'NoName'" |
| 3 | price.describe() — mean $152, max $10,000 |
"give me summary stats for the price column" |
| 4 | Histogram of price, split at $1000 | "show me the distribution of price, then split it at $1000 to see both ends" |
| 5 | neighbourhood.value_counts() — 221 unique; Williamsburg leads |
"what are the most common neighbourhoods and how many listings does each have?" |
| 6 | groupby().filter(lambda x: x.count() > 200) — 87% in high-density areas |
"filter to only neighbourhoods with more than 200 listings and count how many listings remain" |
| 7 | groupby('neighbourhood_group')['price'].mean() |
"group by neighbourhood group and compute the average price for each" |
| 8 | room_type.value_counts() + mean price by room type |
"show me the count and average price for each room type" |
| 9 | price.corr(minimum_nights) — near-zero |
"what's the correlation between price and minimum nights?" |
Pandas-specific (won't map to ibis): groupby().filter(lambda) on group-size predicate, .apply(len) on string column, capping outliers via .loc[(cond), col] = value.
Source: https://github.com/AnalystHub-Hub/Superstore-Sales-Analysis/blob/main/SuperStore_data.ipynb Dataset: Sample Superstore — ~10,296 orders, 21 columns (Order Date, Ship Mode, Segment, Region, Category, Sub-Category, Sales, Profit, Discount, Quantity)
| # | Operation | Natural prompt |
|---|---|---|
| 1 | isnull().sum() — 302 nulls, then dropna() |
"how many nulls are there per column, and drop any rows with missing values" |
| 2 | duplicated().sum() — zero |
"check for duplicate rows" |
| 3 | describe() on Sales and Profit — mean sale $229, profit ranging -$6,599 to $8,399 |
"give me summary statistics for sales and profit" |
| 4 | Correlation heatmap — strong positive Sales↔Profit, strong negative Discount↔Profit | "show me a correlation heatmap and tell me which variables are most related to profit" |
| 5 | Outlier detection on Sales using IQR — 1,167 records above $1,068 | "find outliers in the sales column using the IQR method" |
| 6 | groupby('Region')['Profit'].sum().sort_values() |
"which region generates the most total profit?" |
| 7 | groupby('Sub-Category')[['Sales','Profit']].sum().sort_values('Sales') |
"group by sub-category and show total sales and profit, sorted by sales" |
Pandas-specific: subplot loops for frequency bars, box plots per category.
Source: https://chih-ling-hsu.github.io/2018/05/14/NYC Dataset: NYC Yellow Taxi Jan–Jun 2017 — hundreds of millions of rows (loaded in chunks + SQLite); taxi zone shapefile
| # | Operation | Natural prompt |
|---|---|---|
| 1 | Load sample 10 rows, inspect columns | "load a sample of 10 rows and show me the column names and first records" |
| 2 | trip_distance.describe() — mean 2.89, max 151 mi, extremely right-skewed |
"describe the trip_distance column — what's the mean, median, and max?" |
| 3 | Extract pickup hour from datetime | "parse the pickup datetime and extract just the hour into a new column" |
| 4 | groupby('borough').agg({'PUcount':'sum'}).sort_values() — Manhattan 88M, Staten Island 13K |
"group trips by borough and count total pickups per borough, sorted descending" |
| 5 | Pickups and dropoffs by hour — peak 6–7PM, trough 5AM | "show me the number of pickups and dropoffs for each hour of the day" |
| 6 | Split trips by distance (< 30 vs >= 30 miles), compare OD pairs |
"split trips into short and long by distance and compare the most common origin-destination pairs" |
| 7 | WHERE fare_type = 'negotiated' — 40% of long trips |
"what share of long-distance trips used negotiated fares?" |
Pandas-specific: chunked iteration for massive file, substring datetime parsing, shapefile parsing with pyshp/Shapely.
Source: https://github.com/fikrionii/Dicoding-Bike-Sharing/blob/main/notebook-bikeshare-analysis.ipynb Dataset: Capital Bikeshare hourly data — 17,379 rows, 17 columns (normalized temp, humidity, windspeed, season codes, casual/registered user counts)
| # | Operation | Natural prompt |
|---|---|---|
| 1 | isna().sum() — zero nulls; duplicated().sum() — zero |
"check for any missing values or duplicate rows" |
| 2 | describe() on numeric columns |
"give me summary stats for all numeric columns" |
| 3 | Extract year, month name, weekday name from date | "convert the date column to datetime and extract year, month name, and weekday name as new columns" |
| 4 | Map season codes 1–4 to string names | "the season column uses codes 1-4 — replace them with the actual season names" |
| 5 | Denormalize weather: temp × 41, hum × 100 | "the temperature column is normalized 0–1 — convert it back to Celsius by multiplying by 41" |
| 6 | groupby('season')['cnt'].mean() |
"group by season and compute the average number of rentals" |
| 7 | groupby('hr')['cnt'].mean() — hourly usage curve |
"group by hour of day and show average rentals — I want to see the usage curve across the day" |
| 8 | Correlation of temp and hum against cnt | "what's the correlation between temperature and total rentals? what about humidity?" |
| 9 | groupby('weekday')['cnt'].mean() |
"compare average rentals by day of the week" |
Pandas-specific: .dt accessor chain (maps fine to ibis t.ts.hour() etc.), row-by-row season label mapping via Python list (maps to case().when()).
Source: https://github.com/GauravPadawe/Framingham-Heart-Study/blob/master/Framingham%20Heart%20Study.py Dataset: Framingham Heart Study — 4,240 patients, 16 features (demographics, clinical measurements, 10-year CHD outcome)
| # | Operation | Natural prompt |
|---|---|---|
| 1 | isnull().sum() — glucose 9.15% null, education 2.47%, cigsPerDay 0.68% |
"show me the percentage of missing values for each column" |
| 2 | describe() on continuous variables — age 32–70, sysBP 83–295 |
"give me descriptive statistics for all numeric columns" |
| 3 | TenYearCHD.value_counts() — 85:15 class imbalance |
"how many patients have the positive outcome vs negative? what's the ratio?" |
| 4 | groupby('male')['TenYearCHD'].value_counts() — males have higher CHD rate |
"group by gender and show the CHD diagnosis rate for each group" |
| 5 | groupby('currentSmoker')['cigsPerDay'].median() |
"what's the median cigarettes per day for smokers vs non-smokers?" |
| 6 | Median imputation per group for cigsPerDay and BMI | "fill missing cigsPerDay using the median for each smoker/non-smoker group" |
| 7 | Create age bands (adult/middle-aged/senior) | "bin the age column into three groups: adult, middle-aged, and senior" |
Pandas-specific: .transform('mean') for group-based imputation in place — maps to ibis window function mutate(fill=col.mean().over(ibis.window(group_by=...))). SMOTE is ML-only.
Source: https://github.com/dipalira/Melbourne-Housing-Data-Kaggle/blob/master/housing.ipynb Dataset: Melbourne Housing Full — ~34K listings (post-filter ~27K), 21 columns (Suburb, Type, Price, Rooms, Bathroom, Landsize, BuildingArea, YearBuilt, Regionname, coordinates)
| # | Operation | Natural prompt |
|---|---|---|
| 1 | isnull().sum() — Price 7,610 nulls, BuildingArea 21,115, YearBuilt 19,306 |
"how many nulls are in each column? which columns have the most missing data?" |
| 2 | Drop rows where Price or Regionname is null | "drop any rows where price is missing, and drop rows with no region" |
| 3 | groupby('Rooms')['Bathroom'].transform('mean') — fill Bathroom per room count |
"fill missing bathroom counts using the average for properties with the same number of rooms" |
| 4 | groupby('Type')['Car'].transform('mean') — fill Car spaces by type |
"fill missing car spaces using the average for each property type" |
| 5 | groupby('Regionname')['Price'].mean().sort_values() — Southern Metropolitan $1.4M, Western Victoria $433K |
"group by region and compute the mean sale price per region, sorted high to low" |
Pandas-specific: .transform() for group-fill maps to ibis window mutate; seaborn factorplot doesn't translate.
| pandas pattern | ibis equivalent |
|---|---|
groupby(col).agg(mean/sum/count) |
group_by().aggregate() |
value_counts() |
group_by(col).aggregate(n=_.count()).order_by(ibis.desc('n')) |
describe() on a column |
aggregate(mean=, std=, min=, max=) |
| Boolean filter | filter() |
sort_values() |
order_by() |
| Computed column | mutate() |
.dt.hour / .dt.month |
t.ts.hour() / t.ts.month() (ibis method names differ from pandas) |
case/when mapping |
ibis.case().when(...).else_(...).end() |
corr() between two columns |
aggregate(r=t.x.corr(t.y)) — verify xorq supports |
head(n) |
limit(n) |
crosstab(a, b) |
group_by([a, b]).aggregate(n=_.count()) |
isnull().sum() |
aggregate(null_count=col.isnull().sum()) |
groupby().filter(lambda x: x.count() > N)— group-size predicate applied back to rows; requires self-join or window in ibisgroupby().transform('mean')— group-mean imputation; maps to windowmutatebut LLM won't get there first.apply()with arbitrary Python functionspd.melt()/pivot_table()reshaping- Chunk-by-chunk file loading
.iloc/.atpositional access- SMOTE / sklearn preprocessing
Using citibike_large (2.5M Sep 2020 rides, columns: ride_id, rideable_type, started_at, ended_at, start_station_name, start_station_id, end_station_name, end_station_id, start_lat, start_lng, end_lat, end_lng, member_casual, trip_duration).
| # | Prompt | Expected ibis pattern | Predicted difficulty |
|---|---|---|---|
| 1 | "count the number of null values in each column" | aggregate(null_count=col.isnull().sum()) per column |
easy |
| 2 | "group rides by start_station_name and count how many rides started from each, show top 20" | group_by().aggregate(n=_.count()).order_by(desc).limit(20) |
easy |
| 3 | "what's the average trip_duration by rideable_type?" | group_by('rideable_type').aggregate(avg=_.trip_duration.mean()) |
easy |
| 4 | "add a column for the hour of day from started_at, then group by hour and count rides" | mutate(hour=t.started_at.hour()).group_by('hour').aggregate(n=_.count()) |
easy |
| 5 | "what share of rides have a null end_station_name?" | aggregate(pct=_.end_station_name.isnull().sum() / _.count()) |
medium — denominator |
| 6 | "compute the mean and standard deviation of trip_duration" | aggregate(mean=, std=) |
easy |
| 7 | "for each rideable_type, show min, max, mean, and median trip_duration" | multi-agg group_by().aggregate(min=, max=, mean=, approx_median=) |
medium — median |
| 8 | "filter to rides longer than 60 minutes and count how many there are per rideable_type" | filter(t.trip_duration > 3600).group_by().aggregate(n=_.count()) |
easy |
| 9 | "group by member_casual and rideable_type, give me count and average duration" | group_by([a, b]).aggregate(n=, avg=) |
easy |
| 10 | "flag rides as 'long' if trip_duration > 3600 and 'short' otherwise — add this as a new column" | mutate(flag=ibis.case().when(...).else_(...).end()) |
easy |
| 11 | "fill missing end_station_name with 'unknown'" | mutate(col=t.col.fillna('unknown')) |
easy |
| 12 | "for each start station, compute average trip duration — then filter to only stations with more than 100 rides" | requires having-clause or self-join; LLM will try groupby().filter(lambda) |
hard |
| 13 | "compute the correlation between trip_duration and hour of day" | requires mutate first, then aggregate(r=a.corr(b)) |
medium |
| 14 | "show the median trip duration for each hour of the day" | mutate(hour=).group_by('hour').aggregate(med=_.trip_duration.approx_median()) |
medium — median method name |
| 15 | "fill missing end_station_name using the most common station name for that rideable_type" | window aggregate + fillna — hard even in pandas | hard |
.describe()shorthand — LLM will call.describe()on an ibis expression; it doesn't exist. Must decompose into explicit aggregates.isnull().mean()— natural way to express null share; ibis needsisnull().sum() / count().- Datetime accessor names — LLM trained on pandas will emit
.dt.hour; ibis uses.hour(). groupby().filter(lambda)— LLM's default for group-size filtering; requires window or self-join in ibis.- Group-based imputation — LLM will reach for
.transform('mean'); ibis needscol.mean().over(window). approx_median()vsmedian()— ibis usesapprox_median(); LLM will trymedian()and fail or silently get nothing.select(ibis._, ...)— ibis wildcard not supported in xorq's vendored ibis; prefermutate.import ibisdirectly — must useimport xorq.vendor.ibis as ibis.