fix: surface valid table names on SQL validation 'table not found' retry#1890
fix: surface valid table names on SQL validation 'table not found' retry#1890ghostiee-11 wants to merge 1 commit into
Conversation
The SQL agent's retry feedback was just the raw error, so when the model referenced a non-existent table (e.g. the source name for a multi-table XArraySQLSource) it kept guessing and exhausted retries. Include the source's actual table names in the feedback so the retry recovers.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1890 +/- ##
==========================================
+ Coverage 70.60% 70.66% +0.05%
==========================================
Files 195 196 +1
Lines 33630 33675 +45
==========================================
+ Hits 23746 23795 +49
+ Misses 9884 9880 -4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| # source name instead of one of its tables, common for | ||
| # multi-table sources like XArraySQLSource). Surface the | ||
| # actual table names so the retry stops guessing. | ||
| available = source.get_tables() |
There was a problem hiding this comment.
What happens if there are over 100 of tables in a database? Won't that consume too much context? I think we may need to do similarity search, perhaps through Metaset?
There was a problem hiding this comment.
Good catch on the ordering. It's already capped at [:50] so context stays bounded, but get_tables() comes back in arbitrary order, so with a lot of tables the slice can drop the one the model actually needs.
The metaset's a better fit, and it's already in context here. get_top_tables() ranks by similarity to the query, so I'll pull the relevant names for that source from it and only fall back to get_tables() when it's empty. Pushing that shortly.
There was a problem hiding this comment.
I think we need to mention it's truncated if it's over 50
When the SQL agent generates a query that references a table that doesn't exist (e.g. the source name instead of one of its tables, which the model does for multi-table sources like
XArraySQLSourcewhere each variable is its own table), validation fails with "table not found". The retry feedback was just the raw error string, so the model kept making the same mistake and exhausted its retries (RetriesExceededError). This surfaces the source's actual table names in the retry feedback so the model recovers.Before
After
Surfaced while testing the Arraylake source control (#1889) against a real multi-table GOES
XArraySQLSource. Adds two deterministic tests for_validate_sql(no LLM needed).