-
-
Notifications
You must be signed in to change notification settings - Fork 30
fix: surface valid table names on SQL validation 'table not found' retry #1890
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
ghostiee-11
wants to merge
1
commit into
holoviz:main
Choose a base branch
from
ghostiee-11:fix/sql-agent-table-not-found-feedback
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| from unittest.mock import AsyncMock, MagicMock | ||
|
|
||
| import pytest | ||
|
|
||
| try: | ||
| import lumen.ai # noqa | ||
| except ModuleNotFoundError: | ||
| pytest.skip( | ||
| "lumen.ai could not be imported, skipping tests.", | ||
| allow_module_level=True, | ||
| ) | ||
|
|
||
| from lumen.ai.agents.sql import SQLAgent | ||
|
|
||
|
|
||
| @pytest.mark.asyncio | ||
| async def test_validate_sql_surfaces_table_names_on_not_found(): | ||
| """On a 'table not found' error the retry feedback should list the | ||
| source's real table names so the model stops using the source name | ||
| (the failure mode for multi-table sources like XArraySQLSource).""" | ||
| agent = SQLAgent() | ||
|
|
||
| source = MagicMock() | ||
| source.dialect = "duckdb" | ||
| source.get_tables.return_value = ["CMI_C01", "DQF_C01"] | ||
|
|
||
| attempts = {"n": 0} | ||
|
|
||
| def execute(sql, *a, **k): | ||
| attempts["n"] += 1 | ||
| if attempts["n"] == 1: | ||
| # First attempt: model referenced the source name as a table. | ||
| raise ValueError("Error during planning: table 'MySource' not found") | ||
| return None # Corrected query validates. | ||
|
|
||
| source.execute.side_effect = execute | ||
|
|
||
| captured = {} | ||
|
|
||
| async def fake_revise(feedback, *a, **k): | ||
| captured["feedback"] = feedback | ||
| return "SELECT * FROM CMI_C01" | ||
|
|
||
| agent.revise = fake_revise | ||
|
|
||
| result = await agent._validate_sql( | ||
| context={}, | ||
| sql_query='SELECT * FROM "MySource"', | ||
| expr_slug="x", | ||
| source=source, | ||
| messages=[], | ||
| step=MagicMock(), | ||
| max_retries=2, | ||
| ) | ||
|
|
||
| # Recovered using the corrected SQL. | ||
| assert "CMI_C01" in result | ||
| # The retry feedback surfaced the real table names + the hint. | ||
| assert "CMI_C01" in captured["feedback"] | ||
| assert "DQF_C01" in captured["feedback"] | ||
| assert "not the source name" in captured["feedback"] | ||
|
|
||
|
|
||
| @pytest.mark.asyncio | ||
| async def test_validate_sql_valid_query_no_retry(): | ||
| """A valid query validates on the first try with no retry/feedback.""" | ||
| agent = SQLAgent() | ||
|
|
||
| source = MagicMock() | ||
| source.dialect = "duckdb" | ||
| source.execute.return_value = None # valid | ||
|
|
||
| agent.revise = AsyncMock( | ||
| side_effect=AssertionError("revise should not be called") | ||
| ) | ||
|
|
||
| result = await agent._validate_sql( | ||
| context={}, | ||
| sql_query="SELECT * FROM CMI_C01", | ||
| expr_slug="x", | ||
| source=source, | ||
| messages=[], | ||
| step=MagicMock(), | ||
| max_retries=2, | ||
| ) | ||
| assert "CMI_C01" in result | ||
| source.get_tables.assert_not_called() |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What happens if there are over 100 of tables in a database? Won't that consume too much context? I think we may need to do similarity search, perhaps through Metaset?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch on the ordering. It's already capped at
[:50]so context stays bounded, butget_tables()comes back in arbitrary order, so with a lot of tables the slice can drop the one the model actually needs.The metaset's a better fit, and it's already in context here.
get_top_tables()ranks by similarity to the query, so I'll pull the relevant names for that source from it and only fall back toget_tables()when it's empty. Pushing that shortly.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we need to mention it's truncated if it's over 50