[Bug]: RetryGuidelineQueryEngine/GuidelineEvaluator causes ~400k token prompts when used with NLSQLTableQueryEngine(synthesize_response=False)

[csttsn]

Bug Description

I have a working Text-to-SQL setup using NLSQLTableQueryEngine with synthesize_response=False (intentionally disabled to return raw database rows and avoid extra summarization cost/latency).

This setup worked perfectly and used reasonable token counts (~20k tokens max, coming from my custom text_to_sql_prompt).

Previously working code (minimal relevant part)

# Create the base Text-to-SQL query engine
query_engine = NLSQLTableQueryEngine(
    sql_database=sql_database,
    tables=tables_to_include,
    llm=llm,
    text_to_sql_prompt=prompt_template,
    synthesize_response=False,   # We need raw rows, not a summary
)

This engine alone behaved correctly: generated SQL, executed it, returned raw results, token usage stayed within expected bounds.

What changed

I wanted to add safety by validating generated SQL against guidelines (e.g. correct PostgreSQL syntax, avoid dangerous patterns, and so on). I wrapped the engine like this:

from llama_index.core.evaluation import GuidelineEvaluator
from llama_index.core.query_engine import RetryGuidelineQueryEngine

# Create the base Text-to-SQL query engine
base_query_engine = NLSQLTableQueryEngine(
    sql_database=sql_database,
    tables=tables_to_include,
    llm=_llm,
    text_to_sql_prompt=prompt_template,
    synthesize_response=False, 
 )

# Create the GuidelineEvaluator using our LLM and guidelines from config
# This evaluator will check the generated SQL against rules semantically.
guideline_evaluator = GuidelineEvaluator(
    llm=_llm, 
    guidelines=GUIDELINES 
)

# Wrap the base engine with RetryGuidelineQueryEngine for retries
# This adds error recovery: If evaluation fails, it refines and retries.
retry_query_engine = RetryGuidelineQueryEngine(
    query_engine=base_query_engine,
    guideline_evaluator=guideline_evaluator, 
    resynthesize_query=True,  # Enable query refinement using feedback
    max_retries=1
)

As soon as a retry is triggered (i.e. the evaluator rejects the first SQL), the next LLM call during the rewrite phase receives an extremely large prompt in my logs up to ~400,000 tokens, far exceeding my 20k-token custom prompt.

What appears to be happening (but I'm not 100% sure)

When synthesize_response=False:

• The Response object returned by NLSQLTableQueryEngine contains the entire raw result set (thousands of rows) as a large string in response.response.
• GuidelineEvaluator seems to send response.response to the LLM for evaluation.
• Since my guideline is about SQL syntax, I would expect only the generated SQL query to be evaluated.
• Instead, it looks like the evaluator is receiving and sending the full database result set (CSV-like text of all returned rows), causing massive token usage and hitting rate limits immediately.

Expected behavior

When a guideline is about the SQL query itself (syntax, safety, etc.), the evaluator should inspect only the generated SQL (available in response.source_nodes[0].text or similar), not the executed result data.

Or maybe I’m doing something wrong; in that case, I’d really appreciate a solution for implementing the validation of SQL queries generated by NLSQLTableQueryEngine.

Version

0.12.49

Steps to Reproduce

Minimal reproduction (copied from bug description)

from llama_index.core.evaluation import GuidelineEvaluator
from llama_index.core.query_engine import RetryGuidelineQueryEngine

# Create the base Text-to-SQL query engine
base_query_engine = NLSQLTableQueryEngine(
    sql_database=sql_database,
    tables=tables_to_include,
    llm=_llm,
    text_to_sql_prompt=prompt_template,
    synthesize_response=False, 
 )

# Create the GuidelineEvaluator using our LLM and guidelines from config
# This evaluator will check the generated SQL against rules semantically.
guideline_evaluator = GuidelineEvaluator(
    llm=_llm, 
    guidelines=GUIDELINES 
)

# Wrap the base engine with RetryGuidelineQueryEngine for retries
# This adds error recovery: If evaluation fails, it refines and retries.
retry_query_engine = RetryGuidelineQueryEngine(
    query_engine=base_query_engine,
    guideline_evaluator=guideline_evaluator, 
     resynthesize_query=True,  # Enable query refinement using feedback
     max_retries=1
)

That is, adding GuidelineEvaluator and RetryGuidelineQueryEngine causes extremely large prompts when the result obtained from base_query_engine is rewritten by RetryGuidelineQueryEngine.

Relevant Logs/Tracbacks

[dosubot]

You're correct: the huge token usage happens because GuidelineEvaluator (via BaseEvaluator) sends the entire response string—including all raw database rows—to the LLM for evaluation when synthesize_response=False. This is the default behavior, and there’s no built-in config to change it. The generated SQL is actually stored in response.metadata["sql_query"], but the evaluator ignores this by default and just uses response.response, which can be massive with large result sets citation.

To fix this, subclass GuidelineEvaluator (or BaseEvaluator) and override evaluate_response to extract only the SQL query from response.metadata["sql_query"] and pass that to the LLM for evaluation. Here’s a minimal example:

from llama_index.core.evaluation import GuidelineEvaluator

class SQLOnlyGuidelineEvaluator(GuidelineEvaluator):
    def evaluate_response(self, query=None, response=None, **kwargs):
        sql_query = None
        if response is not None and "sql_query" in response.metadata:
            sql_query = response.metadata["sql_query"]
        # Fallback to empty string or raise if not found
        return self.evaluate(query=query, response=sql_query or "", **kwargs)

Then use this custom evaluator in your RetryGuidelineQueryEngine setup:

guideline_evaluator = SQLOnlyGuidelineEvaluator(
    llm=_llm,
    guidelines=GUIDELINES,
)
retry_query_engine = RetryGuidelineQueryEngine(
    query_engine=base_query_engine,
    guideline_evaluator=guideline_evaluator,
    resynthesize_query=True,
    max_retries=1,
)

This ensures only the generated SQL is sent to the LLM for guideline checks, not the full result set, so your token usage will stay low citation. If you want async support, override aevaluate_response similarly.

Let me know if you need a version that handles missing metadata or more advanced extraction!

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[logan-markewich]

Its always going to pass the full results to the evaluator. Maybe a good option is just implementing this outside/around the existing query engine