why so slow. compare langchaindatabase and vanna ....

[colorwlof]

System Info

2.0.44, the same database ,the same question , langchain cost 7s, vanna cost 8s(2s for plot), i test 48 hours to check why pandasai so slow , find that ,all by the generate_python_code.tmpl and generate_python_code_with_sql.tmpl.
Why use a huge prompt instead of separating it and use different prompts for different stages. To make a large model, reduce the amount of reading and thinking each time, which can save a lot of time and tokens.

🐛 Describe the bug

2024-05-30 00:50:25,773 - logger.py[line:75] - INFO: Persisting Agent Training data in E:\LANGCHAT\Langchain-Chatchat\chromadb
2024-05-30 00:50:25,810 - segment.py[line:189] - INFO: Collection pandasai-qa is not created.
2024-05-30 00:50:25,811 - segment.py[line:189] - INFO: Collection pandasai-docs is not created.
2024-05-30 00:50:25,811 - logger.py[line:75] - INFO: Successfully initialized collection pandasai
2024-05-30 00:50:26,037 - logger.py[line:75] - INFO: Question: 列出12号楼所有电表中度数最高的前5个,生成图表
2024-05-30 00:50:26,037 - logger.py[line:75] - INFO: Running PandasAI with langchain_tongyi LLM...
2024-05-30 00:50:26,037 - logger.py[line:75] - INFO: Prompt ID: c2c97c1c-fc13-4fd8-87d1-a5f647ce100f
2024-05-30 00:50:26,038 - logger.py[line:75] - INFO: Executing Pipeline: GenerateChatPipeline
2024-05-30 00:50:28,044 - logger.py[line:75] - INFO: Executing Step 0: ValidatePipelineInput
2024-05-30 00:50:28,045 - logger.py[line:75] - INFO: Executing Step 1: CacheLookup
2024-05-30 00:50:28,045 - logger.py[line:75] - INFO: Executing Step 2: PromptGeneration
2024-05-30 00:50:30,261 - logger.py[line:75] - INFO: Executing Step 3: CodeGenerator
2024-05-30 00:50:45,749 - logger.py[line:75] - INFO: Executing Step 4: CachePopulation
2024-05-30 00:50:45,750 - logger.py[line:75] - INFO: Executing Step 5: CodeCleaning
2024-05-30 00:50:45,750 - logger.py[line:75] - INFO: Saving charts to exports\charts\c2c97c1c-fc13-4fd8-87d1-a5f647ce100f.png
2024-05-30 00:50:45,752 - logger.py[line:75] - INFO:

[dosubot]

PandasAI uses a comprehensive prompt instead of separating it into different stages to ensure that all necessary information is included in a single prompt. This approach can be more efficient and cohesive for the execution process, as it includes various configurations and the last code generated ^[1].

However, there are mechanisms in place to optimize performance, such as caching queries to reduce processing time. The Cache class in pandasai/helpers/cache.py is designed to cache queries, which can save time and money by reusing parts of the prompt ^[2].

Additionally, PandasAI employs modular prompts for specific tasks, which helps in breaking down the overall process into smaller, manageable parts. Examples include CheckIfRelevantToConversationPrompt, ClarificationQuestionPrompt, ExplainPrompt, and RephraseQueryPrompt ^[3].

These strategies aim to balance the need for comprehensive prompts with performance optimization.

Details

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[gventuri]

@colorwlof PandasAI is meant to analyze data generating python code. To do so, it needs to provide enough context to the LLM, in a way that the code generated can be in line with the quality standard we are trying to get.

I am not sure I get the point of separating. If you separate it, you'll still have multiple steps and each step would probably need the context of the previous steps, so you should pass the whole conversation. Based on my experience, using GPT-3.5turbo PandasAI usually takes 3 to 6 seconds. Which LLM are you using?