-
Notifications
You must be signed in to change notification settings - Fork 2
Automatic output modality detection in eager mode #87
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
databao/core/executor.py
Outdated
|
|
||
|
|
||
| class OutputModalityHints(BaseModel): | ||
| """Hints on how to present the execution results.""" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From this description I was expecting that it will define the main modality. But it has information only about the visualization.
Probably we should define list of possible modalities and list of main modalities for particular Result. In this case main modalities will be used in repr
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Valid point. The current idea was that in Pipe we can just show results without any LLM:
- if a plot is available, show it
- if a dataframe is available, show it
- else, show text
But fair enough, I will add another hint as you suggested for the modality priorities so that the Executor can suggest a different priority order.
Also btw, the Executor does not know if the visualization will actually be generated, as that depends on the Pipe auto_output_modality setting.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After thinking about it more, I decided to keep the hints only for visualizations for now. I think the solution with dropdowns for all modalities solves the issue of deciding which modality to show, as we can just show them all :)
And to give Executors the flexibility to change how to display results, I moved ExecutionResult rendering from Pipe to ExecutionResult._repr_mimebundle_.
2db1504 to
f7b061f
Compare
b0a5b54 to
030ff40
Compare
# Conflicts: # databao/agents/lighthouse/agent.py # databao/agents/react_duckdb/agent.py
|
@dmzolotarev These are the two ways to disable automatic plotting: session = databao.open_session("my_session", default_auto_output_modality=False)
thread = session.thread(auto_output_modality=False)I've also made it so a plot will be generated only if the agent decides to output a visualization prompt. This way, fewer plots will be generated automatically. |
Closes #50
The Executor can now return "output modality hints" which are used to automatically generate a plot after
Pipe.askis called in eager mode. Currently, there is no change in lazy mode.thread.ask(...)will now automatically generate a plot after getting the Executor result if the Executor returns a visualization prompt.The output displays all available modalities as dropdowns:

Unfortunately, Jupyter notebooks in PyCharm can be quite buggy when showing HTML elements.