Skip to content

Conversation

@mare5x
Copy link
Contributor

@mare5x mare5x commented Nov 11, 2025

Closes #50

The Executor can now return "output modality hints" which are used to automatically generate a plot after Pipe.ask is called in eager mode. Currently, there is no change in lazy mode.

thread.ask(...) will now automatically generate a plot after getting the Executor result if the Executor returns a visualization prompt.

The output displays all available modalities as dropdowns:
image

Unfortunately, Jupyter notebooks in PyCharm can be quite buggy when showing HTML elements.

@mare5x mare5x changed the title Automatic output modality detection with eager mode Automatic output modality detection in eager mode Nov 11, 2025
@mare5x mare5x linked an issue Nov 11, 2025 that may be closed by this pull request


class OutputModalityHints(BaseModel):
"""Hints on how to present the execution results."""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From this description I was expecting that it will define the main modality. But it has information only about the visualization.
Probably we should define list of possible modalities and list of main modalities for particular Result. In this case main modalities will be used in repr

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid point. The current idea was that in Pipe we can just show results without any LLM:

  • if a plot is available, show it
  • if a dataframe is available, show it
  • else, show text

But fair enough, I will add another hint as you suggested for the modality priorities so that the Executor can suggest a different priority order.

Also btw, the Executor does not know if the visualization will actually be generated, as that depends on the Pipe auto_output_modality setting.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After thinking about it more, I decided to keep the hints only for visualizations for now. I think the solution with dropdowns for all modalities solves the issue of deciding which modality to show, as we can just show them all :)

And to give Executors the flexibility to change how to display results, I moved ExecutionResult rendering from Pipe to ExecutionResult._repr_mimebundle_.

Base automatically changed from mhostnik/eager-mode to main November 11, 2025 14:45
@mare5x mare5x force-pushed the mhostnik/auto-modality branch from 2db1504 to f7b061f Compare November 11, 2025 14:47
@mare5x mare5x force-pushed the mhostnik/auto-modality branch from b0a5b54 to 030ff40 Compare November 12, 2025 12:00
# Conflicts:
#	databao/agents/lighthouse/agent.py
#	databao/agents/react_duckdb/agent.py
@mare5x mare5x requested a review from dmzolotarev November 14, 2025 11:27
@mare5x
Copy link
Contributor Author

mare5x commented Nov 14, 2025

@dmzolotarev These are the two ways to disable automatic plotting:

session = databao.open_session("my_session", default_auto_output_modality=False)
thread = session.thread(auto_output_modality=False)

I've also made it so a plot will be generated only if the agent decides to output a visualization prompt. This way, fewer plots will be generated automatically.

@mare5x mare5x merged commit 1ebb5ee into main Nov 17, 2025
2 checks passed
@mare5x mare5x deleted the mhostnik/auto-modality branch November 17, 2025 08:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Automatically detect output modality with eager mode

3 participants