Support vision input for Planner (#472)

- Modified message formatting to support vision input for OpenAI API
- Added a role ImageReader to process input images so the Planner can
get the URL/content of the image

Author: liqul

README.md

@@ -23,6 +23,7 @@ Unlike many agent frameworks that only track the chat history with LLMs in text,
 
 
 ## 🆕 News
+- 📅2025-03-13: TaskWeaver now supports vision input for the Planner role. Please check the [vision input](https://microsoft.github.io/TaskWeaver/blog/vision) for more details.👀
 - 📅2025-01-16: TaskWeaver has been enhanced with an experimental role called [Recepta](https://microsoft.github.io/TaskWeaver/blog/reasoning) for its reasoning power.🧠
 - 📅2024-12-23: TaskWeaver has been integrated with the [AgentOps](https://microsoft.github.io/TaskWeaver/docs/observability) for better observability and monitoring.🔍
 - 📅2024-09-13: We introduce the shared memory to store information that is shared between the roles in TaskWeaver. Please check the [memory](https://microsoft.github.io/TaskWeaver/docs/memory) for more details.🧠
@@ -31,7 +32,7 @@ Unlike many agent frameworks that only track the chat history with LLMs in text,
 - 📅2024-05-07: We have added two blog posts on [Evaluating a LLM agent](https://microsoft.github.io/TaskWeaver/blog/evaluation) and [Adding new roles to TaskWeaver](https://microsoft.github.io/TaskWeaver/blog/role) in the documentation.📝
 - 📅2024-03-28: TaskWeaver now offers all-in-one Docker image, providing a convenient one-stop experience for users. Please check the [docker](https://microsoft.github.io/TaskWeaver/docs/usage/docker) for more details.🐳
 - 📅2024-03-27: TaskWeaver now switches to `container` mode by default for code execution. Please check the [code execution](https://microsoft.github.io/TaskWeaver/docs/code_execution) for more details.🐳
-- 📅2024-03-07: TaskWeaver now supports configuration of different LLMs for various components, such as the Planner and CodeInterpreter. Please check the [multi-llm](https://microsoft.github.io/TaskWeaver/docs/llms/multi-llm) for more details.🔗
+<!-- - 📅2024-03-07: TaskWeaver now supports configuration of different LLMs for various components, such as the Planner and CodeInterpreter. Please check the [multi-llm](https://microsoft.github.io/TaskWeaver/docs/llms/multi-llm) for more details.🔗 -->
 <!-- - 📅2024-03-04: TaskWeaver now supports a [container](https://microsoft.github.io/TaskWeaver/docs/code_execution) mode, which provides a more secure environment for code execution.🐳 -->
 <!-- - 📅2024-02-28: TaskWeaver now offers a [CLI-only](https://microsoft.github.io/TaskWeaver/docs/advanced/cli_only) mode, enabling users to interact seamlessly with the Command Line Interface (CLI) using natural language.📟 -->
 <!-- - 📅2024-02-01: TaskWeaver now has a plugin [document_retriever](https://github.com/microsoft/TaskWeaver/blob/main/project/plugins/README.md#document_retriever) for RAG based on a knowledge base.📚 -->
@@ -43,7 +44,8 @@ Unlike many agent frameworks that only track the chat history with LLMs in text,
 <!-- - 📅2023-12-21: TaskWeaver now supports a number of LLMs, such as LiteLLM, Ollama, Gemini, and QWen🎈.) -->
 <!-- - 📅2023-12-21: TaskWeaver Website is now [available]&#40;https://microsoft.github.io/TaskWeaver/&#41; with more documentations.) -->
 <!-- - 📅2023-12-12: A simple UI demo is available in playground/UI folder, try it [here](https://microsoft.github.io/TaskWeaver/docs/usage/webui)! -->
-<!-- - 📅2023-11-30: TaskWeaver is released on GitHub🎈.  -->
+- ......
+- 📅2023-11-30: TaskWeaver is released on GitHub🎈.  
 
 
 ## 💥 Highlights
@@ -68,7 +70,6 @@ We are looking forward to your contributions to make TaskWeaver better.
 - [ ] Support for prompt template management
 - [ ] Better plugin experiences, such as displaying updates or stopping in the middle of running the plugin and user confirmation before running the plugin
 - [ ] Async interaction with LLMs
-- [ ] Support for vision input for Roles such as the Planner and CodeInterpreter
 - [ ] Support for remote code execution
 
 

project/examples/code_generator_examples/example1-codeinterpreter.yaml renamed to project/examples/code_generator_examples/example-codeinterpreter-default-1.yaml

@@ -498,7 +498,7 @@ def _reset_session(self, first_session: bool = False):
             self.session.stop()
             self.session = self.app.get_session()
 
-        self._system_message("--- new session starts ---")
+        self._system_message("--- new session started ---")
         self._assistant_message(
             "I am TaskWeaver, an AI assistant. To get started, could you please enter your request?",
         )

project/examples/code_generator_examples/example2-codeinterpreter.yaml renamed to project/examples/code_generator_examples/example-codeinterpreter-default-2.yaml

@@ -251,7 +251,7 @@ def compose_conversation(
                     # for code correction
                     user_message += self.user_message_head_template.format(
                         FEEDBACK=format_code_feedback(post),
-                        MESSAGE=f"{post.get_attachment(AttachmentType.revise_message)[0]}",
+                        MESSAGE=f"{post.get_attachment(AttachmentType.revise_message)[0].content}",
                     )
 
                     assistant_message = self.post_translator.post_to_raw_text(

project/examples/planner_examples/example-planner.yaml renamed to project/examples/planner_examples/example-planner-default-1.yaml

@@ -60,9 +60,12 @@ def reply(
             prompt_log_path=prompt_log_path,
         )
 
-        code = post_proxy.post.get_attachment(type=AttachmentType.reply_content)[0]
+        code = post_proxy.post.get_attachment(type=AttachmentType.reply_content)[0].content
         if len(code) == 0:
-            post_proxy.update_message(post_proxy.post.get_attachment(type=AttachmentType.thought)[0], is_end=True)
+            post_proxy.update_message(
+                post_proxy.post.get_attachment(type=AttachmentType.thought)[0].content,
+                is_end=True,
+            )
             return post_proxy.end()
 
         code_to_exec = "! " + code

project/examples/planner_examples/example-planner-2.yaml renamed to project/examples/planner_examples/example-planner-default-2.yaml

@@ -78,7 +78,7 @@ def reply(
             return post_proxy.end()
 
         functions = json.loads(
-            post_proxy.post.get_attachment(type=AttachmentType.function)[0],
+            post_proxy.post.get_attachment(type=AttachmentType.function)[0].content,
         )
         if len(functions) > 0:
             code: List[str] = []