Skip to content

Commit b768543

Browse files
sachitolanisachitolani
authored
publish-blog-browser-use-milvus (#377)
1 parent 9927f15 commit b768543

File tree

1 file changed

+136
-0
lines changed

1 file changed

+136
-0
lines changed

‎blog/en/ai-for-smarter-browsing-filtering-web-content-with-pirxtral-milvus-browser-use.md‎

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
---
2+
id: ai-for-smarter-browsing-filtering-web-content-with-pixtral-milvus-browser-use.md
3+
title: AI for Smarter Browsing: Filtering Web Content with Pixtral, Milvus, and Browser Use
4+
author: Stephen Batifol
5+
date: 2025-02-25
6+
desc: Learn how to build an intelligent assistant that filters content by combining Pixtral for image analysis, Milvus vector database for storage, and Browser Use for web navigation.
7+
cover: assets.zilliz.com/Multimodal_Semantic_Search_with_Images_and_Text_1_3da9b83015.png
8+
tag: Engineering
9+
tags: Vector Database Milvus, AI Content Filtering, Pixtral Image Analysis, Browser Use Web Navigation, Intelligent Agent Development
10+
recommend: true
11+
canonicalUrl: https://milvus.io/blog/ai-for-smarter-browsing-filtering-web-content-with-pixtral-milvus-browser-use.md
12+
13+
As a Developer Advocate for Milvus, I spend a lot of time on Socials, listening to what people have to say about us and if I can help as well. There is a slight clash of worlds though when you look for "Milvus". It is both a Vector DB and genus of bird, meaning that one moment I'm deep in a thread about vector similarity algorithms, the next I'm admiring stunning photographs of black birds flying through the sky.
14+
15+
While both topics are interesting, mixing them up isn't really helpful in my case, what if there was a smart way to solve this problem without me having to check manually?
16+
17+
Let's build something smarter - by combining visual understanding with context awareness, we can build an assistant that knows the difference between a black kite's migration patterns and a new article from us.
18+
19+
20+
## The tech stack
21+
22+
We combine three different technologies:
23+
24+
- **Browser-Use:** This tool navigates various websites (e.g., Twitter) to fetch content.
25+
- **Pixtral**: A vision-language model that analyzes images and context. In this example, it distinguishes between a technical diagram about our Vector DB and a stunning bird photograph.
26+
- **Milvus:** A high performance and open-source Vector DB. His is where we will store the relevant posts for later querying.
27+
28+
![](https://assets.zilliz.com/the_tech_stack_ad695ccf9e.png)
29+
30+
31+
## Seeing it in action
32+
33+
Let's have a look at those 2 posts:
34+
35+
![](https://assets.zilliz.com/Langchian_tweet_1_with_Milvus_f2bd988503.png)![](https://assets.zilliz.com/Tweet_2_with_Bird_4b534efced.png)
36+
37+
For the one on the left, Pixtral recognizes that this is a post about Milvus, the Vector DB. It mentioned some implementation details, but also an image showing a system diagram, indicating that this is indeed, about the Vector DB. We can see below that Pixtral also thinks the same :)
38+
39+
```Shell
40+
INFO [src.agent.custom_agent] 🧠 New Memory: 1. The post by LangChainAI discusses the 'GraphRAG Agent' which uses Neo4j and Milvus vector search to enhance RAG systems.
41+
2. Replies include praise for the use of Milvus in intelligent retrieval and vector search.
42+
3. Specific mention of Milvus as a key component in innovative RAG systems.
43+
4. No content related to birds or photography was found.
44+
INFO [src.agent.custom_agent] ⏳ Task Progress:
45+
1. Navigated to the post URL. 2. Extracted relevant post content. 3. Analyzed the content for relevance to Milvus, the Vector DB.
46+
INFO [src.agent.custom_agent] 📋 Future Plans:
47+
1. Summarize the findings and confirm the relevance to Milvus. 2. Complete the task by providing a clear and concise report.
48+
INFO [src.agent.custom_agent] 🤔 Thought: The task required analyzing the post content for relevance to Milvus, the Vector DB, ignoring any content about birds or photography. The extracted content is entirely relevant to Milvus and its applications in vector search and RAG systems. I will now summarize the findings and complete the task.
49+
INFO [src.agent.custom_agent] 🎯 Summary: The post and its replies are relevant to Milvus, the Vector DB, focusing on its use in intelligent retrieval and vector search. No content about birds or photography was found.
50+
INFO [src.agent.custom_agent] 🛠️ Action 1/1: {"done":{"text":"The post by LangChainAI discusses the 'GraphRAG Agent,' which uses Neo4j and Milvus vector search to enhance RAG systems. Replies highlight the innovative use of Milvus for intelligent retrieval and vector search, with no mention of birds or photography. The content is entirely relevant to Milvus, the Vector DB."}}
51+
INFO [src.agent.custom_agent] 🧠 All Memory:
52+
1. The post by LangChainAI discusses the 'GraphRAG Agent' which uses Neo4j and Milvus vector search to enhance RAG systems.
53+
2. Replies focus on the innovative use of Milvus for intelligent retrieval and vector search alongside other technologies.
54+
3. No posts or replies are related to birds or photography.
55+
1. The post by LangChainAI discusses the 'GraphRAG Agent' which uses Neo4j and Milvus vector search to enhance RAG systems.
56+
2. Replies include praise for the use of Milvus in intelligent retrieval and vector search.
57+
3. Specific mention of Milvus as a key component in innovative RAG systems.
58+
4. No content related to birds or photography was found.
59+
```
60+
61+
The one on the right on the other hand isn't, we can see that this picture, as beautiful as it is, isn't about a Vector DB. We can see a bird flying in the sky, therefore, Pixtral will consider this one irrelevant.
62+
63+
```Shell
64+
INFO [src.agent.custom_agent] 🧠 New Memory: The post and comments primarily discuss photography and birds. No references to Milvus the Vector Database are found.
65+
INFO [src.agent.custom_agent] ⏳ Task Progress:
66+
Navigated to the post. Analyzed the content for relevance to Milvus, the Vector DB. No relevant information found.
67+
INFO [src.agent.custom_agent] 🤔 Thought: The content of the post and comments only discusses birds and photography. Since the task specifies ignoring such topics, there is no relevant information for Milvus, the Vector DB. The task can now be concluded.
68+
INFO [src.agent.custom_agent] 🎯 Summary: The post does not contain relevant information for Milvus, the Vector DB. I will conclude the task.
69+
INFO [src.agent.custom_agent] 🛠️ Action 1/1: {"done":{"text":"The post and comments focus on birds and photography. No relevant information related to Milvus, the Vector DB, was found."}}
70+
INFO [src.agent.custom_agent] 🧠 All Memory:
71+
The post and comments primarily discuss photography and birds. No references to Milvus the Vector Database are found.
72+
```
73+
74+
Now that we have filtered out the posts we don't want, we can save the relevant ones in Milvus. Making it possible to query them later using either Vector Search or Full Text Search.
75+
76+
This content is only supported in a Lark Docs
77+
78+
79+
## Storing Data in Milvus
80+
81+
[Dynamic Fields](https://milvus.io/docs/enable-dynamic-field.md#Dynamic-Field) are a must in this case because it's not always possible to respect the schema that Milvus expects. With Milvus, you just use `enable_dynamic_field=True` when creating your schema, and that's it. Here is a code snippet to showcase the process:
82+
83+
```Python
84+
from pymilvus import MilvusClient
85+
86+
# Connect to Milvus
87+
client = MilvusClient(uri="http://localhost:19530")
88+
89+
# Create a Schema that handles Dynamic Fields
90+
schema = self.client.create_schema(enable_dynamic_field=True)
91+
schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True, auto_id=True)
92+
schema.add_field(field_name="text", datatype=DataType.VARCHAR, max_length=65535, enable_analyzer=True)
93+
schema.add_field(field_name="vector", datatype=DataType.FLOAT_VECTOR, dim=384)
94+
schema.add_field(field_name="sparse", datatype=DataType.SPARSE_FLOAT_VECTOR)
95+
```
96+
97+
Then we define the data we want to have access to:
98+
99+
```Python
100+
# Prepare data with dynamic fields
101+
data = {
102+
'text': content_str,
103+
'vector': embedding,
104+
'url': url,
105+
'type': content_type,
106+
'metadata': json.dumps(metadata or {})
107+
}
108+
109+
# Insert into Milvus
110+
self.client.insert(
111+
collection_name=self.collection_name,
112+
data=[data]
113+
)
114+
```
115+
116+
This simple setup means you don’t have to worry about every field being perfectly defined upfront. Just set up the schema to allow for dynamic additions and let Milvus do the heavy lifting.
117+
118+
119+
## Conclusion
120+
121+
By combining Browser Use's web navigation, Pixtral's visual understanding, and Milvus's efficient storage, we've built an intelligent assistant that truly understands context. Now I am using it to distinguish between birds and vector DB, but the same approach could help with another problem you may be facing.
122+
123+
On my end, I wanna continue working on agents that can help me in my daily work in order to reduce my cognitive load 😌
124+
125+
126+
## We'd Love to Hear What You Think!
127+
128+
If you like this blog post, please consider:
129+
130+
- ⭐ Giving us a star on [GitHub](https://github.com/milvus-io/milvus)
131+
- 💬 Joining our [Milvus Discord community](https://discord.gg/FG6hMJStWu) to share your experiences or if you need help building Agents
132+
- 🔍 Exploring our [Bootcamp repository](https://github.com/milvus-io/bootcamp) for examples of applications using Milvus
133+
134+
135+
136+
---

0 commit comments

Comments
 (0)