Added user-agent headers to avoid reject of default bot like python-requests (open-edge-platform#996)

Co-authored-by: Raghu Bhat <[email protected]>

Author: Supriya-Krishnamurthi

microservices/document-ingestion/pgvector/app/main.py

@@ -389,7 +389,7 @@ async def delete_urls(
     url: Optional[str] = None, delete_all: Optional[bool] = False
 ) -> None:
     """
-    Delete a document or all documents from storage and their embeddings from Vector DB.
+    Delete a URL or all URLs from storage and their embeddings from Vector DB.
 
     Args:
         url (str): URL to be deleted

microservices/document-ingestion/pgvector/app/url.py

@@ -5,6 +5,7 @@
 import psycopg
 import ipaddress
 import socket
+import os
 from urllib.parse import urlparse
 from http import HTTPStatus
 from fastapi import HTTPException
@@ -117,6 +118,11 @@ def ingest_url_to_pgvector(url_list: List[str]) -> None:
             HTML parsing, or any other errors during the ingestion process.
     """
 
+    default_user_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36"
+
+    headers = {
+        "User-Agent": os.getenv("USER_AGENT_HEADER", default_user_agent)
+    }
 
     try:
         invalid_urls = 0
@@ -132,7 +138,7 @@ def ingest_url_to_pgvector(url_list: List[str]) -> None:
                     adapter = requests.adapters.HTTPAdapter()
                     session.mount("http://", adapter)
                     session.mount("https://", adapter)
-                    response = session.get(url, timeout=5, allow_redirects=False)
+                    response = session.get(url, headers=headers, timeout=5, allow_redirects=True)
 
                 if response.status_code != HTTPStatus.OK:
                     logger.info(f"Failed to fetch URL: {url} with status code {response.status_code}")
@@ -182,7 +188,7 @@ def ingest_url_to_pgvector(url_list: List[str]) -> None:
                 logger.error(f"Error while parsing HTML content for URL - {url}: {e}")
                 raise HTTPException(status_code=HTTPStatus.INTERNAL_SERVER_ERROR, detail=f"Error while parsing URL")
 
-            logger.info(f"[ ingest url ] url: {url} content: {content}")
+            logger.info(f"[ ingest url ] url: {url} content: {content} headers: {headers}")
             metadata = [{"url": url}]
 
             chunks = text_splitter.split_text(content)

microservices/document-ingestion/pgvector/docs/get-started.md

@@ -43,6 +43,13 @@ export PGDB_PASSWD=<user_db_password>
 export PGDB_NAME=<user_db_name>
 export PGDB_INDEX=<user_db_index>
 
+# HTTP request configuration (optional)
+# Set USER_AGENT_HEADER to define a custom User-Agent string for outgoing HTTP requests.
+# If not set, a robust default User-Agent is automatically applied
+# Setting a clear and descriptive User-Agent helps external servers identify the application and
+# reduces the chance of requests being treated as bot traffic.
+export USER_AGENT_HEADER=<your_user_agent_string>
+
 # OPTIONAL - If user wants to push the built images to a remote container registry, user needs to name the images accordingly. For this, image name should include the registry URL as well. To do this, set the following environment variable from shell. Please note that this URL will be prefixed to application name and tag to form the final image name.
 
 export CONTAINER_REGISTRY_URL=<user_container_registry_url>
@@ -103,7 +110,9 @@ This method provides the fastest way to get started with the microservice.
 2. Examples of expected outputs for validation.
 -->
 
-## First Use: Running a Predefined Task
+## Application Usage:
+
+## Type 1: Upload Files
 
 Try uploading a sample PDF file and verify that the embeddings and files are stored. Run the commands from the same shell as where the environment variables are set.
 
@@ -135,6 +144,53 @@ Try uploading a sample PDF file and verify that the embeddings and files are sto
    rm -rf ./minimal-document.pdf
    ```
 
+## Type 2: Upload URLs
+
+Try uploading web page URLs and verify that the embeddings are created and stored. Run the commands from the same shell as where the environment variables are set.
+
+ > **Note**: This URL ingestion microservice works best with pages that are not heavily reliant on JavaScript such as Wikipedia, which serve as ideal URL input sources. For JavaScript-intensive pages (social media feeds, Single Page Applications), the API may indicate a successful request but the actual content might not be captured. Such pages should be avoided or handled separately.
+ 
+1. **Get stored URLs**:
+   Retrieve a list of all URLs that have been processed and stored in the system.
+   ```bash
+   curl -X 'GET' \
+     "http://${host_ip}:${DATAPREP_HOST_PORT}/urls" \
+     -H 'accept: application/json'
+   ```
+
+2. **Upload URLs to create and store embeddings**:
+   Submit one or more URLs to be processed for embedding creation.
+   ```bash
+   curl -X 'POST' \
+     "http://${host_ip}:${DATAPREP_HOST_PORT}/urls" \
+     -H 'accept: application/json' \
+     -H 'Content-Type: application/json' \
+     -d '[
+     "https://en.wikipedia.org/wiki/Fiat",
+     "https://en.wikipedia.org/wiki/Lunar_eclipse"
+   ]'
+   ```
+
+3. **Verify the URLs were processed**:
+   Check that the URLs were successfully processed and stored.
+   ```bash
+   curl -X 'GET' \
+     "http://${host_ip}:${DATAPREP_HOST_PORT}/urls" \
+     -H 'accept: application/json'
+   ```
+   Expected output: A JSON response with the list of processed URLs should be printed.
+
+4. **Delete a specific URL or all URLs**:
+   Get the URL from the GET call response in step 3 and use it in the DELETE request below.
+   ```bash
+   curl -X 'DELETE' \
+     "http://${host_ip}:${DATAPREP_HOST_PORT}/urls?url=<url_to_be_deleted>&delete_all=false" \
+     -H 'accept: */*'
+   ```
+
+   **Note**:
+   - Optionally set `delete_all=true` if you want to delete all URLs from the database instead of a specific URL
+
 ## Advanced Setup Options
 
 To customize the microservice, refer to [customization documentation](./how-to-customize.md).

sample-applications/chat-question-and-answer/docs/user-guide/overview-architecture.md

@@ -35,6 +35,9 @@ ChatQ&A application is a combination of the core LangChain application logic tha
 1. **Input Sources**:
    - **Documents**: The document ingestion microservice supports ingesting documents in various formats. Supported formats are word and pdf.    
    - **Web pages**: Contents of accessible web pages can also be parsed and used as input for the RAG pipeline.
+     
+     > **Note**: This application works best with non–JavaScript-heavy pages (e.g., Wikipedia, blogs, news sites) that render most of their content directly in HTML. JavaScript-heavy pages (e.g., social media platforms, single-page applications) load content dynamically via JavaScript, so their raw HTML often lacks useful text. The current implementation only parses raw HTML and does not execute JavaScript, so such pages may return incomplete or inaccurate results and should be avoided or handled separately.
+
 2. **Create the context**
    - **Upload input documents and web links**: The UI microservice allows the developer to interact with the ChatQ&A backend. It provides the interface to upload the documents and weblinks on which the RAG pipeline will be executed. The documents are uploaded and stored in object store. MinIO is the database used for object store.
    - **Convert to embeddings space**: The ChatQ&A backend microservice creates the embeddings out of the uploaded documents and web pages using the document ingestion microservice. The Embeddings microservice is used to create the embeddings. The embeddings are stored in a vector database. PGVector is used in the sample application.