Merge pull request #3 from kagent-dev/denis-github-source

Adding github source

Author: djannot

README.md

@@ -1,12 +1,13 @@
 # Doc2Vec
 
-This project provides a configurable tool (`doc2vec`) to crawl specified websites (typically documentation sites), extract relevant content, convert it to Markdown, chunk it intelligently, generate vector embeddings using OpenAI, and store the chunks along with their embeddings in a vector database (SQLite with `sqlite-vec` or Qdrant).
+This project provides a configurable tool (`doc2vec`) to crawl specified websites (typically documentation sites) and GitHub repositories, extract relevant content, convert it to Markdown, chunk it intelligently, generate vector embeddings using OpenAI, and store the chunks along with their embeddings in a vector database (SQLite with `sqlite-vec` or Qdrant).
 
 The primary goal is to prepare documentation content for Retrieval-Augmented Generation (RAG) systems or semantic search applications.
 
 ## Key Features
 
 *   **Website Crawling:** Recursively crawls websites starting from a given base URL.
+*   **GitHub Issues Integration:** Retrieves GitHub issues and comments, processing them into searchable chunks.
 *   **Content Extraction:** Uses Puppeteer for rendering JavaScript-heavy pages and `@mozilla/readability` to extract the main article content.
 *   **HTML to Markdown:** Converts extracted HTML to clean Markdown using `turndown`, preserving code blocks and basic formatting.
 *   **Intelligent Chunking:** Splits Markdown content into manageable chunks based on headings and token limits, preserving context.
@@ -15,8 +16,9 @@ The primary goal is to prepare documentation content for Retrieval-Augmented Gen
     *   **SQLite:** Using `better-sqlite3` and the `sqlite-vec` extension for efficient vector search.
     *   **Qdrant:** A dedicated vector database, using the `@qdrant/js-client-rest`.
 *   **Change Detection:** Uses content hashing to detect changes and only re-embeds and updates chunks that have actually been modified.
+*   **Incremental Updates:** For GitHub sources, tracks the last run date to only fetch new or updated issues.
 *   **Cleanup:** Removes obsolete chunks from the database corresponding to pages that are no longer found during a crawl.
-*   **Configuration:** Driven by a YAML configuration file (`config.yaml`) specifying sites, database types, metadata, and other parameters.
+*   **Configuration:** Driven by a YAML configuration file (`config.yaml`) specifying sites, repositories, database types, metadata, and other parameters.
 *   **Structured Logging:** Uses a custom logger (`logger.ts`) with levels, timestamps, colors, progress bars, and child loggers for clear execution monitoring.
 
 ## Prerequisites
@@ -25,6 +27,7 @@ The primary goal is to prepare documentation content for Retrieval-Augmented Gen
 *   **npm:** Node Package Manager (usually comes with Node.js).
 *   **TypeScript:** As the project is written in TypeScript (`ts-node` is used for execution via `npm start`).
 *   **OpenAI API Key:** You need an API key from OpenAI to generate embeddings.
+*   **GitHub Personal Access Token:** Required for accessing GitHub issues (set as `GITHUB_PERSONAL_ACCESS_TOKEN` in your environment).
 *   **(Optional) Qdrant Instance:** If using the `qdrant` database type, you need a running Qdrant instance accessible from where you run the script.
 *   **(Optional) Build Tools:** Dependencies like `better-sqlite3` and `sqlite-vec` might require native compilation, which could necessitate build tools like `python`, `make`, and a C++ compiler (like `g++` or Clang) depending on your operating system.
 
@@ -56,50 +59,81 @@ Configuration is managed through two files:
     # Required: Your OpenAI API Key
     OPENAI_API_KEY="sk-..."
 
+    # Required for GitHub sources
+    GITHUB_PERSONAL_ACCESS_TOKEN="ghp_..."
+
     # Optional: Required only if using Qdrant
     QDRANT_API_KEY="your-qdrant-api-key"
     ```
 
 2.  **`config.yaml` file:**
-    This file defines the sites to crawl and how to process them. Create a `config.yaml` file (or use a different name and pass it as an argument).
+    This file defines the sources to process and how to handle them. Create a `config.yaml` file (or use a different name and pass it as an argument).
 
     **Structure:**
 
-    *   `sites`: An array of site configurations.
+    *   `sources`: An array of source configurations.
+        *   `type`: Either `'website'` or `'github'`
+        
+        For websites (`type: 'website'`):
         *   `url`: The starting URL for crawling the documentation site.
-        *   `database_type`: Specifies the storage backend (`sqlite` or `qdrant`).
+        
+        For GitHub repositories (`type: 'github'`):
+        *   `repo`: Repository name in the format `'owner/repo'` (e.g., `'istio/istio'`).
+        *   `start_date`: (Optional) Starting date to fetch issues from (e.g., `'2025-01-01'`).
+        
+        Common configuration for both types:
         *   `product_name`: A string identifying the product (used in metadata).
         *   `version`: A string identifying the product version (used in metadata).
-        *   `max_size`: Maximum **raw HTML content size** (in characters) fetched by Puppeteer. If a page's initial HTML exceeds this, it will be skipped *before* performing expensive DOM parsing, Readability, and Markdown conversion. Recommending 1MB (1048576).
-        *   `database_params`: Parameters specific to the chosen `database_type`.
-            *   For `sqlite`:
-                *   `db_path`: (Optional) Path to the SQLite database file. Defaults to `./<product_name>-<version>.db`.
-            *   For `qdrant`:
-                *   `qdrant_url`: (Optional) URL of your Qdrant instance. Defaults to `http://localhost:6333`.
-                *   `qdrant_port`: (Optional) Port for the Qdrant REST API. Defaults to `443` if `qdrant_url` starts with `https`, otherwise `6333`.
-                *   `collection_name`: (Optional) Name of the Qdrant collection to use. Defaults to `<product_name>_<version>` (lowercased, spaces replaced with underscores).
+        *   `max_size`: Maximum raw content size (in characters). For websites, this limits the raw HTML fetched by Puppeteer. Recommending 1MB (1048576).
+        *   `database_config`: Configuration for the database.
+            *   `type`: Specifies the storage backend (`'sqlite'` or `'qdrant'`).
+            *   `params`: Parameters specific to the chosen database type.
+                *   For `sqlite`:
+                    *   `db_path`: (Optional) Path to the SQLite database file. Defaults to `./<product_name>-<version>.db`.
+                *   For `qdrant`:
+                    *   `qdrant_url`: (Optional) URL of your Qdrant instance. Defaults to `http://localhost:6333`.
+                    *   `qdrant_port`: (Optional) Port for the Qdrant REST API. Defaults to `443` if `qdrant_url` starts with `https`, otherwise `6333`.
+                    *   `collection_name`: (Optional) Name of the Qdrant collection to use. Defaults to `<product_name>_<version>` (lowercased, spaces replaced with underscores).
 
     **Example (`config.yaml`):**
     ```yaml
-    sites:
-      - url: 'https://argo-cd.readthedocs.io/en/stable/'
-        database_type: 'sqlite'
+    sources:
+      # Website source example
+      - type: 'website'
         product_name: 'argo'
         version: 'stable'
+        url: 'https://argo-cd.readthedocs.io/en/stable/'
         max_size: 1048576
-        database_params:
-          db_path: './vector-dbs/argo-cd.db'
-
-      - url: 'https://istio.io/latest/docs/'
-        database_type: 'qdrant'
+        database_config:
+          type: 'sqlite'
+          params:
+            db_path: './vector-dbs/argo-cd.db'
+
+      # GitHub repository source example
+      - type: 'github'
+        product_name: 'istio'
+        version: 'latest'
+        repo: 'istio/istio'
+        start_date: '2025-01-01'
+        max_size: 1048576
+        database_config:
+          type: 'sqlite'
+          params:
+            db_path: './istio-issues.db'
+      
+      # Qdrant example
+      - type: 'website'
         product_name: 'Istio'
         version: 'latest'
+        url: 'https://istio.io/latest/docs/'
         max_size: 1048576
-        database_params:
-          qdrant_url: 'https://your-qdrant-instance.cloud'
-          qdrant_port: 6333
-          collection_name: 'istio_docs_latest'
-      # ... more sites
+        database_config:
+          type: 'qdrant'
+          params:
+            qdrant_url: 'https://your-qdrant-instance.cloud'
+            qdrant_port: 6333
+            collection_name: 'istio_docs_latest'
+      # ... more sources
     ```
 
 ## Usage
@@ -119,42 +153,49 @@ If no path is provided, the script defaults to looking for `config.yaml` in the
 The script will then:
 1.  Load the configuration.
 2.  Initialize the structured logger.
-3.  Iterate through each site defined in the config.
+3.  Iterate through each source defined in the config.
 4.  Initialize the specified database connection.
-5.  Crawl the site.
-6.  For each valid page: extract content, convert to Markdown, chunk, check for changes, generate embeddings (if needed), and store/update in the database.
+5.  Process each source according to its type:
+    - For websites: Crawl the site, extract content, convert to Markdown
+    - For GitHub repos: Fetch issues and comments, convert to Markdown
+6.  For all sources: Chunk content, check for changes, generate embeddings (if needed), and store/update in the database.
 7.  Cleanup obsolete chunks.
 8.  Output detailed logs.
 
 ## Database Options
 
-### SQLite (`database_type: 'sqlite'`)
+### SQLite (`database_config.type: 'sqlite'`)
 *   Uses `better-sqlite3` and `sqlite-vec`.
 *   Requires `db_path`.
 *   Native compilation might be needed.
 
-### Qdrant (`database_type: 'qdrant'`)
+### Qdrant (`database_config.type: 'qdrant'`)
 *   Uses `@qdrant/js-client-rest`.
 *   Requires `qdrant_url`, `qdrant_port`, `collection_name` and potentially `QDRANT_API_KEY`.
 
 ## Core Logic Flow
 
 1.  **Load Config:** Read and parse `config.yaml`.
 2.  **Initialize Logger:** Set up the structured logger.
-3.  **Iterate Sites:** For each site in the config:
+3.  **Iterate Sources:** For each source in the config:
     1.  **Initialize Database:** Connect to SQLite or Qdrant, create necessary tables/collections.
-    2.  **Crawl:**
-        *   Start at the base `url`.
-        *   Use Puppeteer (`processPage`) to fetch and render HTML.
-        *   Use Readability to extract main content.
-        *   Sanitize HTML.
-        *   Convert HTML to Markdown using Turndown.
-        *   Use `axios`/`cheerio` on the *original* fetched page (before Puppeteer) to find new links to add to the crawl queue.
-        *   Keep track of all visited URLs.
-    3.  **Process Content:** For each crawled page's Markdown:
+    2.  **Process by Source Type:**
+        - **For Websites:**
+          *   Start at the base `url`.
+          *   Use Puppeteer (`processPage`) to fetch and render HTML.
+          *   Use Readability to extract main content.
+          *   Sanitize HTML.
+          *   Convert HTML to Markdown using Turndown.
+          *   Use `axios`/`cheerio` on the *original* fetched page (before Puppeteer) to find new links to add to the crawl queue.
+          *   Keep track of all visited URLs.
+        - **For GitHub Repositories:**
+          *   Fetch issues and comments using the GitHub API.
+          *   Convert to formatted Markdown.
+          *   Track last run date to support incremental updates.
+    3.  **Process Content:** For each processed page or issue:
         *   **Chunk:** Split Markdown into smaller `DocumentChunk` objects based on headings and size.
         *   **Hash Check:** Generate a hash of the chunk content. Check if a chunk with the same ID exists in the DB and if its hash matches.
         *   **Embed (if needed):** If the chunk is new or changed, call the OpenAI API (`createEmbeddings`) to get the vector embedding.
         *   **Store:** Insert or update the chunk, metadata, hash, and embedding in the database (SQLite `vec_items` table or Qdrant collection).
-    4.  **Cleanup:** After crawling the site, query the database for all chunks associated with that site's URL prefix. Delete any chunks whose URLs were *not* in the set of visited URLs for the current run.
-4.  **Complete:** Log completion status.
+    4.  **Cleanup:** After processing, remove any obsolete chunks from the database.
+4.  **Complete:** Log completion status.