Skip to content

Latest commit

 

History

History
215 lines (166 loc) · 9.07 KB

README.md

File metadata and controls

215 lines (166 loc) · 9.07 KB

Todofy - Self-Hosted Task Management Tool

CI/CD Pipeline codecov

Todofy is a self-hosted task management tool designed to help you organize and prioritize your tasks efficiently. It's built as a collection of microservices communicating over gRPC, with email-driven task creation powered by Google Gemini LLM summarization.

🏗️ Architecture

graph TB
    %% External systems and users
    User[👤 User<br/>HTTP Client]
    Email[📧 Email System<br/>Cloudmailin Webhook]
    
    %% External services
    Gemini[🤖 Google Gemini<br/>LLM API]
    Mailjet[📬 Mailjet<br/>Email Service]
    Notion[📝 Notion<br/>Database API]
    Todoist[✅ Todoist<br/>Task API v1]
    
    %% Main HTTP Server
    Main[🌐 Todofy Main Server<br/>HTTP REST API<br/>Port: 8080<br/>Basic Auth + Rate Limiting]
    
    %% Microservices
    LLM[🧠 LLM Service<br/>gRPC Server<br/>Port: 50051]
    Todo[📋 Todo Service<br/>gRPC Server<br/>Port: 50052]
    DB[🗄️ Database Service<br/>gRPC Server<br/>Port: 50053<br/>SQLite Backend]
    
    %% API endpoints
    Summary[📊 /api/summary<br/>GET endpoint]
    UpdateTodo[📝 /api/v1/update_todo<br/>POST endpoint]
    Recommend[🏆 /api/recommendation<br/>GET endpoint<br/>?top=N]
    
    %% Data flow connections
    User -->|HTTPS GET| Summary
    User -->|HTTPS GET| Recommend
    Email -->|Webhook POST| UpdateTodo
    
    Summary --> Main
    UpdateTodo --> Main
    Recommend --> Main
    
    Main -->|gRPC Health Check| LLM
    Main -->|gRPC Health Check| Todo  
    Main -->|gRPC Health Check| DB
    
    %% Dedup cache flow: check DB first, then conditionally call LLM
    Main -->|CheckExist<br/>hash_id lookup| DB
    Main -.->|LLMSummaryRequest<br/>only on cache miss| LLM
    Main -->|TodoRequest| Todo
    Main -->|Write<br/>with hash_id| DB
    
    LLM -->|API Calls| Gemini
    Todo -->|Email Send| Mailjet
    Todo -->|Task Creation| Notion
    Todo -->|Task Creation| Todoist
    
    %% Service descriptions
    Main -.->|Container| MainContainer[🐳 ghcr.io/ziyixi/todofy:latest]
    LLM -.->|Container| LLMContainer[🐳 ghcr.io/ziyixi/todofy-llm:latest]
    Todo -.->|Container| TodoContainer[🐳 ghcr.io/ziyixi/todofy-todo:latest] 
    DB -.->|Container| DBContainer[🐳 ghcr.io/ziyixi/todofy-database:latest]
    
    %% Styling
    classDef external fill:#e1f5fe,stroke:#0277bd,stroke-width:2px
    classDef service fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
    classDef endpoint fill:#e8f5e8,stroke:#388e3c,stroke-width:2px
    classDef container fill:#fff3e0,stroke:#f57c00,stroke-width:2px,stroke-dasharray: 5 5
    
    class User,Email,Gemini,Mailjet,Notion,Todoist external
    class Main,LLM,Todo,DB service
    class Summary,UpdateTodo,Recommend endpoint
    class MainContainer,LLMContainer,TodoContainer,DBContainer container
Loading

✨ Features

  • Task Management: Core functionality for creating, updating, and managing tasks.
  • LLM Integration: Leverages Google Gemini models for email summarization with automatic model fallback (via todofy-llm service).
  • Cost Controls: Daily token limit with 24-hour sliding window (default: 3M tokens) to prevent runaway API costs, plus email content truncation (50K character hard limit).
  • Dedup Cache: SHA-256 hash-based deduplication — identical emails skip the expensive LLM call and reuse the cached summary from the database.
  • Task Recommendations: GET /api/recommendation?top=N queries recent 24h tasks, asks the LLM to pick the top-N most important ones (default 3, max 10), and returns structured JSON with rank, title, and reason for each.
  • Email/API Task Population: Allows tasks to be populated or managed via email or API interactions (via todofy-todo service).
  • Persistent Storage: Uses SQLite for storing task data with hash-indexed lookups (via todofy-database service).
  • Containerized Services: All components are containerized using Docker for easy deployment and scaling.
  • Comprehensive Testing: Unit tests, e2e tests with mock Gemini client injection, and Docker-based integration tests.

🧠 LLM Service Details

The LLM service uses Google Gemini for email summarization with several cost-control and reliability features:

Supported Models

Models are tried in priority order for automatic fallback:

  1. gemini-2.5-flash-lite (fastest, cheapest)
  2. gemini-2.5-flash (balanced)
  3. gemini-3-flash-preview (latest)

Additionally, gemini-2.5-pro is available when explicitly requested.

Cost Controls

Feature Default Description
Daily token limit 3,000,000 24-hour sliding window; configurable via --daily-token-limit flag (0 = unlimited)
Email content limit 50,000 chars Hard truncation of email body before LLM processing
Token counting Per-request Content is iteratively truncated (to 90%) until under the per-model token limit (1M tokens)
Dedup cache Always on SHA-256 hash of prompt + email content; duplicate emails return cached summary without LLM call

Configuration Flags

Flag Default Description
--port 50051 gRPC server port
--gemini-api-key (required) Google Gemini API key
--daily-token-limit 3000000 Max tokens per 24h sliding window (0 = unlimited)

🛠️ Services

The application is composed of the following services:

  1. Todofy (Main App)

    • Description: The primary user-facing application and HTTP API gateway.
    • Dockerfile: ./Dockerfile
    • Default Port: 8080 (configurable via PORT env var)
    • Image: ghcr.io/ziyixi/todofy:latest

  2. LLM Service (todofy-llm)

    • Description: Email summarization via Google Gemini with model fallback and daily token tracking.
    • Dockerfile: llm/Dockerfile
    • Default Port: 50051 (configurable via --port flag)
    • Image: ghcr.io/ziyixi/todofy-llm:latest

  3. Todo Service (todofy-todo)

    • Description: Manages task creation via Todoist (API v1), Notion, and email (Mailjet).
    • Dockerfile: todo/Dockerfile
    • Default Port: 50052 (configurable via --port flag)
    • Image: ghcr.io/ziyixi/todofy-todo:latest

  4. Database Service (todofy-database)

    • Description: Provides database access and management using SQLite. Supports Write, QueryRecent, and CheckExist (hash-based dedup lookup) RPCs.
    • Dockerfile: database/Dockerfile
    • Default Port: 50053 (configurable via PORT env var)
    • Image: ghcr.io/ziyixi/todofy-database:latest

🔄 CI/CD Pipeline

The CI/CD pipeline uses GitHub Actions with reusable workflows organized as a dependency graph:

graph LR
    T[Test] --> B[Build]
    L[Lint] --> B
    S[Security] --> B
    I[Integration Test] --> B
    T --> N[Notify]
    L --> N
    S --> N
    I --> N
Loading
Workflow Description
Test Runs go test -race with coverage, uploads to Codecov
Lint Runs golangci-lint
Security Runs gosec with SARIF upload to GitHub Security
Integration Test Builds all 4 Docker images and validates with health checks
Build Pushes Docker images to GHCR on main only when build-relevant files change (or manual dispatch)
Notify Reports pass/fail status

📦 GitHub Packages (GHCR)

Docker images for each service are automatically built and pushed to GitHub Container Registry (GHCR) by the CI/CD pipeline. You can pull them using:

  • docker pull ghcr.io/ziyixi/todofy:latest
  • docker pull ghcr.io/ziyixi/todofy-llm:latest
  • docker pull ghcr.io/ziyixi/todofy-todo:latest
  • docker pull ghcr.io/ziyixi/todofy-database:latest

🧪 Testing

Run all tests:

go test ./...

Run with coverage:

go test -race -coverprofile=coverage.out -covermode=atomic ./...
go tool cover -func=coverage.out

The LLM service includes e2e tests with a mock Gemini client (no real API calls or costs), covering:

  • Full summarization flow and model fallback
  • Daily token limit enforcement and sliding window expiry
  • Token usage tracking (with UsageMetadata and CountTokens fallback)
  • Content truncation for oversized inputs
  • Error handling (empty responses, client failures, missing API key)

The database service includes tests for:

  • CheckExist RPC — cache hit, cache miss, empty hash validation, uninitialized DB
  • Full integration workflow: create → write (with hash_id) → query → CheckExist verification

The recommendation handler includes tests for:

  • No tasks / database error / LLM error handling
  • Valid JSON parsing with correct ranks, titles, and reasons
  • Markdown code fence stripping (\``json ... ````)
  • Fallback when LLM returns plain text instead of JSON
  • ?top=N parameter validation (default 3, range 1-10, invalid values)
  • Prompt content verification (correct format string interpolation)
  • task_count reflects DB entries, not recommendation count