plan.md

Email Wizard

A wizard who structures all your emails for you.

Description

Codes in this repo should be able to parse raw email contents into structured contents, including email title, sender, content summary, and important dates. Optionally, it may be able to pull emails from modern mail boxes (such as outlook, gmail, 126/163.com, etc.), store parsed results in a calendar format and present them to users.

Design

Workflow

Architecture

Architecture for asynchronous services (with kafka streams marked with /)

Tasks

Stage 1 (basic functionality)

A1. retrieve emails

• use IMAP/POP3 protocol to retrieve emails from various mailboxes
  • support 126/163.com mailboxes (with IMAP)
  • support outlook mailboxes

A2. clean email into plain texts

• retrieve raw email contents from retrieved emails
• transform raw email contents into plain text
  • discard image (for now)
  • extract and keep plain text content
  • extract email subject, sender, date and time
• deploy with microservice (together with A1)
  • benchmark microservice performance

B1. parse plain text for summary and dates

• construct prompts for ChatGPT/GPT
  • build a small dataset of emails and the structured summary
  • explore different models and different prompts
    • set up evaluation metrics
    • concurrent ChatGPT requests
• use OpenAI API or other models to get structured summary
  • obtain model output from prompts
  • parse model output into structured summary
• migrate to langchain?
• deploy with microservice
  • benchmark microservice performance

C1. store summary to database

• design API protocol
  • CRUD operations interface
• Implement APIs
  • database design
    • choose database engine
    • design database schema
  • CRUD for email db
    • add_row
    • update_item
    • delete_row
    • query
  • CRUD for user db
    • add_row
    • update_item
    • delete_row
    • query
  • CRUD for event db
    • add_row
    • update_item
    • delete_row
    • query
  • testing
• deploy as a microservice
  • testing
  • benchmark microservice performance

C2. API requests

• backend API interface design
  • refresh events, query events, and more
• handle requests by interacting with microservices
  • choose microservice framework
  • implement logic with prepared microservices
    • retrieve emails from user mailboxes
    • extract new emails
    • parse to events
    • store results to database
    • query database for results
    • user registration
• testing
  • script testing
  • API tools testing

C3. calendar view UI

• frontend UI design and development
  • UI wireframe
  • build UI with React
    • implement main calendar page (use fake user_id and secret for now)
    • implement add email account (use fake autheticate for now)
    • implement login page (get user_id and secret)
    • implement register page (get user_id and secret)
    • implement logout mechanism
    • implement intro page
    • implement account authorization grants
    • integrate with backend API
      • integrate with updated APIs
• deploy to production server

Stage 2 (minimum useful requirements)

A1. retrieve emails

• support gmail mailboxes

C2. API requests

• refine API interface
  • migrate to RESTful APIs

C3. calendar view UI

• refine API interface
• add remove mailbox functionality
  • revoke authorized mailbox access
• add event search (pure frontend for now)
• make the UI more beautiful
  • iterate with @mui

Stage 3 (performance optimization)

Following the best practices for building scalable backend system

A1. retrieve emails

• refactor the microservice to listen from kafka logs
• async / multi-threading retrieving emails

B1. parse plain text for summary and dates

• optimize QPS
  • refactor the microservice to listen from kafka logs
  • async / multi-threading calling LLM API

C1. store summary to database

• optimize query

C2. API requests

• implement caching
• implement auto-scaling
• monitor systems
• use CDN
• improve responsiveness via message queues
  • set up kafka servers
  • refactor retrieve-parse-display flow to be asynchronous
    • listen to kafka logs
    • return events via websocket

C3. calendar view UI

• faster source retrival

Stage 4 (more features and supports)

A1. retrieve emails

• update outlook oauth to auth code flow
• support exchange protocol

A2. clean email into plain texts

• handle html content

B1. parse plain text for summary and dates

• better precision and recall
• timezone
• refine API interface

C1. store summary to database

• refine API interface

C2. API requests

• API testing
• microservices testing
• migrate to full docker orchestration
  • migrate to docker compose
  • deploy with k8s (depending on machine resources)

C3. calendar view UI

• update outlook auth to auth code flow
• add forget password functionaility
• make the UI more beautiful

C4. update calendar

• add/remove events in user calendar based on user actions
  • find libraries to access user calendar (possibly outlook calendar)
  • add links to user calendar in UI

Environments

Benchmarking

Use ghz with Docker for benchmarking gRPC microservice performance:

DOCKER_BUILDKIT=1 sudo docker build --output=/usr/local/bin --target=ghz-binary-built https://github.com/bojand/ghz.git