feat: remove redis and use in-memory queue instead for easier self ho…

…sting

.env.example

@@ -13,6 +13,3 @@ DIRECT_URL= ''
 # url for background task 
 NEXT_BG_TASK_URL='http://localhost:3001'
 
-# redist host
-REDIS_HOST='localhost'
-

README.md

@@ -1,4 +1,3 @@
-
 <img width="1596" alt="Screenshot 2024-11-12 at 1 42 58 PM" src="https://github.com/user-attachments/assets/964b5154-ae4e-4216-a060-37e0a5eda965">
 
 ## Important
@@ -13,7 +12,6 @@ I started this open-source project because I found that most existing solutions,
 
 I realized that leveraging AI to read and extract text from credit card PDF statements is a more efficient way to store and categorize expenses. By storing this data in a database, we can streamline the entire process. My goal is to create a self-hosted, local-first solution—much like Firefly or ActualBudget—where users have more control over their data.
 
-
 ## Features
 
 The main feature to upload credit card PDFs or zip files containing credit card PDFs will work as follows:
@@ -45,21 +43,18 @@ The main feature to upload credit card PDFs or zip files containing credit card
 1. **Background Job Processing**  
    PDF statements are sent to the background job server for processing.
 
-2. **Parsing Workflow**  
+1. **Parsing Workflow**
+
    - **Convert PDFs to Images**: The PDF files are converted into images.
    - **OCR (Optical Character Recognition)**: OCR is run on the images to extract text from the statement.
    - **OpenAI API**: The extracted text is sent to the OpenAI API, which returns parsed expenses, categorized according to custom rules.
 
-3. **Temporary Storage in Redis**  
-   The parsed expenses are temporarily stored in Redis.
-
-4. **Frontend Options**  
+1. **Frontend Options**  
    On the frontend, users can:
    - Store the parsed data in the database
    - Download it as a CSV file
    - Edit the data before taking any further action
 
-
 ## Database and OpenAI key
 
 ### Open AI Key
@@ -76,51 +71,52 @@ This [medium](https://medium.com/@lorenzozar/how-to-get-your-own-openai-api-key-
 
 Make sure you have sufficient credit in your OpenAI account for billing purposes. It costs approximately $0.03 USD per credit card statement. Detailed cost calculations will be provided here.
 
-
 ### Postgresql DB Connection
 
 You can setup postgresql locally, but here is the guide on setting up in supabase
 
 1. Create a Supabase Account
-    - Go to the [Supabase website](https://supabase.com/) and sign up for an account.
-    - If you already have an account, simply log in.
+
+   - Go to the [Supabase website](https://supabase.com/) and sign up for an account.
+   - If you already have an account, simply log in.
 
 1. Create a New Project
-    - After logging in, click the **New Project** button on your dashboard.
-    - Enter your project details:
-        - **Name**: Choose a unique name for your project.
-        - **Organization**: Select your organization or create a new one if necessary.
-        - **Database Password**: Create a secure password for your database. Keep this password safe since you'll need it to connect to your database later.
-        - **Region**: Choose the closest server location to optimize performance.
 
-    Once done, click **Create new project**.
+   - After logging in, click the **New Project** button on your dashboard.
+   - Enter your project details:
+     - **Name**: Choose a unique name for your project.
+     - **Organization**: Select your organization or create a new one if necessary.
+     - **Database Password**: Create a secure password for your database. Keep this password safe since you'll need it to connect to your database later.
+     - **Region**: Choose the closest server location to optimize performance.
 
-3. Wait for the Database to be Set Up
-    - It may take a few minutes for Supabase to set up your PostgreSQL database. Once ready, you’ll be taken to your project dashboard.
+   Once done, click **Create new project**.
 
-4. Retrieve Your Database Connection String
-    - From your project dashboard, navigate to the **Settings** tab.
-    - Under **Database**, you’ll see details for connecting to your database.
-    - Look for **Connection String** or **Database URL**. It will look something like this:
+1. Wait for the Database to be Set Up
 
-    ```plaintext
-    postgres://username:password@host:port/database
-    ```
+   - It may take a few minutes for Supabase to set up your PostgreSQL database. Once ready, you’ll be taken to your project dashboard.
 
-You will need this connection string to connect your application to the Supabase database.
+1. Retrieve Your Database Connection String
 
-5. Save Your Connection String
-    - Make sure to copy and securely store the connection string. You will need it to set up the database in your app.
+   - From your project dashboard, navigate to the **Settings** tab.
+   - Under **Database**, you’ll see details for connecting to your database.
+   - Look for **Connection String** or **Database URL**. It will look something like this:
 
+   ```plaintext
+   postgres://username:password@host:port/database
+   ```
 
+You will need this connection string to connect your application to the Supabase database.
+
+5. Save Your Connection String
+   - Make sure to copy and securely store the connection string. You will need it to set up the database in your app.
 
 ## Docker Setup
 
 ### Prerequisite
 
 1. **Docker**
-    - Install docker
-    - Run Docker
+   - Install docker
+   - Run Docker
 
 ### Installation
 
@@ -135,8 +131,8 @@ You will need this connection string to connect your application to the Supabase
    ```sh
    cd self-hosted-expense-tracker
    ```
-   
-1. Run npm install 
+
+1. Run npm install
 
    ```sh
    npm install
@@ -151,7 +147,7 @@ You will need this connection string to connect your application to the Supabase
 1. Run prisma migrate for the first time to migrate database.
 
    ```sh
-    npx prisma migrate deploy --schema apps/web/prisma/schema.prisma 
+    npx prisma migrate deploy --schema apps/web/prisma/schema.prisma
    ```
 
 1. Run Docker Compose and build
@@ -160,17 +156,14 @@ You will need this connection string to connect your application to the Supabase
    docker compose up --build
    ```
 
-
 ## Development Setup
 
-
 ### Prerequisite
+
 1. NodeJs
 1. Ghostscript and graphicsmagick
-    - Both Ghostscript and Graphicsmagick are require to PDF parsing on the background.
-    - Follow this [guide](https://github.com/yakovmeister/pdf2image/blob/HEAD/docs/gm-installation.md) to install both of them
-1. Redis
-    - Setup and run redis on port 6379. Guide [here](https://redis.io/docs/latest/operate/oss_and_stack/install/install-redis/).
+   - Both Ghostscript and Graphicsmagick are require to PDF parsing on the background.
+   - Follow this [guide](https://github.com/yakovmeister/pdf2image/blob/HEAD/docs/gm-installation.md) to install both of them
 
 ### Installation
 
@@ -185,8 +178,8 @@ You will need this connection string to connect your application to the Supabase
    ```sh
    cd self-hosted-expense-tracker
    ```
-   
-1. Run npm install 
+
+1. Run npm install
 
    ```sh
    npm install
@@ -201,22 +194,20 @@ You will need this connection string to connect your application to the Supabase
 1. Run prisma migrate for the first time to migrate database.
 
    ```sh
-    npx prisma migrate deploy --schema apps/web/prisma/schema.prisma 
+    npx prisma migrate deploy --schema apps/web/prisma/schema.prisma
    ```
 
 1. Run in development mode
 
    ```sh
    npm run dev
    ```
-
-
- ## Start Up Guide
-
- 1. Go to `/sign-up` to create account with `username` and `password` after everything has set up.
-
- 1. The categories will be seeded from `/apps/web/prisma/categoryList.json`.
-
+
+## Start Up Guide
+
+1.  Go to `/sign-up` to create account with `username` and `password` after everything has set up.
+
+1.  The categories will be seeded from `/apps/web/prisma/categoryList.json`.
 
 ## Roadmap
 

apps/background-job/package.json

@@ -9,12 +9,12 @@
     "@hono/node-server": "^1.13.1",
     "bullmq": "^5.14.0",
     "dotenv": "^16.4.5",
+    "fastq": "^1.17.1",
     "file-type": "^19.6.0",
     "hono": "^4.6.3",
-    "ioredis": "^5.4.1",
+    "node-cache": "^5.1.2",
     "openai": "^4.67.0",
     "pdf2pic": "^3.1.3",
-    "redis": "^4.7.0",
     "tesseract.js": "^5.1.1",
     "yauzl": "^3.1.3"
   },