Skip to content

Commit 5f6213c

Browse files
committed
feat: remove redis and use in-memory queue instead for easier self hosting
1 parent 9d0f0a9 commit 5f6213c

File tree

9 files changed

+171
-409
lines changed

9 files changed

+171
-409
lines changed

.env.example

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,3 @@ DIRECT_URL= ''
1313
# url for background task
1414
NEXT_BG_TASK_URL='http://localhost:3001'
1515

16-
# redist host
17-
REDIS_HOST='localhost'
18-

README.md

Lines changed: 43 additions & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
21
<img width="1596" alt="Screenshot 2024-11-12 at 1 42 58 PM" src="https://github.com/user-attachments/assets/964b5154-ae4e-4216-a060-37e0a5eda965">
32

43
## Important
@@ -13,7 +12,6 @@ I started this open-source project because I found that most existing solutions,
1312

1413
I realized that leveraging AI to read and extract text from credit card PDF statements is a more efficient way to store and categorize expenses. By storing this data in a database, we can streamline the entire process. My goal is to create a self-hosted, local-first solution—much like Firefly or ActualBudget—where users have more control over their data.
1514

16-
1715
## Features
1816

1917
The main feature to upload credit card PDFs or zip files containing credit card PDFs will work as follows:
@@ -45,21 +43,18 @@ The main feature to upload credit card PDFs or zip files containing credit card
4543
1. **Background Job Processing**
4644
PDF statements are sent to the background job server for processing.
4745

48-
2. **Parsing Workflow**
46+
1. **Parsing Workflow**
47+
4948
- **Convert PDFs to Images**: The PDF files are converted into images.
5049
- **OCR (Optical Character Recognition)**: OCR is run on the images to extract text from the statement.
5150
- **OpenAI API**: The extracted text is sent to the OpenAI API, which returns parsed expenses, categorized according to custom rules.
5251

53-
3. **Temporary Storage in Redis**
54-
The parsed expenses are temporarily stored in Redis.
55-
56-
4. **Frontend Options**
52+
1. **Frontend Options**
5753
On the frontend, users can:
5854
- Store the parsed data in the database
5955
- Download it as a CSV file
6056
- Edit the data before taking any further action
6157

62-
6358
## Database and OpenAI key
6459

6560
### Open AI Key
@@ -76,51 +71,52 @@ This [medium](https://medium.com/@lorenzozar/how-to-get-your-own-openai-api-key-
7671

7772
Make sure you have sufficient credit in your OpenAI account for billing purposes. It costs approximately $0.03 USD per credit card statement. Detailed cost calculations will be provided here.
7873

79-
8074
### Postgresql DB Connection
8175

8276
You can setup postgresql locally, but here is the guide on setting up in supabase
8377

8478
1. Create a Supabase Account
85-
- Go to the [Supabase website](https://supabase.com/) and sign up for an account.
86-
- If you already have an account, simply log in.
79+
80+
- Go to the [Supabase website](https://supabase.com/) and sign up for an account.
81+
- If you already have an account, simply log in.
8782

8883
1. Create a New Project
89-
- After logging in, click the **New Project** button on your dashboard.
90-
- Enter your project details:
91-
- **Name**: Choose a unique name for your project.
92-
- **Organization**: Select your organization or create a new one if necessary.
93-
- **Database Password**: Create a secure password for your database. Keep this password safe since you'll need it to connect to your database later.
94-
- **Region**: Choose the closest server location to optimize performance.
9584

96-
Once done, click **Create new project**.
85+
- After logging in, click the **New Project** button on your dashboard.
86+
- Enter your project details:
87+
- **Name**: Choose a unique name for your project.
88+
- **Organization**: Select your organization or create a new one if necessary.
89+
- **Database Password**: Create a secure password for your database. Keep this password safe since you'll need it to connect to your database later.
90+
- **Region**: Choose the closest server location to optimize performance.
9791

98-
3. Wait for the Database to be Set Up
99-
- It may take a few minutes for Supabase to set up your PostgreSQL database. Once ready, you’ll be taken to your project dashboard.
92+
Once done, click **Create new project**.
10093

101-
4. Retrieve Your Database Connection String
102-
- From your project dashboard, navigate to the **Settings** tab.
103-
- Under **Database**, you’ll see details for connecting to your database.
104-
- Look for **Connection String** or **Database URL**. It will look something like this:
94+
1. Wait for the Database to be Set Up
10595

106-
```plaintext
107-
postgres://username:password@host:port/database
108-
```
96+
- It may take a few minutes for Supabase to set up your PostgreSQL database. Once ready, you’ll be taken to your project dashboard.
10997

110-
You will need this connection string to connect your application to the Supabase database.
98+
1. Retrieve Your Database Connection String
11199

112-
5. Save Your Connection String
113-
- Make sure to copy and securely store the connection string. You will need it to set up the database in your app.
100+
- From your project dashboard, navigate to the **Settings** tab.
101+
- Under **Database**, you’ll see details for connecting to your database.
102+
- Look for **Connection String** or **Database URL**. It will look something like this:
114103

104+
```plaintext
105+
postgres://username:password@host:port/database
106+
```
115107

108+
You will need this connection string to connect your application to the Supabase database.
109+
110+
5. Save Your Connection String
111+
- Make sure to copy and securely store the connection string. You will need it to set up the database in your app.
116112

117113
## Docker Setup
118114

119115
### Prerequisite
120116

121117
1. **Docker**
122-
- Install docker
123-
- Run Docker
118+
- Install docker
119+
- Run Docker
124120

125121
### Installation
126122

@@ -135,8 +131,8 @@ You will need this connection string to connect your application to the Supabase
135131
```sh
136132
cd self-hosted-expense-tracker
137133
```
138-
139-
1. Run npm install
134+
135+
1. Run npm install
140136

141137
```sh
142138
npm install
@@ -151,7 +147,7 @@ You will need this connection string to connect your application to the Supabase
151147
1. Run prisma migrate for the first time to migrate database.
152148

153149
```sh
154-
npx prisma migrate deploy --schema apps/web/prisma/schema.prisma
150+
npx prisma migrate deploy --schema apps/web/prisma/schema.prisma
155151
```
156152

157153
1. Run Docker Compose and build
@@ -160,17 +156,14 @@ You will need this connection string to connect your application to the Supabase
160156
docker compose up --build
161157
```
162158

163-
164159
## Development Setup
165160

166-
167161
### Prerequisite
162+
168163
1. NodeJs
169164
1. Ghostscript and graphicsmagick
170-
- Both Ghostscript and Graphicsmagick are require to PDF parsing on the background.
171-
- Follow this [guide](https://github.com/yakovmeister/pdf2image/blob/HEAD/docs/gm-installation.md) to install both of them
172-
1. Redis
173-
- Setup and run redis on port 6379. Guide [here](https://redis.io/docs/latest/operate/oss_and_stack/install/install-redis/).
165+
- Both Ghostscript and Graphicsmagick are require to PDF parsing on the background.
166+
- Follow this [guide](https://github.com/yakovmeister/pdf2image/blob/HEAD/docs/gm-installation.md) to install both of them
174167

175168
### Installation
176169

@@ -185,8 +178,8 @@ You will need this connection string to connect your application to the Supabase
185178
```sh
186179
cd self-hosted-expense-tracker
187180
```
188-
189-
1. Run npm install
181+
182+
1. Run npm install
190183

191184
```sh
192185
npm install
@@ -201,22 +194,20 @@ You will need this connection string to connect your application to the Supabase
201194
1. Run prisma migrate for the first time to migrate database.
202195

203196
```sh
204-
npx prisma migrate deploy --schema apps/web/prisma/schema.prisma
197+
npx prisma migrate deploy --schema apps/web/prisma/schema.prisma
205198
```
206199

207200
1. Run in development mode
208201

209202
```sh
210203
npm run dev
211204
```
212-
213-
214-
## Start Up Guide
215-
216-
1. Go to `/sign-up` to create account with `username` and `password` after everything has set up.
217-
218-
1. The categories will be seeded from `/apps/web/prisma/categoryList.json`.
219-
205+
206+
## Start Up Guide
207+
208+
1. Go to `/sign-up` to create account with `username` and `password` after everything has set up.
209+
210+
1. The categories will be seeded from `/apps/web/prisma/categoryList.json`.
220211

221212
## Roadmap
222213

apps/background-job/package.json

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,12 +9,12 @@
99
"@hono/node-server": "^1.13.1",
1010
"bullmq": "^5.14.0",
1111
"dotenv": "^16.4.5",
12+
"fastq": "^1.17.1",
1213
"file-type": "^19.6.0",
1314
"hono": "^4.6.3",
14-
"ioredis": "^5.4.1",
15+
"node-cache": "^5.1.2",
1516
"openai": "^4.67.0",
1617
"pdf2pic": "^3.1.3",
17-
"redis": "^4.7.0",
1818
"tesseract.js": "^5.1.1",
1919
"yauzl": "^3.1.3"
2020
},

0 commit comments

Comments
 (0)