update docs (#99)

Author: elenamik

.env.example

@@ -4,7 +4,7 @@ INDEXER_SUPABASE_ADMIN_KEY='JWT'
 INDEXER_ETH_RPC_WS_URL='wss://url.com'
 INDEXER_POSTGRES_CONNECTION_STRING='postgresql://postgres:[YOUR-PASSWORD]@127.0.0.1:5432/postgres'
 
-# For API
+# For Next.js API
 API_SUPABASE_URL='http://127.0.0.1:54321'
 API_SUPABASE_ANON_KEY='JWT'
 API_AUTH_KEY='random_string'

README.md

@@ -1,26 +1,99 @@
-# Indexer + API app
-The project has 3 parts
-- Postgres DB - schema defined in `supabase` hosted by supabase, uses some of their tools
-- Indexer "ETL" go code in `go-indexer` - long-running server that
-    - connects to an ethereum node
-    - reads events for specific contracts (configured in `config`)
-    - write events to DB
-    - transforms events into data required for the app (accounts, positions, etc)
-- Next JS API app in `app` - easy way to host the API. reads from the same schema. Deployed on vercel.
-
-## Database
-### Prerequisites
-- npm
-- docker
-- supabase installed globally (`npm install supabase --save-dev`)
-
-### Start locally
-start docker
+# ERC4626 Ethereum Vault Indexer and API Gateway
+This project demonstrates the design and implementation of a production-grade Ethereum indexer and API gateway, written entirely in Go.
+It ingests on-chain events from an ERC-4626 vault, processes and normalizes them into a Supabase PostgreSQL database, and exposes a public REST API with automatically generated documentation.
+The system is built around a concurrent, multi-threaded architecture designed for data integrity, fault tolerance, and maintainable scaling — with live and historical event ingestion, chain reorganization handling, and finalized block tracking.
+Deployed on a Linux server and managed via systemd, the project demonstrates a full end-to-end data pipeline: from blockchain ingestion to queryable structured data.
+
+## Features
+- ***Ethereum Indexer in Go***: Listens to Deposit, WithdrawRequested, and Transfer events from an ERC-4626 vault contract
+- ***Supabase Integration***: Stores normalized event data in a hosted Postgres database
+- ***API Gateway***: Exposes REST endpoints (via Next.js) for querying indexed data
+- ***Auto-Generated Docs***: OpenAPI documentation generated automatically from Next.js endpoints, available at `<API_URL>/docs`
+- ***Resilient Architecture***: Includes queue-based event processing and reorg verification 
+- ***Production-Ready Deployment***: Can run as a Linux systemd service for reliability and observability
+
+## Architecture
+![Architecture](/docs/architecture.png)
+The Vault Indexer is designed as a concurrent Go service that ingests, processes, and serves Ethereum vault events in real time.
+
+It operates through three coordinated threads — each responsible for a specific stage of the indexing pipeline — and exposes the processed data through an API gateway backed by Supabase Postgres.
+
+### Main Indexer Thread
+- Connects to an Ethereum node via JSON-RPC and subscribes to the vault’s on-chain events.
+- Handles both historical backfill and live streaming of events such as Deposit, WithdrawRequested, and Transfer.
+- Writes all captured events into an in-memory event queue, ensuring they’re processed in chronological order.
+- Acts as the entry point for all blockchain data entering the system.
+
+### Event Processing Queue
+- Serves as a buffer between event ingestion and downstream processing.
+- Maintains ordered delivery of events to prevent race conditions and ensure deterministic indexing.
+- Enables the system to handle bursts of incoming events without blocking network I/O.
+
+### Data Processor Thread
+- Continuously fetches newly queued events and transforms them into structured index records suitable for querying.
+- Normalizes Ethereum event data (e.g., addresses, amounts, block metadata) into relational database entries.
+- Persists these records into the Postgres database, providing a clean and queryable representation of vault state over time.
+
+### Finality Processor Thread
+- Monitors blockchain finality and safety levels to maintain data integrity.
+- Detects potential chain reorganizations (reorgs) by verifying block hashes and canonicality.
+- Triggers cleanup and rollback operations if non-finalized data becomes invalid.
+- Updates internal tables to reflect finalized blocks and ensures all indexed data corresponds to the canonical Ethereum chain.
+
+### Postgres Database (Supabase)
+- Acts as the central data store for both raw events and processed indices.
+- Enables efficient queries over user positions, vault balances, and event histories.
+- Managed through Supabase, which provides hosted PostgreSQL, authentication, and built-in REST access.
+
+### Next.js API Gateway
+- Provides a RESTful API layer for external clients, dashboards, or analytics tools.
+- Offers auto-generated documentation (Swagger / OpenAPI) for ease of exploration.
+- Fetches data directly from the Postgres database, exposing endpoints for querying vault events, user histories, and position summaries.
+- Enables fast, queryable access to indexed data by contract address or user wallet.
+
+
+## Getting Started
+### 1. Prerequisites
+- indexer
+    - go
+    - ethereum node (HTTP endpoint)
+- api gateway
+    - npm
+    - node.js
+- local postgres development
+    - supabase installed globally (`npm install supabase --save-dev`)
+    - docker
+
+### 2. Supply environment
+Copy `.env.example` and create `.env.local`. You can also create an `.env.prod` and run the process with an environment arg. The default environment is dev.
+
+### 3. Configure vault addresses to index at `go-indexer/config/config.yaml`.
+
+### 4. Start Database
+- start docker
+- start database
 ```bash
-npx supabase start
-npx supabase status
+npx supabase start # will deploy existing schema
+npx supabase status # view the available endpoints
 ```
 
+### 5. Start indexer
+```bash
+go mod tidy # installs Go dependencies
+./start-indexer.sh  # Start server in development (default)
+./start-indexer.sh prod # example with .env.prod file
+```
+
+### 6. Start API gateway / docs
+```bash
+npm install
+npm run dev
+```
+Swagger docs will be available at `/docs`
+
+## Working with Supabase
+Here are some useful commands for working with the database.
+
 ### Deploy schema change 
 ```bash
 npx suapbase migration new <name>
@@ -55,94 +128,33 @@ docker volume ls
 docker volume rm <volume name>
 ```
 
-## Indexer
+## Deployment
+### Indexer
+The indexer is meant to run as a constant-uptime script on a remote server. It is safe to stop and restart, and will backfill the data on each restart.
 
-### Prereqs
-- go
-- docker
+This process was previously deployed on Linux servers, run with systemd commands which were configured via Nix. 
 
-### Local development
-1. install dependencies
-```bash
-go mod tidy
-```
-
-2. set config in `go-indexer/config/config.yaml`
-- add contracts info + abis
-- check supabase local studio (http://127.0.0.1:54323) for anon key and connection string
+Persistent errors will cause the process to stop running. If you plan to deploy this yourself, make sure to add a configuration which will restart the process with a cooldown if it stops.
 
-3. run server
-```bash
-# For development (default)
-./start-indexer.sh
-
-# For production
-./start-indexer.sh prod
-```
-
-###
-Run with nix
+## Running with nix
 ```bash
 nix run .#indexer
 ```
 
 ## Testing
 ```bash
 go test ./...  
-
 go test ./go-indexer/indexer -v
-
 ```
 
+
 ## Health checks
-Health checks should be deployed for every process. Currently there are only 2:
-1. historicla loading + event ingestion (main routine) 
+The main thread spins up a lightweight HTTP server available for health checks.
 ```bash
 curl http://localhost:8080/health
 ```
-2. event processing (transform.go)
-```bash
-curl http://localhost:8081/health
-```
-
-## Deploying
-1. stop server + update
-2. run database migration [doc](https://supabase.com/docs/guides/deployment/database-migrations#deploy-your-project)
-3. if you don't want to refetch all historical events, update `go-indexer/config/config.yaml` to fetch events after a specific block
-4. restart server
-```
-
-```
-
-# API server
-Simple nextjs app to provide an API for the indexer database
-
-## Prereqs
-- node.js
-- npm
-
-## Start API server
-1. install dependencies
-```bash
-npm install
-```
-
-2. copy `.env.example`
-- SUPABASE_URL: http url for postgres db
-- SUPABASE_ANON_KEY: read-only key
-- API_KEY: random string to give access to API
-
-3. start server
-```bash
-npm run dev
-```
-Visit the api at `localhost:3000/v1`
-
-## API docs and schema
-Docs served at `/docs`. Generated automatically
-
 
-# Data Testing
+## Data Testing
 Manaul check which can be run to load all historical data locally, and compare that production matches the historical load.
 
 1. Start with fresh local DB