docs and readme

newgnart · newgnart · commit d6755c5261ee · 2025-11-03T09:46:37.000+07:00
diff --git a/README.md b/README.md
@@ -1,123 +1,75 @@
-# Ethereum Blockchain Data Analytics Platform
+# Stables Analytics
 
-Capstone project for [Foundry AI Academy](https://www.foundry.academy/) Data & AI Engineering program. An ELT pipeline for extracting, loading, and transforming Ethereum blockchain data with focus on stablecoin analytics.
+Production-grade ELT pipeline for on-chain stablecoin analytics. Built for [Foundry AI Academy](https://www.foundry.academy/) Data & AI Engineering program.
 
-Inspired by [Visa on Chain Analytics](https://visaonchainanalytics.com/).
+**Pipeline**: `HyperSync GraphQL → PostgreSQL/Snowflake → dbt → Analytics`
 
 ## Quick Start
 
-### Prerequisites
 ```bash
-# Create Docker network
-docker network create fa-dae2-capstone_kafka_network
-
-# Start PostgreSQL
+# Setup
 docker-compose up -d
-
-# Install dependencies
 uv sync
-
-# Setup environment
 cp .env.example .env
-export $(cat .env | xargs)
-```
 
-### Extract Data
-```bash
-# Extract logs and transactions from Etherscan
-uv run python scripts/el/extract_etherscan.py \
-  -c ethereum \
-  -a 0x02950460e2b9529d0e00284a5fa2d7bdf3fa4d72 \
-  --logs --transactions \
-  --from_block 18.5M --to_block 20M \
-  -v
-```
-
-### Load Data
-```bash
-# Load Parquet to PostgreSQL
-uv run python scripts/el/load.py \
-  -f .data/raw/ethereum_0xaddress_logs_18500000_20000000.parquet \
-  -c postgres \
-  -s raw \
-  -t logs \
-  -w append
-```
+# Extract blockchain data via GraphQL
+# Run the indexer separately: https://github.com/newgnart/envio-stablecoins
+uv run python scripts/el/extract_graphql.py --from_block 18500000 --to_block 20000000 -v
 
-### Transform Data
-```bash
-# Run dbt models
-./scripts/dbt.sh run
+# Load to database
+uv run python scripts/el/load.py -f .data/raw/data_*.parquet -c postgres -s raw -t raw_transfer -w append
 
-# Run specific model
-./scripts/dbt.sh run --select stg_logs_decoded
+# Transform with dbt
+cd dbt_project && dbt run
 ```
 
 ## Architecture
 
-**Extract** → **Load** → **Transform**
+```
+HyperSync GraphQL API → Parquet Files → PostgreSQL/Snowflake → dbt → Analytics Tables
+```
 
-1. **Extract** (`scripts/el/extract_etherscan.py`): Pulls blockchain data from Etherscan API to `.data/raw/*.parquet`
-2. **Load** (`scripts/el/load.py`): Loads Parquet files into PostgreSQL/Snowflake `raw` schema
-3. **Transform** (`dbt_project/`): dbt models transform raw data into analytics-ready tables
+**Key Components:**
+
+- **Extract**: High-performance GraphQL API (HyperSync) for blockchain data with block-range filtering
+- **Load**: Pluggable loaders for PostgreSQL (dev) and Snowflake (prod) with `append`/`replace`/`merge` modes
+- **Transform**: dbt three-tier modeling (Staging → Intermediate → Marts) with SCD Type 2 support
+- **Migrate**: PostgreSQL to Snowflake data transfer via `pg2sf_raw_transfer.py`
 
 ### Project Structure
+
 ```
-├── scripts/el/              # Extract & Load scripts
-├── src/onchaindata/         # Reusable Python package
-│   ├── data_extraction/     # Etherscan/GraphQL clients
-│   ├── data_pipeline/       # Loader classes
-│   └── utils/              # Database clients
-├── dbt_project/            # dbt transformation layer
-│   ├── models/01_staging/  # Raw data cleanup (views)
-│   ├── models/intermediate/# Business logic (ephemeral)
-│   └── models/marts/       # Analytics tables (tables)
-└── .data/raw/             # Extracted Parquet files
+scripts/el/           # Extract & Load scripts
+src/onchaindata/      # Python package (extraction, loading, database clients)
+dbt_project/          # dbt models, snapshots, macros
+docs/                 # MkDocs documentation
 ```
 
-## Key Features
+## Features
 
-- **Multi-chain support**: Ethereum, Polygon, BSC via chainid mapping
-- **Automatic retry**: Failed extractions retry with smaller chunks (10x reduction)
-- **Flexible loading**: PostgreSQL and Snowflake support
-- **Block number shortcuts**: Use `18.5M` instead of `18500000`
-- **dbt transformations**: Staging → Intermediate → Marts layers
+- **High-Performance Extraction**: HyperSync GraphQL API for fast blockchain data retrieval
+- **Flexible Loading**: PostgreSQL & Snowflake support with multiple write modes
+- **Multi-Chain**: Ethereum, Polygon, BSC via configurable endpoints
+- **SCD Type 2**: Historical tracking for stablecoin metadata via dbt snapshots
+- **Cross-Database Migration**: Seamless PostgreSQL → Snowflake transfers
 
-## Environment Variables
+## Tech Stack
 
-Required (see `.env.example`):
-- `POSTGRES_*`: Database connection
-- `ETHERSCAN_API_KEY`: API access
-- `DB_SCHEMA`: Default schema
+Python 3.11+ • SQL • Polars • dlt • PostgreSQL • Snowflake • dbt Core • Docker • uv
 
-Optional (for Snowflake):
-- `SNOWFLAKE_*`: Snowflake connection details
+## Documentation
 
-## Common Commands
+- **Full Docs**: [https://newgnart.github.io/stables-analytics/](https://newgnart.github.io/stables-analytics/)
+- **Dev Guide**: [CLAUDE.md](CLAUDE.md)
 
-```bash
-# SQL operations
-./scripts/sql_pg.sh ./scripts/sql/init.sql
-
-# dbt operations
-./scripts/dbt.sh test                    # Run tests
-./scripts/dbt.sh docs generate           # Generate docs
-./scripts/dbt.sh run --select staging.*  # Run staging models
-
-# Extract with time range
-uv run python scripts/el/extract_etherscan.py \
-  -a 0x02950460e2b9529d0e00284a5fa2d7bdf3fa4d72 \
-  --logs --transactions \
-  --last_n_days 7
-```
+## Environment Setup
 
-## Database Schema
+Create `.env` file with database credentials:
+- `POSTGRES_*`: PostgreSQL connection details
+- `SNOWFLAKE_*`: (Optional) Snowflake connection details
 
-- `raw.logs`: Raw event logs with JSONB topics
-- `raw.transactions`: Transaction data
-- `staging.stg_logs_decoded`: Decoded logs with parsed topics (topic0-topic3)
-- Marts: Analytics tables created by dbt
+See `.env.example` for full configuration.
 
-## Documentation
+---
 
-For detailed documentation, see [CLAUDE.md](CLAUDE.md) or the [docs/](docs/) directory.
+**License**: MIT • Educational capstone project
diff --git a/docs/index.md b/docs/index.md
@@ -1,8 +1,52 @@
-# Home
-The capstone project for [Foundry AI Academy](https://www.foundry.academy/) Data&AI Engineering program.
+# Stables Analytics Platform
 
-Inspired by [Visa on Chain Analytics](https://visaonchainanalytics.com/) which showcases how fiat-backed stablecoins move via public blockchains globally
-- Key Metrics: Stablecoin Supply, Transactions Volume, Addresses and Lending
-- Stablecoins: USDC, USDT, PYUSD, FDUSD, USDP and USDG on several blockchains
+A production-grade data analytics platform for on-chain stablecoin transactions, built as a capstone project for [Foundry AI Academy](https://www.foundry.academy/) Data & AI Engineering program.
 
-This project is built as Analytics Platform for decentralized stablecoins usages on Ethereum.
+## Technical Overview
+
+### Data Engineering Architecture
+
+The platform implements a modern **ELT (Extract, Load, Transform)** pipeline optimized for blockchain data:
+
+```
+HyperSync GraphQL API → PostgreSQL/Snowflake → dbt Transformations → Analytics Tables
+```
+
+**Key Engineering Components:**
+
+#### 1. **Extraction Layer** (`Python + GraphQL`)
+- High-performance blockchain data extraction via HyperSync GraphQL API
+- Block-range filtering with dynamic query generation (supports custom `from_block`/`to_block` parameters)
+- Batch mode for large historical extracts with automatic Parquet serialization
+- Streaming mode for real-time data ingestion directly to databases
+- Multi-chain support (Ethereum, Polygon, BSC) through configurable endpoints
+
+#### 2. **Loading Layer** (`dlt + SQL`)
+- Pluggable database clients supporting PostgreSQL and Snowflake
+- Multiple write modes: `append`, `replace`, `merge` (upsert) with composite key support, with dlt pipeline
+- Connection pooling and optimized batch loading for high-throughput ingestion
+
+#### 3. **Transformation Layer** (`dbt`)
+- Three-tier modeling: Staging (views) → Intermediate (ephemeral) → Marts (tables)
+- Slowly Changing Dimension (SCD Type 2) implementation via dbt snapshots for stablecoin metadata
+- Custom Ethereum macros for address extraction and uint256 conversion
+- Cross-database compatibility (PostgreSQL for dev, Snowflake for production)
+
+#### 4. **Data Migration** (`Python`)
+- Block-range based PostgreSQL to Snowflake data transfer for cloud warehousing
+- Polars-powered efficient data transformation and loading
+
+### Tech Stack
+
+- **Languages**: Python 3.11+, SQL
+- **Data Processing**: Polars, Pandas, PyArrow, dlt
+- **Databases**: PostgreSQL, Snowflake
+- **Transformation**: dbt Core (Postgres/Snowflake adapters)
+- **Infrastructure**: Docker, uv (dependency management)
+- **Documentation**: MkDocs Material
+
+## Getting Started
+
+Detailed setup instructions and API reference available in the navigation menu.
+
+For development workflows, see [CLAUDE.md](https://github.com/newgnart/stables-analytics/blob/main/CLAUDE.md).
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -1,60 +1,57 @@
 site_name: Stables Analytics
+# Repository
+repo_url: https://github.com/newgnart/stables-analytics
+repo_name: stables-analytics
+# Theme configuration
 theme:
   name: material
-  features:
-    - announce.dismiss
-    - content.action.edit
-    - content.action.view
-    - content.code.annotate
-    - content.code.copy
-    # - content.code.select
-    # - content.footnote.tooltips
-    # - content.tabs.link
-    - content.tooltips
-    # - header.autohide
-    # - navigation.expand
-    - navigation.footer
-    - navigation.indexes
-    # - navigation.instant
-    # - navigation.instant.prefetch
-    # - navigation.instant.progress
-    # - navigation.prune
-    - navigation.sections
-    - navigation.tabs
-    # - navigation.tabs.sticky
-    - navigation.top
-    - navigation.tracking
-    - search.highlight
-    - search.share
-    - search.suggest
-    - toc.follow
   palette:
-    - media: "(prefers-color-scheme)"
-      toggle:
-        icon: material/link
-        name: Switch to light mode
-    - media: "(prefers-color-scheme: light)"
-      scheme: default
+    # Light mode
+    - scheme: default
       primary: indigo
       accent: indigo
       toggle:
-        icon: material/toggle-switch
+        icon: material/brightness-7
         name: Switch to dark mode
-    - media: "(prefers-color-scheme: dark)"
-      scheme: slate
-      primary: black
+    # Dark mode
+    - scheme: slate
+      primary: indigo
       accent: indigo
       toggle:
-        icon: material/toggle-switch-off
-        name: Switch to system preference
-  font:
-    text: Roboto
-    code: Roboto Mono
-  # favicon: assets/favicon.png
-  # icon:
-  # logo: logo
-  # - toc.integrate
+        icon: material/brightness-4
+        name: Switch to light mode
+  features:
+    - navigation.tabs
+    - navigation.sections
+    - navigation.top
+    - navigation.tracking
+    - search.suggest
+    - search.highlight
+    - content.code.copy
+    - content.code.annotate
 
+# Extensions
+markdown_extensions:
+  - pymdownx.highlight:
+      anchor_linenums: true
+      line_spans: __span
+      pygments_lang_class: true
+  - pymdownx.inlinehilite
+  - pymdownx.snippets
+  - pymdownx.superfences:
+      custom_fences:
+        - name: mermaid
+          class: mermaid
+          format: !!python/name:pymdownx.superfences.fence_code_format
+  - pymdownx.tabbed:
+      alternate_style: true
+  - pymdownx.details
+  - admonition
+  - toc:
+      permalink: true
+  - tables
+  - attr_list
+  - md_in_html
 nav:
   - Home: index.md
   - Getting Started: 01_getting_started.md
