Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
113 lines (93 loc) · 11.1 KB

ARCHITECTURE.md

File metadata and controls

113 lines (93 loc) · 11.1 KB
Raw

NanoAi Studio by Geekatplay Studio: System Architecture Specification

Hands-on setup, upload, and troubleshooting: see GETTING_STARTED.md and the root README.md.

1. High-Level Architecture Overview

NanoAi Studio (by Vladimir Chopine / Geekatplay Studio) is an end-to-end TinyML generative digital camera architecture deployed on the ESP32-S3 MCU with OV2640 camera module, MicroSD card storage, and onboard WS2812 RGB LED indicator.

+-------------------------------------------------------------------------------+
|                 Desktop Training Pipeline (Python / TensorFlow 2.x)            |
|                                                                               |
|  Synthetic Paired Data --> Depthwise Sep CNN --> INT8 --> models/model_nn_N.h
+-------------------------------------------------------------------------------+
                                                                       |
                                                                       v
+-------------------------------------------------------------------------------+
|                      Edge Firmware Pipeline (C++ / ESP32-S3)                  |
|                                                                               |
|  [OV2640 Camera] --(QVGA 320x240 RGB565)--> [PSRAM Framebuffer]                |
|                                                       |                       |
|                                                       v                       |
|                                            [Center Crop 240x240]              |
|                                                       |                       |
|                                                       v                       |
|                                            [Downsample to NxN]                |
|                                            N = NANOAI_NN_SIZE (32..128)       |
|                                                       |                       |
|                                                       v                       |
|                                            [Normalize to int8_t]              |
|                                                       |                       |
|                                                       v                       |
|                                            [TFLite Micro Engine]              |
|                                                       |                       |
|                                                       v                       |
|                                            [De-quantize to RGB565]            |
|                                                       |                       |
|                                                       v                       |
|                                            [Upscale → LCD / styled JPEG]      |
|                                                       |                       |
|                                     +-----------------+-----------------+     |
|                                     |                                   |     |
|                                     v                                   v     |
|                             [Status RGB LED]                   [MicroSD Card] |
+-------------------------------------------------------------------------------+

This diagram shows the neural/style branch. In parallel, the shutter also saves the original at the sensor's native 1600×1200 resolution as its own JPEG — see "Photo pipeline" in §3 and DEC-050.

2. Technical Boundary Definitions

Environment 1: Model Training (training/)

  • Runtime: Python 3.10+
  • Frameworks: TensorFlow 2.x, Keras, TFLite Converter, NumPy, PyTest
  • Input Specs: NxNx3 RGB float tensor [0.0, 1.0] where N ∈ {32,48,64,96,128} (DEC-022)
  • Output Specs: NxNx3 RGB float tensor [0.0, 1.0]
  • Bridge: python -m model.train --size N → firmware/include/models/model_nn_N.h; firmware selects via -DNANOAI_NN_SIZE=N and model_data.h
  • Quantization Specs: Full Integer INT8 (input_type=int8, output_type=int8); calibration uses a disjoint synthetic sample stream (seed 1337) from the training set (seed 42)

Environment 2: Edge Firmware (firmware/)

  • Runtime: Arduino-ESP32 (PlatformIO), ESP32-S3 N16R8 @ 240 MHz dual-core, 512 KB SRAM + 8 MB octal PSRAM, 16 MB flash
  • Peripherals: OV2640/OV3660 DVP camera, ST7789 1.14" LCD (240×135 landscape), SDMMC 1-bit MicroSD, WS2812 status LED, NS4168 I2S speaker, 5-way ADC joystick (GPIO19)
  • Inference: TFLite Micro, int8 tensors. All ten models (5 style + 5 enhance sizes, 132,800 B total) are embedded in flash; the active tier is chosen at runtime (QUALITY n or the device Config menu) with no reflash. Arena is computed as 2·N²·32 + 96 KB — 160 KB at N=32 up to 1120 KB at N=128 — allocated internal-SRAM-first, spilling to PSRAM for the larger tiers (DEC-037). Only styles 0 (Pixel Art) and 8 (AI Enhance) run the network; styles 1–7 and 10–15 are on-device generative CPU filters compiled directly into inference_pipeline.cpp — no additional flash model per style
  • Style selection: a style_mask bitfield (persisted in NVS, STYLESEL/STYLES over serial) narrows the 16-style table to whatever the dashboard's checklist selected; No AI (9) can never be excluded (DEC-053)
  • Orientation: persisted rotation 0–3; 180° handled in-sensor, 90°/270° as a buffer rotate at capture and in the viewfinder. The hi-res sensor-JPEG original cannot be buffer-rotated without a decode/re-encode, so 90°/270° captures instead carry a minimal EXIF Orientation APP1 segment; the on-device gallery decode and any EXIF-aware viewer (including the web dashboard's <img>) rotate to match (DEC-052)
  • Storage: dual JPEG per capture — /pictureN_normal.jpg + /pictureN_styled.jpg (320×320 or 480×480 hybrid luma/chroma compose, crossfading to a pure-style export above 70% strength for the geometric looks — DEC-053)

Environment 3: Web Dashboard (ui/)

  • Runtime: static HTML/JS/CSS, no build step; Chrome/Edge WebSerial at 115200 8N1
  • Contract: command/response line protocol — see SERIAL_PROTOCOL.md
  • Photo retrieval: LIST enumerates SD files; DUMP /path streams base64 JPEG (≤512 KB) for in-browser preview and download of both original and styled captures
  • Local training bridge (optional, training/dashboard_server.py): a ThreadingHTTPServer that serves ui/ and exposes POST /api/train + GET /api/train/status, running the real python -m model.train pipeline as a subprocess (one job at a time, argv-list only, bound to 127.0.0.1). The dashboard's Train Model button and Model Test Lab manifest read use this; without it, or when the page is opened as file:// (fetch is blocked from any file-origin page), both features degrade to CLI instructions rather than failing silently (DEC-055/056)
  • Model Test Lab: runs the exact exported INT8 .tflite in-browser via @tensorflow/tfjs-tflite (pinned to 0.0.1-alpha.9 — alpha.10's published dist is missing its WASM binaries). Quantizes/dequantizes with the model's own scale and zero-point (matching the firmware's normalize_rgb888_to_int8/dequantize_int8_to_rgb565_nn), and reads the input resolution from the loaded model's own tensor shape rather than trusting a manifest that might be stale (DEC-054)

3. What Nano GenAI is (and is not)

We use We do not use
TensorFlow / Keras depthwise CNN (desktop train) CLIP / text encoders
Full INT8 TFLite → models/model_nn_N.h VAE / latent autoencoders
TFLite Micro on ESP32-S3 (tiered NxN, ~13KB+ at 32) UNet / DiT / Stable Diffusion / FLUX
Synthetic NxN gradient pairs (procedural) ImageNet / LAION / scraped photo corpora
OV2640 RGB565 (styles 0–3) + JPEG (style 4) Cloud GPU inference

Style map: Style 0 = neural Pixel Art; Styles 1–7 = on-device look filters (Steampunk / 3D Relief / Clay / Oil Painting / Victorian / Retro Film / Cartoon, 4–18 ms each); Style 8 = neural AI Enhance (denoise, composed with high photo-luma weight to preserve realism); Style 9 = No-AI JPEG; Styles 10–15 = the artistic pack (Picasso Cubist / Van Gogh Swirl / Watercolor / Pencil Sketch / Pop Art Warhol / Mosaic Glass, 8–25 ms each), all using the pure-style crossfade compose so they read as generative restructuring rather than a color filter.

Neural tiers (DEC-022): selectable at runtime — 32 Fast (1.49 s), 48 (3.37 s), 64 Balanced (6.01 s), 96 (13.54 s), 128 Best (24.08 s), measured on hardware. Latency follows O(N²) to within 1%.

Photo pipeline: Shutter saves the original at sensor resolution (1600×1200 UXGA JPEG, sensor bytes written directly, no re-encode — DEC-050) while separately downsampling the live 240×240 preview frame to N×N for the neural/CPU style path; converting composes a hybrid export and saves it as the paired _styled.jpg.

Joystick UX (DEC-034/047): Center tap = shutter, Center hold = menu (Files / Nano Ai / Config); input is sampled on a dedicated core-0 task independent of render speed, not once per render loop.

4. NanoAi Studio Web UI (ui/index.html)

  • Neural Quality Tier selector: train/flash commands per size + VERSION nn_size mismatch warning.
  • How NanoAi Generative Style Works: user-facing stack, train data, photo pipeline, style table.
  • Model Training studio: Train Model runs the real pipeline locally via training/dashboard_server.py (train → INT8 quantize → firmware header + ui/models/*.tflite), with live epoch progress and a pixel-art T-Rex training mascot; falls back to CLI pointers when no local server is behind the page (no browser-side TensorFlow either way — training itself always runs as a real Python subprocess, never simulated in JS).
  • Model Test Lab: upload an image, pick a trained model (from the manifest or loaded from disk), run the real exported INT8 flatbuffer via TFLite WASM, see input/output side by side.
  • Style selection panel: checklist over all 16 styles → STYLESEL bitmask sent over WebSerial, persisted in NVS; drives which styles the on-device menu and dashboard row show.
  • Generative Style Selector: dual canvas preview for all 16 styles, filtered to the current selection.
  • WebSerial device panel: LIST / SNAP / VERSION / DIAG / STYLES / STYLESEL — live firmware only (DEC-013).

5. Large Generative Models (FLUX / SDXL) vs. NanoAi Studio Edge TinyML CNN

Attribute Large Diffusion Models (FLUX / SDXL) NanoAi Studio Edge TinyML CNN (ESP32-S3)
Parameter Count 1.2 Billion to 12 Billion parameters 3,347 (constant — fully convolutional)
Memory Footprint 4GB to 24GB VRAM (NVIDIA RTX 4090) 13,280 B INT8 weights + 160 KB–1.1 MB arena
Generative Mechanism Iterative Latent Denoising (UNet / DiT + Text Attention) Depthwise Separable Image-to-Image Convolution
Connectivity Requires Cloud GPU Servers 100% Disconnected On-Device Edge Compute
Latency 1.5s to 15s per image 1.5 s (N=32) to 24 s (N=128), measured