GEEKATPLAY STUDIO · PROGRAMME STATUS · AS OF AUGUST 2026 · VLADIMIR CHOPINE ON LINKEDIN ↗
TinyML generative style camera on ESP32-S3 — TensorFlow train → INT8 TFLite Micro on device.
Not CLIP, VAE, or Stable Diffusion. Shoot · style · save — 100% offline.
Geekatplay Studio · Vladimir Chopine
How it works · Model flow · Dashboard · Installation · Hardware · Architecture · Decision log · Serial protocol · XL model plan
| NanoAi is | NanoAi is not |
|---|---|
| TinyML image-to-image style camera | Stable Diffusion / FLUX |
| TensorFlow → TFLite Micro on ESP32-S3 | CLIP / text prompts |
| ~3.4k params → ~13KB INT8 model | VAE / latent diffusion |
| 16 styles: 2 neural + 14 on-device generative looks, pick which ship | Cloud GPU / LoRA hubs |
| Dual JPEG: photo + styled hybrid | Billion-param foundation models |
| Train + test the real model in the browser (dashboard server) | A canvas filter pretending to be AI |
Authority: docs/DECISION_LOG.md · docs/ARCHITECTURE.md · docs/SERIAL_PROTOCOL.md · firmware/src/
📷 Hardware: everything here runs on the Freenove ESP32-S3-N16R8 Media Kit — get it on Amazon (affiliate link — purchases may earn Geekatplay Studio a small commission at no extra cost to you).
flowchart LR
subgraph Desktop["Desktop · TensorFlow / Keras"]
A["Synthetic N×N pairs"] --> B["Depthwise CNN"]
B --> C["INT8 quantize"]
C --> D["models/model_nn_N.h"]
end
D -->|pio upload| Cam
subgraph Device["ESP32-S3 · TFLite Micro"]
Cam["OV3660 full-auto"] --> Crop["Center crop → N×N"]
Crop --> Style{"Style"}
Style -->|0| NN["Neural Pixel Art"]
Style -->|8| EN["Neural AI Enhance"]
Style -->|1–7,10–15| CPU["13 look filters"]
Style -->|9| JPEG["No-AI JPEG"]
NN --> Hybrid["Hybrid gen-over-photo"]
EN --> Hybrid
CPU --> Hybrid
Hybrid --> SD["MicroSD dual JPEG"]
JPEG --> SD2["/pictureN.jpg"]
end
The dashboard's scrubbable pipeline, paused on the neural stage. Every box is a real step in firmware/src/inference_pipeline.cpp.
- Shutter /
SNAP— joystick tap or serial; LED purple while processing - Original — camera switches to full-sensor JPEG (1600×1200 UXGA, ~85 KB) and saves the sensor's own JPEG bytes straight to SD as
/pictureN_normal.jpg— no decode/re-encode, no quality loss. Toggle: Config → HiRes original (on by default; falls back to the 240×240 preview-size save if the mode switch fails). For 90°/270° capture rotations the file carries an EXIF Orientation tag instead of rotated pixels, so it always previews the way the LCD showed it — see the styles/serial protocol section for why - Neural input — separately, the live 240×240 RGB565 preview frame (115,200 B) is center-cropped and downsampled to the selected neural size
N(32–128, runtime switchable) - Quantize float
[0,1]→ int8 using the tensor's own scale and zero-point - Style route (only on
CONVERT/SHOOT, or from the on-device Nano Ai menu) — only the styles selected in the dashboard's STYLESEL panel are offered- 0 Pixel Art / 8 AI Enhance — TFLite Micro
Invoke→ dequantize - 1–7, 10–15 — on-device generative look filters (4–22 ms)
- 9 No AI — hardware JPEG only
- 0 Pixel Art / 8 AI Enhance — TFLite Micro
- Export — hybrid blend (photo luma + style chroma), crossfading to the pure generated style above 70% strength for the geometric looks (Pixel Art, Cartoon, and the whole artistic pack) →
/pictureN_styled.jpg, paired by number with the original
The dashboard (ui/index.html) animates each of these live and interactively.
Static frames below; open the page to scrub, step, and play them.
Conv2D-16 (3×3) → SepConv-32 → SepConv-32 → SepConv-16 → Conv2D-3 (1×1, sigmoid).
Because it is fully convolutional, the same weights run at any input size — which is
exactly what makes a runtime quality menu possible. A bigger tier is not a bigger brain;
it is the same net seeing more pixels.
Splitting a convolution into a depthwise pass plus a 1×1 pointwise pass costs
1/C_out + 1/(K_w·K_h) of a standard convolution — roughly 1/9 here. That is the
difference between fitting on this microcontroller and not.
Weights and activations are mapped to 8-bit integers by real = scale × (q − zero_point).
The Xtensa LX7 has no NPU and no useful float SIMD, so integer MACs are the only fast path.
Result: a 13,280-byte model that runs entirely on-chip.
Arena demand is 2 · N² · 32 + 96 KB — two live activations of the widest (32-channel)
layer. Tiers 32 and 48 fit in on-die SRAM; 64 and up spill to PSRAM. The weights are
identical at every tier.
One styled capture, with real byte counts: sensor → PSRAM framebuffer → crop → int8 tensor → arena → compose → JPEG encode → MicroSD.
16 selectable looks — 2 real neural nets, 14 on-device generative filters. The
dashboard's Styles your device should support panel (S2) lets you narrow
this down to a subset (STYLESEL bitmask, persisted in NVS); the on-device
menu and BOOT-cycle only ever show what you picked. 9 No AI can never be
disabled.
| # | Style | Engine | LED |
|---|---|---|---|
| 0 | Pixel Art | Neural TinyML CNN (INT8) | Cyan |
| 1 | Steampunk | Copper / amber color matrix | Amber |
| 2 | 3D Relief | Emboss / metallic depth look | White |
| 3 | Clay / Ceramic | Soft terracotta facets | Terracotta |
| 4 | Oil Painting | Smoothed daubs + darkened edges | Gold |
| 5 | Victorian | Sepia + vignette + plate grain | Sepia |
| 6 | Retro Film | Faded curve + warm cast + grain | Orange |
| 7 | Cartoon | Flattened color + bold ink edges | Magenta |
| 8 | AI Enhance | Neural TinyML denoise (INT8) | Teal |
| 9 | No AI | Original JPEG — no network | Blue |
| 10 | Picasso Cubist | Sheared facets, displaced sampling, ink seams | Violet |
| 11 | Van Gogh Swirl | Vortex-warped resampling, stroke bands | Yellow |
| 12 | Watercolor | Pigment pooling, paper-white lift, wet edges | Sky |
| 13 | Pencil Sketch | Graphite edges + shadow cross-hatch | Silver |
| 14 | Pop Art Warhol | 4 flat silkscreen inks + halftone dots | Pink |
| 15 | Mosaic Glass | Jittered flat tiles + dark grout | Emerald |
| — | Camera idle | Ready | Green |
| — | Processing | Inference / SD write | Purple |
| — | Error | Camera / SD fault | Red |
Styles 10–15 (and 0/7 at high strength) use a pure-style crossfade in the
compose step — above 70% strength the export drops the photo's luma channel
entirely, so the geometric looks (Picasso, Mosaic, Pixel Art…) read as real
generative restructuring, not a color filter over the photo. Training targets
for the whole pack: python -m model.train --style picasso|van_gogh|watercolor|sketch|pop_art|mosaic --size N.
Joystick UX: Center tap → shutter · Center hold → menu (Files / Nano Ai / Config) · in Files, Center arms delete, Center again confirms · BOOT button is the white LED torch, unrelated to the shutter.
Full command/response contract, including STYLE, STYLES, and STYLESEL:
docs/SERIAL_PROTOCOL.md.
The same engineering dashboard ships in-repo — ui/index.html plus a small local Python server, training/dashboard_server.py, that turns two panels from "prints a CLI command" into "actually does the thing":
| Module | What you get |
|---|---|
| How GenAI works — animated | Six interactive canvas modules: scrubbable capture→quantize→conv→compose pipeline, network architecture with feature-map animation, live training-pair visualization ported from dataset.py, interactive float→int8 quantization slider, tier/arena memory animation, and an animated ESP32-S3 board schematic with real byte counts |
| Proof it's real AI | Measured-on-hardware evidence box: the ~500× cost gap, the O(N²) curve, the exact arena match — each with a plain-language explanation |
| Technical deep dive | Expert text alongside the animations: silicon, capture path byte-for-byte, per-layer parameter and multiply counts, memory map, latency per tier |
| Neural Quality Tiers | Pick 32 / 48 / 64 / 96 / 128 — each showing its real measured wait (~6.0 s/photo), arena size, and SRAM/PSRAM placement |
| 🚀 Train Model | Runs the real python -m model.train pipeline locally through the dashboard server — train → INT8 quantize → firmware header + ui/models/*.tflite — with live epoch progress and a pixel-art T-Rex mascot who hauls cubes into a pyramid as training advances, bows and celebrates at 100% (gray + head-down on failure) |
| 🔬 Model Test Lab | Upload any image (or use the sample), pick a trained model by size/style, and run the exact exported INT8 flatbuffer in-browser via TFLite WASM — not a canvas approximation. Also accepts a .tflite loaded straight from disk, which is the only path that works when the page is opened as file:// |
| Style selection | Tick which of the 16 styles the device should offer, Apply to Device sends STYLESEL over WebSerial (persisted in NVS) — the on-device menu and BOOT-cycle then only show what you picked |
| Style studio | Dual-canvas preview for all 16 styles, each tagged with its measured cost |
| BOOT + LED sim | Short = TAP · hold = HOLD — mirrors device state machine |
| WebSerial hub | Connect · VERSION · DIAG · QUALITY / ROTATE / STYLESEL controls · live 115200 log · style sync |
| MicroSD photo gallery | LIST real /pictureN_*.jpg · JPEG thumbnails · normal+styled pairs on one card with before/after slider · per-file and bulk download · DUMP progress bar |
| Zero makeup data | No fake files, versions, or capacities — live device only (DEC-013) |
cd training
python dashboard_server.py
# → http://127.0.0.1:8137 (serves ui/ + a local /api/train endpoint)Opening ui/index.html directly from disk (file://) still works for WebSerial,
style selection, and the canvas previews — the page shows an amber banner and
degrades Train Model to printing the CLI command, since Chrome blocks a
file:// page from calling a local server. The Model Test Lab still works
there via "load .tflite from disk."
Disconnect WebSerial before every
pio run -t upload— Chrome owns the COM port.
Switch tiers at runtime — no reflash:
QUALITY 64 # serial command, persisted to NVS…or from the device Config → Quality menu (Fast 32 / 48 / Balanced 64 / 96 / Best 128).
The PlatformIO envs (esp32s3, esp32s3_48, esp32s3_64, esp32s3_96, esp32s3_128)
now only set the default tier for a fresh device.
All five tiers ship real trained INT8 models (~13.3 KB each — the network is fully
convolutional, so parameter count is constant across tiers; only the tensor arena grows).
Every model is embedded in flash and the tier is selectable at runtime from the
device Config menu or the QUALITY n serial command — no reflash needed.
Times from [PERF] logs on a Freenove FNK0102A (ESP32-S3 N16R8, OV3660, -O3 build):
| Quality | Tensor arena | Inference | Compose | Styled export |
|---|---|---|---|---|
| 32 Fast | 160 KB (internal SRAM) | 1.49 s | 0.23 s | 320×320 |
| 48 | 240 KB (internal SRAM) | 3.37 s | 0.24 s | 320×320 |
| 64 Balanced | 352 KB (PSRAM) | 6.01 s | 0.24 s | 320×320 |
| 96 | 672 KB (PSRAM) | 13.54 s | 0.53 s | 480×480 |
| 128 Best | 1120 KB (PSRAM) | 24.08 s | 0.53 s | 480×480 |
Three independent measurements, all read off the running device — no estimates:
| Evidence | Measurement | Why a filter can't do this |
|---|---|---|
| ~500× cost gap | Neural style at N=64: 6,006 ms. CPU look filters on the same frame and same 320×320 export: 4–18 ms (retro 4, victorian 5, cartoon 12, oil 18). | A look filter is one arithmetic pass over the pixels — cheap by construction, and it cannot be made to take seconds. Spending three orders of magnitude more time means three orders of magnitude more arithmetic. |
| O(N²) within 0.7% | Predict tier 64 from tier 32 alone: 1,491 × (64/32)² = 5,964 ms. Measured: 6,006 ms. Holds through 96 and 128. |
A fully convolutional net does fixed work per pixel, so doubling the side must quadruple the time — with no term depending on image content. A preset or cached result would be flat in N. |
| Arena matches layer shapes exactly | At N=128 TFLite requests 1,048,576 B = 2 × 128² × 32, zero residual, across all five tiers. |
That number is not chosen by us — the interpreter derives it from the model's own layer shapes. It matching the topology means real tensors are being allocated for real convolutions. |
Two myth-busters from the same testing: moving the arena from PSRAM to internal SRAM made
no difference (2,043 vs 2,049 ms — the S3 data cache already covers it), and building
with -O3 instead of -Os gave a 27% speedup.
The original as actually saved: 1600×1200 UXGA JPEG, ~85 KB, straight off the sensor via DUMP. No decode/re-encode.
Style samples (earlier firmware, 240×240 preview-resolution captures): Style 0 Pixel Art (neural, N=64), Style 8 AI Enhance (neural denoise), and Style 4 Oil Painting.
Train writes firmware/include/models/model_nn_N.h. Flash the matching env so NANOAI_NN_SIZE matches the model.
Squeezing multi-million-parameter nets onto a tiny MCU is a popular obsession — and usually
impractical for anything interactive. We still measured what that style of network would cost
on this ESP32-S3 N16R8. Phase 0 is done — a hand-written W4A8 kernel (4-bit weights, int8
activations, dual-core row split) hits ~34 MMAC/s on this board, putting a ~5M-param
encoder-decoder style net at roughly 40 s/frame (15–30 s with S3 SIMD). Full plan, measured
tables, and phases: docs/XL_MODEL_PLAN.md. Benchmark firmware ships
in-repo: pio run -e esp32s3_xlbench -t upload.
Dense multi-million-param demos prove the silicon can store a big model. They also show why this project is a camera, not a chatbot:
| Dense / token-style on ESP32-S3 | CNN on ESP32-S3 | |
|---|---|---|
| Memory traffic | Every weight streamed from flash once per step — bandwidth-bound, ALUs idle | Each kernel reused at every pixel — stage weights in SRAM once per layer, arithmetic-bound |
| One second of compute buys | A tiny slice of a huge forward pass | A complete styled photograph (one forward pass = one artifact) |
| Quantization failure mode | Semantic — wrong tokens / broken meaning | Perceptual — slightly softer image |
| Working memory | Caches / state that grow with sequence length | Two fixed ping-pong activation buffers, known at compile time |
Same silicon, same watt: the convolutional shape simply matches what a 240 MHz dual-core MCU with slow external memory can win at. That's why NanoAi is an image camera.
| Requirement | Notes |
|---|---|
| Freenove ESP32-S3 Media Kit (FNK0102A) | Buy on Amazon — affiliate link (purchases via this link may earn Geekatplay Studio a small commission at no extra cost to you) |
| MicroSD card | Any class-10 card, FAT32; photos are stored here |
| USB-C data cable | Must be a data cable, not charge-only |
| Python 3.10+ | For model training (training/) |
| PlatformIO | CLI (pip install platformio) or the VS Code extension |
| Chrome or Edge | For the WebSerial dashboard (ui/) |
git clone <this-repo>
cd MicroLM-MicroAiGen/training
python -m pip install -r requirements.txt
python -m pytest tests/ -q # expect all tests greenThe pins in
requirements.txtmatter: NumPy stays<2to avoid the_ARRAY_API not foundpyarrow clash (see Troubleshooting).
python -m model.train --style pixel_art --size 32 # style model
python -m model.train --model enhance --size 32 # AI Enhance (denoise) modelExpect TFL3 OK and exports into firmware/include/models/. Repeat with
--size 48/64/96/128 for the higher quality tiers — all sizes are embedded in
firmware and selectable at runtime from the device menu (QUALITY command).
Windows driver note: the kit's USB-serial chip (CH343) usually installs automatically; if the board does not appear as a COM port, install the CH343 driver from the Freenove docs, unplug and replug USB.
# Disconnect WebSerial / Serial Monitor first — Windows serial is exclusive!
cd ../firmware
pio run -e esp32s3_64 -t uploadPlatformIO auto-detects the port. First build downloads the toolchain (~1–2 GB, one time). A successful flash ends with the board rebooting into the splash screen with the Geekatplay Studio branding.
cd ../training
python dashboard_server.py
# → http://127.0.0.1:8137 (Chrome/Edge)This also enables the Train Model button and the Model Test Lab (see
above); cd ui && python -m http.server still works for WebSerial-only use.
Click Connect, pick the board's COM port — the dashboard auto-sends
VERSION + LIST and syncs styles, quality tier, and the photo gallery.
- Joystick Center tap → shutter (purple while saving, green when done)
- Joystick Center hold → main menu (Files / Nano Ai / Config); full 5-way works once the camera is paused for menu navigation
- In Nano Ai, pick a style and confirm to convert the last shot
- In Files, browse with Left/Right; Center arms delete, Center again confirms
- BOOT button = white LED torch on/off (unrelated to shutter)
- Check MicroSD:
pictureN_normal.jpg(1600×1200 original) +pictureN_styled.jpg(if converted)
LIST · SNAP · CONVERT · SHOOT · XL · DELETE · STYLE n · STYLES · STYLESEL m · QUALITY n · ROTATE n · VERSION · DIAG · DUMP /path
XL runs the multi-million-parameter model from the xlmodel flash partition on the last
still (~70 s/frame — capture-then-render, see the XL model plan).
Flash a model first: python -m model.train_xl --style pixel_art --config C then
training/flash_xl_model.ps1 firmware/xl/pixel_art_C.nq4.
Debug-only (drive the device without physical access): MENU, NAV UP|DOWN|LEFT|RIGHT|CENTER, LCDSHOT (JPEG-dumps whatever the LCD is currently showing). Full contract: docs/SERIAL_PROTOCOL.md.
SHOOT captures and styles in one step, saving the pair /pictureN_normal.jpg +
/pictureN_styled.jpg.
Full contract: docs/SERIAL_PROTOCOL.md
Nano-Ai/
├── training/ # Python only — dataset, train, INT8, export
│ ├── model/ # network / dataset / train / quantize / exporter
│ └── dashboard_server.py # serves ui/ + local /api/train for the dashboard
├── firmware/ # C++ only — camera, TFLite Micro, UI, SD, LED
│ └── include/models/
│ ├── model_nn_{32,48,64,96,128}.h ← style models (generated)
│ └── enhance_nn_{32,48,64,96,128}.h ← AI Enhance models (generated)
├── ui/ # Engineering dashboard + WebSerial + animated explainer
│ ├── index.html · app.js · edu.js · style.css
│ └── models/ # local Model Test Lab artifacts (gitignored — regenerate by training)
└── docs/ # Architecture, decisions, serial protocol, assets
| Boundary | Rule |
|---|---|
| Python / TF / pytest | stays in training/ |
| C++ / PIO / TFLite Micro | stays in firmware/ |
| Bridge | generated model_nn_N.h / enhance_nn_N.h, looked up at runtime via model_data.h; the same .tflite bytes are also copied to ui/models/ for the browser Model Test Lab |
# Python
cd training && python -m pytest tests/ -q
# Host C++ math (crop / quant / hybrid compose) — builds src/inference_pipeline.cpp
# for the host per platformio.ini's [env:native] (test_build_src + build_src_filter)
cd firmware && pio test -e nativeFreenove ESP32-S3 Media Kit (FNK0102A) — available on Amazon (affiliate link) (this is an affiliate link; purchases may earn Geekatplay Studio a small commission at no extra cost to you).
| Spec | Value |
|---|---|
| MCU | ESP32-S3 N16R8 — 2× Xtensa LX7 @ 240 MHz |
| SRAM | 512 KB on-die (single-cycle) |
| PSRAM | 8 MB octal-SPI (framebuffers + tensor arena live here) |
| Flash | 16 MB (all 10 neural models embedded ≈ 133 KB total) |
| Camera | OV2640/OV3660 DVP — 1600×1200 UXGA JPEG originals, 240×240 RGB565 live/neural |
| Display | ST7789 1.14" IPS, 240×135 landscape |
| Storage | MicroSD via SDMMC 1-bit (CMD 38 · CLK 39 · D0 40) |
| Audio | NS4168 I2S speaker + MEMS microphone |
| Input | 5-way joystick (ADC ladder, GPIO19) + BOOT button |
| LED | WS2812 RGB on GPIO48 |
| Subsystem | Notes |
|---|---|
| Camera | Full auto AWB/AEC/AGC; RGB565 240×240 live/neural; original saved as UXGA JPEG (Config → HiRes original, on by default) |
| Memory | LCD framebuffers + tensor arena live in PSRAM — internal SRAM use is ~14.5% |
| SD | CMD 38 · CLK 39 · D0 40 — never wipe card on boot |
| LED | GPIO 48 = WS2812 only (never drive as LCD backlight) |
| LCD | ST7789 live viewfinder, gallery (newest-first, paired), style picker, progress overlay |
| Joystick | Polled on a dedicated core-0 FreeRTOS task at 10 ms, independent of render speed (tap = shutter, hold = menu) |
| Audio | I2S shutter sound (synthesized two-transient "clack") |
Serial contract between firmware and dashboard: docs/SERIAL_PROTOCOL.md
| Symptom | Fix |
|---|---|
Upload Access is denied |
Disconnect WebSerial / close monitors |
| Style 0 unchanged | Retrain → confirm TFL3 → re-flash matching env |
_ARRAY_API not found |
pip install -r requirements.txt (NumPy 1.x pin) |
| Photos in preview, not on card | Reseat SD · look for [PHOTO_CAPTURED] vs _PREVIEW_ONLY |
| Dead BOOT / LED | Flash current firmware; ensure GPIO48 not used as backlight |
| Joystick taps not registering | Flash current firmware — older builds sampled input once per render loop (~160 ms), which could miss a normal tap entirely; input now runs on its own 10 ms task |
| Gallery photo "looks modified" / wrong colors on LCD | Flash current firmware — the on-device viewer had an RGB565 byte-order bug independent of the saved file; LCDSHOT (serial) dumps exactly what the LCD is showing for verification |
| Refresh empties the dashboard gallery | Flash current firmware — refresh now reconciles against the live file list instead of clearing first; a dropped response can no longer wipe it |
| Original photo previews 90° rotated on gallery/web, but the LCD showed it correctly | Flash current firmware — 90°/270° hi-res originals now carry an EXIF Orientation tag instead of rotated pixels; older captures on the card predate the fix and stay as-is |
| Console shows CORS errors, Train Model / Model Test Lab dropdown empty | Page was opened as file:// — Chrome blocks it from calling a local server. Serve with python training/dashboard_server.py instead (see Studio dashboard features) |
| "Train Model" only prints a CLI command | Page isn't served by training/dashboard_server.py (either file:// or a plain static server) — see above |
Full walkthrough: docs/GETTING_STARTED.md
Created by Geekatplay Studio — Vladimir Chopine.
NanoAi Studio demonstrates honest TinyML generative style on microcontroller hardware — not a foundation multimodal model.
LinkedIn · geekatplay.com · Patreon · YouTube @geekatplay · @geekatplay-ru · @v-code-studio
Hardware: Freenove ESP32-S3 Media Kit (Amazon affiliate link) — may earn Geekatplay Studio a small commission at no extra cost to you.