v2.0.0 - ESPHome Native Integration

[francescopace]

🚀 Major - ESPHome Native Integration

Complete platform migration from ESP-IDF to ESPHome

This release represents a major architectural shift from standalone ESP-IDF firmware to a native ESPHome component, enabling seamless Home Assistant integration.

⚠️ Note: Extensively tested on ESP32-S3 and ESP32-C6, but bugs may still exist. Community contributions, bug reports, and support for additional ESP32 variants are welcome!

🎯 Two-Platform Strategy

ESPectre now follows a dual-platform development model:

Platform	Role	Focus	Target
ESPectre (ESPHome - C++)	Production	Motion detection only	Smart home users, Home Assistant
Micro-ESPectre (Micro Python)	R&D	Features + Filters for research	Researchers, developers, academics

ESPectre focuses on core motion detection for Home Assistant integration.
Micro-ESPectre provides features (variance, skewness, kurtosis, entropy, IQR, spatial_, temporal_) and advanced filters (Butterworth, Wavelet, Savitzky-Golay) for research/ML applications.

New Architecture:

• Native ESPHome component: Full C++ implementation as ESPHome external component
• Home Assistant auto-discovery: Automatic device and sensor registration via Native API
• YAML configuration: All parameters configurable via simple YAML files
• OTA updates: Wireless firmware updates via ESPHome

Implementation:

• components/espectre/: Complete ESPHome component with Python config and C++ implementation
• Modular C++ architecture: calibration_manager, csi_manager, sensor_publisher, etc.
• Binary sensor for motion detection
• Movement score sensor
• Adjustable threshold (number entity) - controllable from Home Assistant

🔄 Micro-ESPectre

R&D Platform for Wi-Fi CSI Motion Detection - Pure Python implementation for MicroPython

Micro-ESPectre is the research and development platform of the ESPectre project, designed for rapid prototyping, algorithmic experimentation, and academic/industrial research. It implements motion detection algorithms in pure Python, enabling fast iterations without compilation overhead.

Key Features:

• ⚡ Instant Deploy: ~5 seconds to update code (no compilation)
• 🔧 MQTT Integration: Runtime configuration via MQTT commands
• 🧬 Auto Calibration Algorithm: Automatic subcarrier selection (F1=97.1%)
• 📊 Analysis Tools: Complete suite for CSI analysis and algorithm optimization
• 🧮 Feature Extraction: Statistical features (variance, skewness, kurtosis, entropy, IQR)
• 🎯 Confidence Score: Experimental motion detection confidence estimation
• 💾 NVS Persistence: Persistent configuration on filesystem

Advanced Applications (ML/DL ready):

• 🔬 People counting
• 🏃 Activity recognition (walking, falling, sitting, sleeping)
• 📍 Localization and tracking
• 👋 Gesture recognition

Dependencies:

• micropython-esp32-csi - Custom MicroPython fork with native CSI support for ESP32 family
• MQTT broker (e.g., Mosquitto)

🧪 Test Suite Refactoring

Migration from Unity (ESP-IDF) to PlatformIO Unity for ESPHome consistency

The test suite has been migrated from ESP-IDF's Unity framework to PlatformIO Unity, aligning with the ESPHome ecosystem and enabling native (desktop) test execution without hardware.

Complete test suite with 68 test cases organized in 5 suites and Memory leak detection:

Suite	Tests	Focus
test_csi_processor	19	API, initialization, validation, memory management
test_hampel_filter	16	Outlier removal filter behavior
test_calibration	21	NBVI algorithm, variance, percentile calculations
test_calibration_file_storage	9	Calibration persistence and file I/O
test_motion_detection	3	MVS performance with real CSI data (2000 packets)

# Run tests locally
cd test && pio test -e native

🔄 CI/CD Pipeline

GitHub Actions integration for automated quality assurance

• Automated testing: Runs on push to main/develop and pull requests
• ESPHome build verification: Compiles espectre.yaml to validate component
• Status badge: Real-time CI status displayed in README
• Path filtering: Only triggers on changes to components/espectre/ or test/

---

This discussion was created from the release v2.0.0 ESPHome Native Integration.

[hrfried]

Haha, I was literally just getting started testing this out about an hour ago (been on my backburner to check out for a bit, and just finally got a new esp32c6 delivered).

I considered working from your esphome testing branch since I already use esphome, but I thought "surely that's a massive undertaking and won't be ready for a while." Then I slowly noticed the documentation I was reading changing before my eyes. :) You work fast!

Glad you opened this discussion because I wanted to ask but it's definitely not an issue or anything:

1. Are you retiring the use of the esp-idf and moving forward solely with esphome now (aside from the micropython development workflow)?
2. I'm assuming since the release is tagged, it's all set and ready for me to reinitialize from v2.0.0? All docs good to go?

[hrfried]

Thanks for the reply! Mostly just thought it was funny and had to share with someone other than my wife. :) Confused me for a second seeing new contradictory information popping up, haha! Esphome worked perfectly it looks like, although I'm sure I may have been moving during initial calibration...

One thing I noticed that you may want to add in the docs is some notes on which components to pull into an existing esphome workspace (maybe these are already here, but I didn't see them). At least for me personally, I run esphome from a super simple docker-compose and have the config dir bindmounted from host. E.g.:

services:
  esphome:
    container_name: esphome
    image: ghcr.io/esphome/esphome
    volumes:
      - /home/dietpi/repos/esphome/config:/config
      - /etc/localtime:/etc/localtime:ro
    restart: always
    privileged: true
    network_mode: host
    secrets:
      - esphome_secret
secrets:
  esphome_secret:
    file: ./esphome_secret.txt

So it was necessary to copy over the components dir and parameters.csv into my bind-mount. In my case that was just

scp -r components dietpi@dietpi8g.home.arpa:/home/dietpi/repos/esphome/config/
scp partitions.csv dietpi@dietpi8g.home.arpa:/home/dietpi/repos/esphome/config/
# validate the config. I renamed from the default espectre-c6-01.yaml
# and exec'd into a shell to run, but same idea
docker compose exec -it esphome esphome config espectre-c6-01.yaml
docker compose up -d --force-recreate

Honestly I don't know if this is a common setup or not, and probably wouldn't take others too long to figure out, but it'd be nice to have in docs which components need to be visible to esphome. ESPHome actually gives pretty crappy error messages from the WebUI if the platform isn't found (e.g. "Configuration does not match the platform of the connected device. Expected an UNKNOWN device.")

Realistically, unless you're planning on upstreaming some of this to esphome, probably makes sense to suggest tracking the repo from the bindmount/esphome config dir? I'm not really sure how esphome development process is run, e.g. if this kind of stuff is normally upstreamed, or if you're planning on going for that or not. I vaguely remember bindmounting like this while trying to screw around with libretiny stuff, which was then upstreamed IIRC. I don't think about it too much day to day, so I'm forgetting, ha.

Kinda rambling, but just a few comments. Excited to play around, and thanks for all your hard work!

[francescopace]

Thanks for the feedback! It's really helpful to improve the setup experience.
I've just pushed some changes that should make things easier!

What's new:

1. partitions.csv is now automatic - The component includes it automatically during build, so you don't need to copy it manually anymore.

2. New examples/ folder with ready-to-use configurations:

   • espectre-c6.yaml - Production config for ESP32-C6 (uses GitHub source)
   • espectre-s3.yaml - Production config for ESP32-S3 (uses GitHub source)
   • espectre-c6-dev.yaml - Development config with debug sensors
   • espectre-s3-dev.yaml - Development config with debug sensors

For Docker / Home Assistant Add-on users:

The simplest approach is to copy one of the example files to your ESPHome config directory and the component will:

• Download automatically from GitHub
• Include the custom partition table
• Run auto-calibration on first boot

Updated documentation

I've also updated SETUP.md with a dedicated section for Docker users.

Let me know if you run into any issues with the new setup!

[hrfried]

Awesome, I'm taking a look and will test soon! Some other questions/comments:

The traffic generator is just creating plain traffic (DNS requests to google.com of latest tag) to generate consistent traffic by which to collect and analyze the CSI "packets," yeah?

1. It looks like you switched over to an "ICMP ping mechanism to default gateway" mechanism for generating CSI in v1.2.0, but now it's back to using DNS—any reason for that?
   a. It's really kinda annoying having so much extra DNS traffic on the network, and I think there's a pretty good chance anyone using this project will be running their own DNS server and/or DNS filtering application haha.
   b. Also, millions of requests to google.com is probably something most people don't want to see from any device on their network
   c. Ideally using ping or something else would be best IMO, no? Not really sure why this changed or if it was just missed during the esphome migration.

2. Assuming ICMP doesn't work well with ESPHome or something, I think the default domain to query should:
   a. Be configurable
   b. Have a default of something that isn't google.com. My first thought was that example.com would work better, but I'm not sure that's really the right choice. Not 100% sure on best practices here, but from https://datatracker.ietf.org/doc/rfc6761/ it looks like it's recommended to use e.g. any dummy domain with the .test TLD, which is reserved for this use case (see section 6.2 of RFC).
   c. I wouldn't think the return value would matter too much, would it? I.e. if we're using csi-generate.test and it returns NXDOMAIN/0.0.0.0/whatever based on your DNS server config, we should still be able to generate and measure CSI the exact same, right?

Also:
Kinda an aside and unrelated to above comments, but mostly just curiosity. Have you played at all with higher frequency rates for feature detection/filtering? E.g. are you able to do anything interesting in terms of filtering/detection at sample rates above 100 Hz?

For example, if you have a ceiling fan or a box fan running somewhere in the range of 10-100Hz, but with multiple blades (and thus a frequency of motion through any particular cross-sectional area/volume at some multiple of that), you could potentially do some interesting stuff if your sampling frequency is high enough...I haven't really played around with any of this enough to test, but curious if you have?

Could be interesting for, e.g. detecting if a fan or central HVAC kicks on/off? Not sure what kind of vibrational amplitude is necessary to cause changes detectable via CSI, but could be a lot of potentially interesting stuff there I think.

[francescopace]

Let me address each point:

Traffic Generator & DNS

The traffic generator creates WiFi packets to ensure consistent CSI data collection. Without traffic, no CSI packets are generated.

Good catch on the v1.2.0 changelog! Actually, I experimented with ICMP but went back to DNS for these reasons:

1. ICMP can be blocked - Many routers/firewalls block or rate-limit ICMP echo requests
2. Raw sockets complexity - ICMP requires raw sockets with special permissions on ESP-IDF
3. DNS always works - No router blocks DNS to its own gateway

The DNS query is sent to the gateway (your router) on port 53, not to any external DNS server. I use a minimal root domain query (just . - 17 bytes) that generates a response from your router's DNS resolver. The query never leaves your local network.

I had already fixed this in Micro-ESPectre (the MicroPython platform) a while ago, but honestly I forgot to port the change to ESPectre (the ESPHome C++ component). Thanks for bringing it up - it's now fixed and committed!

// Minimal DNS query for root domain (type A)
// 17 bytes - smallest valid DNS query
static const uint8_t DNS_QUERY[] = {
    0x00, 0x01,  // Transaction ID
    0x01, 0x00,  // Flags: standard query
    0x00, 0x01,  // Questions: 1
    0x00, 0x00,  // Answer RRs: 0
    0x00, 0x00,  // Authority RRs: 0
    0x00, 0x00,  // Additional RRs: 0
    0x00,        // Root domain (empty label)
    0x00, 0x01,  // Type: A
    0x00, 0x01   // Class: IN
};

So:

• No traffic to google.com
• No external DNS queries
• Minimal 17-byte payload
• Query stays on your LAN (gateway only)
• Works even if you run Pi-hole, AdGuard, or custom DNS

The response (likely NXDOMAIN or empty) doesn't matter - we just need the WiFi packet to fly through the air to generate CSI.

Higher Frequency Rates

Great question! Yes, I've experimented with rates up to 500 pps.

What I've found:

• Above ~200 pps, diminishing returns for motion detection
• ESP32 CPU usage increases significantly

For fan detection specifically:

• Ceiling fans typically run 50-200 RPM (0.8-3.3 Hz blade frequency)
• With 3-5 blades: 4-17 Hz effective frequency
• 100 pps (Nyquist: 50 Hz) should be sufficient to detect this

The challenge isn't sampling rate but amplitude - fan blades cause very subtle CSI variations compared to human motion.
I haven't specifically tested HVAC detection, but it's an interesting use case! The vibrations might be too subtle for CSI, but airflow changes could be detectable. Would love to hear if you experiment with this.

[hrfried]

Awesome, thanks for your detailed replies! I figured the switch back was likely because of greater overhead/potential to be blocked but didn't see anything in changelogs, so I just wanted to make sure I wasn't missing something! It was pretty clear that it wasn't doing anything crazy from skimming the code, but I can't be the only one to raise an eyebrow any time I see excessive requests to google from a device on my network, even if I know what it's doing haha.

In any case I think it looks much better (and is probably ever so slightly more efficient) querying just the root domain, and can confirm that it works perfectly after rebuilding from git HEAD. BTW—new esphome workflow works perfectly and I greatly appreciate the split config to allow pulling from local repo or upstream.

This is really an awesome project, and it's even nicer when it's clearly a labor of love from someone incredibly knowledgeable. I work in biomedical automation/imaging, although more on the devops side these days (used to run the lab, and my educational background is a hodgepodge of biochemistry/mathematics/biomedical engineering). So projects like this are just an awesome way to scratch that experimental itch...I've got 3 more esp32-s3's on the way and am excited to play around more! Will definitely keep you posted if I end up figuring anything interesting out!

Thank you! :)