Expose embassy_net::Stack

[28Smiles]

I want to use matter and listen on a tcp socket.

What is the recommended way of doing this?

[ivmarkov]

Can you provide some more information? As in, what chip, and then what transport (as in ethernet, wifi or thread)?

Also, do you plan to use it in coex mode or not?

[28Smiles]

I have an esp32-c3 using esp-wifi. I am using wifi commisioning (EmbassyEthernet::new(PreexistingEthDriver::new(wifi_interface), stack). With this the driver is capturing the wifi interface which is not shareable

[ivmarkov]

I have an esp32-c3 using esp-wifi. I am using wifi commisioning (EmbassyEthernet::new(PreexistingEthDriver::new(wifi_interface), stack). With this the driver is capturing the wifi interface which is not shareable

(TL;DR: Look at the https://github.com/ivmarkov/rs-matter-embassy/blob/master/examples/esp/src/bin/light_wifi.rs)

Long story:
This does not make sense. If you use esp-wifi, why do you use PreexistingEthDriver in the first place? You should be using PreexistingWireless instead or even let the Matter stack manage your wireless completely by passing to it the modem peripheral.

I mean, in the real world, your Wifi SSID and password are supposed to be provisioned in the device by the Matter controller itself but for this to happen using PreexistingEthDriver with a Wifi interface is wrong in the first place. Look at the comment here from where you likely copied.

[28Smiles]

But am i able to open an additional socket once the the SSID and Password are provisioned? Smth. Like a custom WifiDriverTask?

[ivmarkov]

If you use PreexistingWireless you should be able even now. But that means you need to bring up the Wifi stack (in disconnected mode of course) and the BLE stack yourself and then pass them to PreexistingWireless.

Now, if you do that, you can open a TCP socket simply because the wireless "pre-exists" and as such you can open your TCP socket anytime - at least at 0.0.0.0. If you need a client TCP socket which can only be opened when the wifi is connected (e.g. for connecting to an MQTT server or so, read on below).

===

If - however - you use any of the other run_* methods, they do currently take a "user" future, but this future does not have access to the UDP that in that case Matter itself creates. Now, I just implemented a change where the "user" future is no longer a simple Future, but rather an async trait (UserTask) with a run method that gets the UDP and Netif, so you'll be able to open your TCP socket from within this UserTask impl even if you don't use PreexistingWireless.

Now, the above would likely land in ~ 2 to 3 weeks. Reason - it is easier for me if I push it as part of a bigger redesign I'm working on ATM, and which is related to this commit in rs-matter as well as a couple follow-up ones.

So you'll have what you are asking for, but you have to wait a bit.

With that said, keep in mind that your UserTask would run only when Matter establishes a Wifi connection, and would only run while this connection is active. If the Wifi connection is lost, then Matter will try to re-connect and therefore it will stop your user task. Upon successful connection, your task will re-run again, until the next disconnect event happens.
Not sure if this lifecycle would work for you, so would be good to actually confirm.deny that. Perhaps explaining why you need this separate TCP socket would help us figure out if that would work or not.

[28Smiles]

Now, if you do that, you can open a TCP socket simply because the wireless "pre-exists" and as such you can open your TCP socket anytime - at least at 0.0.0.0.

If i understand you right, i will be able to open a socket this way and keep it open all the time (before and after commisioning), connecting to it from the outside. <- This is what i need.

The new UserTask should also enable that, with the difference of winding down and up the socket on wifi disconnect?

My goal is to host a small webapplication using picoserve directly from the device offering additional configuration options (not available in matter) and a REST api. Another cool feature would be to attach this to the mdns announcement.

[ivmarkov]

My goal is to host a small webapplication using picoserve directly from the device offering additional configuration options (not available in matter) and a REST api. Another cool feature would be to attach this to the mdns announcement.

For this either option (i.e. PreexistingWireless or the new upcoming UserTask) would do, as you are opening the socket on 0.0.0.0 and it is a server socket. In other words, that should work even if the Wifi interface is currently disconnected (as it would be initially, with the PreexistingWireless option). That is, unless I'm missing a detail as to how embassy-net / smoltcp are supposed to work.

For advertising your service with mDNS, you can't use the "built-in" mDNS in rs-matter as it is (by design) exclusive to rs-matter only and thus conserves some memory. But you could try using this guy specifically which was created for precisely that purpose.

[28Smiles]

For this either option (i.e. PreexistingWireless or the new upcoming UserTask) would do, as you are opening the socket on 0.0.0.0 and it is a server socket. In other words, that should work even if the Wifi interface is currently disconnected (as it would be initially, with the PreexistingWireless option). That is, unless I'm missing a detail as to how embassy-net / smoltcp are supposed to work.

Nice, i will try this, I was unsure if the socket will persist if created before comissioning.

For advertising your service with mDNS, you can't use the "built-in" mDNS in rs-matter as it is (by design) exclusive to rs-matter only and thus conserves some memory. But you could try using this guy specifically which was created for precisely that purpose.

I will also try this.

Thank you very very much for the projects and your support.

[ivmarkov]

Make it run then thank me. :) The project is still in its early stages so there will be issues down the road, I'm sure.

[28Smiles]

Hi,

here i am again, needing some help. So i got PreexistingWireless to compile but it aint commissioning:

    // Configure and start the Wifi first
    let wifi = peripherals.WIFI;
    let (wifi_interface, controller) =
        esp_wifi::wifi::new_with_mode(&init, wifi, WifiStaDevice).unwrap();

    let store = stack.create_shared_store(DummyKvBlobStore);
    let mut resources = EnetMatterStackResources::new();
    let (wifistack, mut runner) = create_enet_stack(wifi_interface, u64::from_le_bytes({
        let mut seed = [0; size_of::<u64>()];
        stack.matter().rand()(&mut seed);
        seed
    }), &mut resources);
    // TODO Use the wifistack to create the http tcp socket.
    let mut matter = pin!(stack.run(
        PreexistingWireless::new(
            EnetNetif::new(wifistack),
            Udp::new(wifistack, make_static!(EnetMatterUdpBuffers, EnetMatterUdpBuffers::new())),
            EspWifiController::new(controller),
            TroubleBtpGattPeripheral::new(
                ExternalController::<_, 20>::new(BleConnector::new(init, peripherals.BT)),
                stack.matter().rand(),
                make_static!(TroubleBtpGattContext<CriticalSectionRawMutex>, TroubleBtpGattContext::new()),
            ),
        ),
        &store,
        (NODE, handler),
        core::future::pending(),
    ));

    select3(
        &mut matter,
        &mut device,
        &mut pin!(async {
            runner.run().await;
            #[allow(unreachable_code)]
            Ok(())
        }),
    )
        .coalesce()
        .await
        .unwrap();

I only show changed lines. The log is showing:

I (30) boot: ESP-IDF v5.1.2-342-gbcf1645e44 2nd stage bootloader
I (30) boot: compile time Dec 12 2023 10:50:58
I (31) boot: chip revision: v0.3
I (34) boot.esp32c3: SPI Speed      : 40MHz
I (39) boot.esp32c3: SPI Mode       : DIO
I (44) boot.esp32c3: SPI Flash Size : 4MB
I (49) boot: Enabling RNG early entropy source...
I (54) boot: Partition Table:
I (58) boot: ## Label            Usage          Type ST Offset   Length
I (65) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (72) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (80) boot:  2 factory          factory app      00 00 00010000 003f0000
I (87) boot: End of partition table
I (92) esp_image: segment 0: paddr=00010020 vaddr=3c130020 size=2f644h (194116) map
I (143) esp_image: segment 1: paddr=0003f66c vaddr=3fc8bb58 size=009ach (  2476) load
I (144) esp_image: segment 2: paddr=00040020 vaddr=42000020 size=11e3e4h (1172452) map
I (409) esp_image: segment 3: paddr=0015e40c vaddr=3fc8c504 size=00a14h (  2580) load
I (410) esp_image: segment 4: paddr=0015ee28 vaddr=3fcc0500 size=00304h (   772) load
I (415) esp_image: segment 5: paddr=0015f134 vaddr=40380000 size=0bb54h ( 47956) load
I (439) boot: Loaded app from partition at offset 0x10000
I (439) boot: Disabling RNG early entropy source...
INFO - Starting...
INFO - mDNS service published: XXXXXXXXX::Commissionable(3840)
INFO - Pairing Code: XXXX-XXXX-XXX
INFO - SetupQRCode: [MT:-XXXXXXXXXXXXXXXXXXXXXXX]
INFO - █████████████████████████████████████
INFO - █████████████████████████████████████
HERE IS THE QR CODE
INFO - █████████████████████████████████████
INFO - █████████████████████████████████████
INFO - Running Matter transport
INFO - Waiting for orphaned RX packets
INFO - Responder: Creating 4 handlers
INFO - Responder: Handlers size: 10696B
INFO - Busy Responder: Creating 2 handlers
INFO - Busy Responder: Handlers size: 712B
INFO - Starting device loop...

[ivmarkov]

Did you first run the example I pasted?
I strongly suggest you run that one first to verify that you have at least a successful commissioning with BLE+Wifi stock example, before starting to modify it.

[28Smiles]

yes, it worked stock

//! An example utilizing the `EmbassyWifiMatterStack` struct.
//!
//! As the name suggests, this Matter stack assembly uses Wifi as the main transport,
//! and thus BLE for commissioning.
//!
//! If you want to use Ethernet, utilize `EmbassyEthMatterStack` instead.
//! If you want to use concurrent commissioning, call `run_coex` instead of `run`.
//! (Note: Alexa does not work (yet) with non-concurrent commissioning.)
//!
//! The example implements a fictitious Light device (an On-Off Matter cluster).
#![no_std]
#![no_main]

extern crate alloc;
use core::pin::pin;

use alloc::boxed::Box;
use bt_hci::controller::ExternalController;
use embassy_executor::Spawner;
use embassy_futures::select::{select3, select4};
use embassy_sync::blocking_mutex::raw::CriticalSectionRawMutex;
use embassy_time::{Duration, Timer};

use esp_backtrace as _;
use esp_hal::gpio::{Level, Output};
use esp_hal::{clock::CpuClock, timer::timg::TimerGroup};
use esp_wifi::ble::controller::BleConnector;
use esp_wifi::wifi::WifiStaDevice;

use log::info;
use rs_matter_embassy::ble::{TroubleBtpGattContext, TroubleBtpGattPeripheral};
use rs_matter_embassy::enet::{create_enet_stack, EnetMatterStackResources, EnetMatterUdpBuffers, EnetNetif, Udp};
use rs_matter_embassy::epoch::epoch;
use rs_matter_embassy::eth::EmbassyEthMatterStack;
use rs_matter_embassy::matter::data_model::cluster_basic_information::BasicInfoConfig;
use rs_matter_embassy::matter::data_model::cluster_on_off;
use rs_matter_embassy::matter::data_model::device_types::DEV_TYPE_ON_OFF_LIGHT;
use rs_matter_embassy::matter::data_model::objects::{Dataver, Endpoint, HandlerCompat, Node};
use rs_matter_embassy::matter::data_model::system_model::descriptor;
use rs_matter_embassy::matter::utils::init::InitMaybeUninit;
use rs_matter_embassy::matter::utils::select::Coalesce;
use rs_matter_embassy::rand::esp::{esp_init_rand, esp_rand};
use rs_matter_embassy::stack::persist::DummyKvBlobStore;
use rs_matter_embassy::stack::test_device::{
    TEST_BASIC_COMM_DATA, TEST_DEV_ATT, TEST_PID, TEST_VID,
};
use rs_matter_embassy::stack::wireless::{PreexistingWireless, Wifi};
use rs_matter_embassy::stack::MdnsType;
use rs_matter_embassy::wireless::esp::EspWifiController;
use rs_matter_embassy::wireless::EmbassyWirelessMatterStack;

#[esp_hal_embassy::main]
async fn main(_s: Spawner) {
    esp_println::logger::init_logger(log::LevelFilter::Info);

    info!("Starting...");

    // Heap strictly necessary only for Wifi+BLE and for the only Matter dependency which needs (~4KB) alloc - `x509`
    // However since `esp32` specifically has a disjoint heap which causes bss size troubles, it is easier
    // to allocate the statics once from heap as well
    esp_alloc::heap_allocator!(128 * 1024);

    // == Step 1: ==
    // Necessary `esp-hal` and `esp-wifi` initialization boilerplate

    let peripherals = esp_hal::init({
        let mut config = esp_hal::Config::default();
        config.cpu_clock = CpuClock::max();
        config
    });

    let timg0 = TimerGroup::new(peripherals.TIMG0);
    let rng = esp_hal::rng::Rng::new(peripherals.RNG);

    // To erase generics, `Matter` takes a rand `fn` rather than a trait or a closure,
    // so we need to initialize the global `rand` fn once
    esp_init_rand(rng);

    let init = &*make_static!(
        esp_wifi::EspWifiController<'static>,
        esp_wifi::init(timg0.timer0, rng, peripherals.RADIO_CLK).unwrap()
    );

    esp_hal_embassy::init(
        esp_hal::timer::systimer::SystemTimer::new(peripherals.SYSTIMER).alarm0,
    );

    // == Step 2: ==
    // Allocate the Matter stack.
    // For MCUs, it is best to allocate it statically, so as to avoid program stack blowups (its memory footprint is ~ 35 to 50KB).
    // It is also (currently) a mandatory requirement when the wireless stack variation is used.
    let stack = &*Box::leak(Box::new_uninit()).init_with(
        EmbassyWirelessMatterStack::<Wifi, ()>::init(
            &BasicInfoConfig {
                vid: TEST_VID,
                pid: TEST_PID,
                hw_ver: 2,
                hw_ver_str: "2",
                sw_ver: 1,
                sw_ver_str: "1",
                serial_no: "aabbccdd",
                device_name: "MyLight",
                product_name: "ACME Light",
                vendor_name: "ACME",
                sai: None,
                sii: None,
            },
            TEST_BASIC_COMM_DATA,
            &TEST_DEV_ATT,
            MdnsType::Builtin,
            epoch,
            esp_rand,
        )
    );

    // Configure and start the Wifi first
    let wifi = peripherals.WIFI;
    let (wifi_interface, controller) =
        esp_wifi::wifi::new_with_mode(&init, wifi, WifiStaDevice).unwrap();

    // == Step 3: ==
    // Our "light" on-off cluster.
    // Can be anything implementing `rs_matter::data_model::AsyncHandler`
    let on_off = cluster_on_off::OnOffCluster::new(Dataver::new_rand(stack.matter().rand()));

    // Chain our endpoint clusters with the
    // (root) Endpoint 0 system clusters in the final handler
    let handler = stack
        .root_handler()
        // Our on-off cluster, on Endpoint 1
        .chain(
            LIGHT_ENDPOINT_ID,
            cluster_on_off::ID,
            HandlerCompat(&on_off),
        )
        // Each Endpoint needs a Descriptor cluster too
        // Just use the one that `rs-matter` provides out of the box
        .chain(
            LIGHT_ENDPOINT_ID,
            descriptor::ID,
            HandlerCompat(descriptor::DescriptorCluster::new(Dataver::new_rand(
                stack.matter().rand(),
            ))),
        );

    // == Step 4: ==
    // Run the Matter stack with our handler
    // Using `pin!` is completely optional, but saves some memory due to `rustc`
    // not being very intelligent w.r.t. stack usage in async functions
    //
    // This step can be repeated in that the stack can be stopped and started multiple times, as needed.
    let store = stack.create_shared_store(DummyKvBlobStore);
    let mut resources = EnetMatterStackResources::new();
    let (wifistack, mut runner) = create_enet_stack(wifi_interface, u64::from_le_bytes({
        let mut seed = [0; size_of::<u64>()];
        stack.matter().rand()(&mut seed);
        seed
    }), &mut resources);
    // TODO Use the wifistack to create the http tcp socket.
    let mut matter = pin!(stack.run(
        // The Matter stack needs the ethernet inteface to run
        PreexistingWireless::new(
            EnetNetif::new(wifistack),
            Udp::new(wifistack, make_static!(EnetMatterUdpBuffers, EnetMatterUdpBuffers::new())),
            EspWifiController::new(controller),
            TroubleBtpGattPeripheral::new(
                ExternalController::<_, 20>::new(BleConnector::new(init, peripherals.BT)),
                stack.matter().rand(),
                make_static!(TroubleBtpGattContext<CriticalSectionRawMutex>, TroubleBtpGattContext::new()),
            ),
        ),
        // The Matter stack needs a persister to store its state
        &store,
        // Our `AsyncHandler` + `AsyncMetadata` impl
        (NODE, handler),
        // No user future to run
        core::future::pending(),
    ));

    let mut led = Output::new(peripherals.GPIO3, Level::Low);
    let mut device = pin!(async {
        info!("Starting device loop...");
        loop {
            Timer::after(Duration::from_millis(50)).await;
            led.set_level(if on_off.get() { Level::High } else { Level::Low });

            // // Simulate user toggling the light with a physical switch every 5 seconds
            // Timer::after(Duration::from_secs(5)).await;
            //
            // // Toggle
            // on_off.set(!on_off.get());
            //
            // // Let the Matter stack know that we have changed
            // // the state of our Light device
            // stack.notify_changed();
            //
            // info!("Light toggled");
        }
    });

    // Schedule the Matter run & the device loop together
    select3(
        &mut matter,
        &mut device,
        &mut pin!(async {
            runner.run().await;
            #[allow(unreachable_code)]
            Ok(())
        }),
    )
        .coalesce()
        .await
        .unwrap();
}

/// Endpoint 0 (the root endpoint) always runs
/// the hidden Matter system clusters, so we pick ID=1
const LIGHT_ENDPOINT_ID: u16 = 1;

/// The Matter Light device Node
const NODE: Node = Node {
    id: 0,
    endpoints: &[
        EmbassyEthMatterStack::<()>::root_metadata(),
        Endpoint {
            id: LIGHT_ENDPOINT_ID,
            device_types: &[DEV_TYPE_ON_OFF_LIGHT],
            clusters: &[descriptor::CLUSTER, cluster_on_off::CLUSTER],
        },
    ],
};

[ivmarkov]

const NODE: Node = Node {
    id: 0,
    endpoints: &[
        EmbassyEthMatterStack::<()>::root_metadata(),
        Endpoint {
            id: LIGHT_ENDPOINT_ID,
            device_types: &[DEV_TYPE_ON_OFF_LIGHT],
            clusters: &[descriptor::CLUSTER, cluster_on_off::CLUSTER],
        },
    ],
};

^^^ Not OK. Replace EmbassyEthMatterStack with EmbassyWifiMatterStack.
There might be other remnants of the original Eth code that you have started with originally, that I'm not spotting right now.

[ivmarkov]

It is still weird though because even with that wrong setup, you should still at least see some "Gatt" info logs, and also BTP (BLE) traffic. Did you censor this out, or is it that you really don't see any BTP traffic?