USB Custom Descriptors

[thestumbler]

I have a project with 20 or so RP2350 modules in a single system. The USB interface isn't used in the application, but for development and testing. It is becoming a pain to sort out which com port is which module.

I found a helpful discussion here by user Hippy. This substantially works, and I understand it well. At the end of that discussion it is noted that version 1.23 now includes dynamic USB drivers. Hippy points out: "Crafting initial versions of those is somewhat more complicated than copying what the MicroPython defaults are." I found some examples of dynamic USB code, but they are for changing more advanced aspects of the interface. I can't find any examples, complicated or not, of how to just change the USB descriptors (and serial number).

With that background, here is what I'd like to accomplish. By assigning a custom product descriptor, I can easily grab the desired com port from scripts on the PC. For example, "proto_meter_00, _01, _02, _03", etc. Here's an example I built last night:

Found port  /dev/cu.usbmodem101
   vid/pid       2e8a:0005
   description   proto_meter04
   manufacturer  TheStumbler
   product       proto_meter04
   serial        d589db7044d42859

Taking this approach, however, means that I would have to build a separate UF2 firmware file for each module. A smoother approach would be to make the product descriptor constant, say "proto_meter", and tweak the serial number by appending the meter number to the existing serial number (note that the meter number is read from DIP switches on each module). Ideally I'd like to be able to generate USB descriptors like below and only have a single UF2 firmware build (manually tweaked example of the above):

Found port  /dev/cu.usbmodem101
   vid/pid       2e8a:0005
   description   proto_meter
   manufacturer  TheStumbler
   product       proto_meter
   serial        d589db7044d42859-04

Questions:

(1) [NEW WAY] Are there any examples of changing USB descriptors using the dynamic USB device feature of the latest 1.23 release?

(2) [OLD WAY]

(2a) Applying the approach suggested by Mr Hippy raises a few build issues outside my previous MicroPython experiences. Other than replacing the CMakeLists.txt file in the rp2 directory, what's the usual process for dealing with custom cmake files? I tried to put CMakeLists.txt in my custom build folder, no joy. Then I extracted just the necessary bits into the .cmake file in the custom board folder, but that didn't work either.

(2b) Reading the documentation for the rp2 port closer, it states that the main way to build things is to use cmake, and the make commands are merely a convenience. But I couldn't figure out how to do a cmake-only build, and there are no examples of such that I could find. Perhaps if I knew that, I could have solved (2a) on my own.

(2c) Unlike Hippy, I put my changed USB files in my custom board folder rather than the main rp2 directory. But otherwise I still had to modify the main CMakeLists.txt file in the rp2 folder. But when I looked into changing the serial number, it seems that I have to tweak mp_usbd_port_get_serial_number() in file usbd.c. Is there an accepted way to localize all these changes to only the custom board folder? I'd like to not touch the main MicroPython code at all, and limit the changes to the custom board folder. But I don't know how to do that.

[mattytrentini]

You might want to watch the demo Damien presented on using USBDevice. You're right that the 'new way' allows you to modify the descriptors dynamically, though you'll probably find it much better to use an FTDI cable to connect to the REPL (otherwise, if your REPL is accessed over the same USB, the connection will drop when you update the descriptors).

If you're not already aware, there are some good examples in micropython-lib/usb.

There should be no need to build firmware to modify descriptors.

[thestumbler]

I'm getting much closer. I figured out how to make the strings, but updating the USB descriptor (as opposed to creating a whole new one) is eluding me so far. But I only want to update strings for the default CDC descriptor. There doesn't seem to be a method to only update the string dictionary (nor even examine the current dictionary). But it's after midnight and maybe things will be clearer in the morning.

Here's my start so far:

import machine
import sys
sys.path.extend(['/lib'])

# from shared/tinyusb/tusb_config.h
#    USBD_STR_MANUF (0x01)
#    USBD_STR_PRODUCT (0x02)
#    USBD_STR_SERIAL (0x03)
# from shared/tinyusb/mp_usbd_descriptor.c
#   .iManufacturer = USBD_STR_MANUF,
#   .iProduct = USBD_STR_PRODUCT,
#   .iSerialNumber = USBD_STR_SERIAL,

# string indices in the default descriptor
i_str_manuf = 0x01
i_str_product = 0x02
i_str_serial = 0x03

# module ID (will be read from DIP switches)
module_id = 0x03

# build some strings
str_module_id = f'{module_id:02x}'
str_unique_id = f'{int.from_bytes(machine.unique_id()):x}'
str_serial = f'{str_unique_id}_{str_module_id}'
str_product = f'proto_meter_{module_id:02x}'
str_manuf = 'TheStumbler'

# make the strings dictionary
mystrings = {}
mystrings[0] = None # need this for language
mystrings[i_str_manuf] = str_manuf
mystrings[i_str_product] = str_product
mystrings[i_str_serial] = str_serial

# for s in mystrings.items(): print(s)

desc_dev = machine.USBDevice.BUILTIN_CDC.desc_dev
desc_cfg = machine.USBDevice.BUILTIN_CDC.desc_cfg

# this isn't working, manually or in boot.py
# machine.USBDevice.active(False)
# machine.USBDevice.config(
#  desc_dev, desc_cfg, \
#  desc_strs=mystrings \
# )
# machine.USBDevice.active(True)

When it fails, I get this message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: argument has wrong type

[thestumbler]

BTW, even though I h ent succeeded yet, I completely prefer this new dynamic method over rebuilding the firmware for each different module ID in my system. I did that last night for just four meters, so I could have some working firmware to begin integration testing. That was a pain in the neck!

I also need to finish watching the presentation you linked. I got about 15 minutes in and said to myself, "Ah ha. I got this" and started trying to implement it. Maybe I missed some key points later on in the video.

[thestumbler]

@mattytrentini so, I stumbled around over the weekend and arriving at a workable but not quite correct solution. The presentation you linked above was indeed helpful. But last night, I finally realized you were pointing me in the direction of the higher-level usb drivers solution found in the micropython-lib, so I had a look and tried that approach.

I can appreciate how this driver makes the task of creating a new USB device much easier. But in the case of just changing the descriptors of the main CDC serial port REPL, I'm not sure it has any advantages.

The CDC example works as promised in the notes. I even found the arguments one passes for customizing the strings. When you run this example, you end up with two identical CDC serial ports connected to the REPL. Good.

Now let's say I want the UART REPL for testing, as advised. I could not figure out how to have three REPLs running. No matter what I tried, there seemed to be a hard-baked limit of two REPLs. The index parameter to os.dupterm() would only accept 0, weird since if there are two REPL "slots", I'd think the index could be 0 and 1.

But on the USB front, I don't want two CDC serial ports. My goal is simply to change the descriptor strings. I could not figure out how to disable the builtin CDC driver once I created the second one. If I set machine.USBDevice.active(False), that killed both CDC ports. Creating the new CDC with builtin_driver = False didn't do what I expected, there were still two CDC ports.

Strangely, the two ports have identical descriptor strings. I would expect the original port to remain as "Board in FS mode", and the new port to have the custom strings. Note -- the two serial port name on my PC are distinguishable because underscores are appended, like _1 and _3.

So, I could create custom descriptor strings using this method. But no matter what I did, it always resulted in two ports with identical descriptor strings. Furthermore, using the usb drivers did not solve the mpremote or rshell soft reset problem. In other words, I still had to include the resume option in all my mpremote commands.

So while this library makes usb device drivers easier, in the very simple case of just changing basic CDC device descriptors, I think it causes more confusion than the direct USBDevice() approach, because it creates two ports instead of one, and no way to also have a UART REPL for debugging.

[scruss]

It looks like you're using MacOS, so I don't know how easily this is done there. But on Linux, the board with machine.unique_id() == 0ecb4f7dccb7027b appears as serial port /dev/serial/by-id/usb-MicroPython_Board_in_FS_mode_0ecb4f7dccb7027b-if00

[thestumbler]

Yeah, for some reason their BSD style doesn't implement that technique. But even if it did, I'd still have to manually sort out the unique ID for each of 20-some modules and map them in an initialization script. I already do this, because even without that /dev folder, I can see the serial number using various utilities like lsusb.

By making my own custom string descriptors, I can completely automate the init script that scans the usb serial ports and initializes some variables for ease of use.

This init script assigns a shell variable to all 20-some modules in an assembly, and then I have a bunch of aliases to the mpremote and rshell tools. End result, I can simply type mpr0 reset or rs18, etc., to work with any desired meter.

[thestumbler]

This seems to have fixed it.
Key changes:
(1) instance vs type
WAS: machine.USBDevice
IS: machine.USBDevice()
(2) adding usbdev.builtin_driver = usbdev.BUILTIN_CDC
(3) REPL on UART for testing, not needed for final
Thanks to folks on the Discord channel for help.

import os
import machine
import sys
sys.path.extend(['/lib'])

# this is useful for testing but not necessary
# once everything is working
#uart = machine.UART(0,115200,tx=16,rx=17)
#os.dupterm(uart)

# from shared/tinyusb/tusb_config.h
#    USBD_STR_MANUF (0x01)
#    USBD_STR_PRODUCT (0x02)
#    USBD_STR_SERIAL (0x03)
# from shared/tinyusb/mp_usbd_descriptor.c
#   .iManufacturer = USBD_STR_MANUF,
#   .iProduct = USBD_STR_PRODUCT,
#   .iSerialNumber = USBD_STR_SERIAL,

# string indices in the default descriptor
i_str_manuf = 0x01
i_str_product = 0x02
i_str_serial = 0x03

# module ID (will be read from DIP switches)
module_id = 0x11

# make strings
str_module_id = f'{module_id:02x}'
str_unique_id = f'{int.from_bytes(machine.unique_id()):x}'
str_serial = f'{str_unique_id}_{str_module_id}'
str_product = f'proto_meter_{module_id:02x}'
str_manuf = 'TheStumbler'

mystrings = {}
mystrings[0] = None # need this for language
mystrings[i_str_manuf] = str_manuf
mystrings[i_str_product] = str_product
mystrings[i_str_serial] = str_serial

for s in mystrings.items(): print(s)

usbdev = machine.USBDevice()
desc_dev = usbdev.BUILTIN_CDC.desc_dev
desc_cfg = usbdev.BUILTIN_CDC.desc_cfg
usbdev.active(False)
usbdev.builtin_driver = usbdev.BUILTIN_CDC
usbdev.config( 
    desc_dev, desc_cfg, \
    desc_strs=mystrings \
)
usbdev.active(True)
usbdev.active(True)

[scruss]

your code, when run as boot.py on a Pico 2W, seems to drop back into the Raw REPL after reset:

$ mpremote a0
Connected to MicroPython at /dev/ttyACM0
Use Ctrl-] or Ctrl-x to exit this shell
(0, None)
(1, 'TheStumbler')
(2, 'proto_meter_11')
(3, 'ecb4f7dccb7027b_11')
raw REPL; CTRL-B to exit
>
MicroPython v1.25.0-preview.365.g3823aeb0f on 2025-03-07; Raspberry Pi Pico 2 W with RP2350
Type "help()" for more information.
>>> 

mpremote does report the board with your custom strings for a while, though:

/dev/ttyACM0 ecb4f7dccb7027b_11 2e8a:0005 TheStumbler proto_meter_11

Your code calls usbdev.active(True) twice: should it? Also, printing to a USB console that's very shortly about to disappear might have unexpected consequences. sys.path already contains /lib, too

[thestumbler]

That's consistent with my observation, too. I have since removed the prints, they were just for debugging and when the code was being run manually or in main.py

I also removed the double call to active(). It didn't seem to make a difference. But when running the code manually, the first call would return False and the second call True. That's why I originally had it in there twice.

So, my temporary approach is something I discovered yesterday. The problem stems from the soft reset that MPREMOTE issues by default at each command. But it has a resume option that prevents the soft reset. That seems to work for me, although I hold out hope that later on I will learn how to do this task (assign custom USB descriptor strings) correctly.

There's a good explanation of this in the MicroPython USBDevice documentation page, explaining how a soft reset will cause the factory default USB descriptor to reemerge. It suggests that putting the code in boot.py will fix that, maybe -- or maybe I'm misreading the text on that point.

For the moment I have settled on putting my code in main.py, on advice from a user in the discord and because in my PCB circuit design, the DIP switches that hold the module's ID value (0 through 20-some modules) are behind an I2C I/O port expander. It's not as simple as reading some GPIO pins. Advice to use main.py vs documentation saying use boot.py aside, I don't think it's a good idea to be instantiating an I2C interface and the associated TCA9554 driver class inside boot.py. If I later learn this is really where it belongs, I can write some simple and dumb bit banging inline code to read the DIP switches.

On more reflection yesterday, I could do this by taking an intermediate course between the hard-baking method and the completely dynamic approach of dynamic USB descriptors. That is, if I could replace the pico's get unique ID function with one of my own. And if my module ID was simply on GPIO pins, I would have tried this already. But as noted, I'd have to fire up I2C or bit-bang it for a one-time read operation. But I have decided to put that idea on the shelf for now.

I have a workable solution at present, if a little shaky. I already have a big script that lets me talk to each module individually, like mpr0 ls so it was trivial to just add the resume option. I'll focus on my application now, and circle back to this issue later on.

[projectgus]

I also removed the double call to active(). It didn't seem to make a difference. But when running the code manually, the first call would return False and the second call True. That's why I originally had it in there twice.

Ah, this is because active() returns the previous active state (i.e. before applying the argument). Will document this better in the API.

There's a good explanation of this in the MicroPython USBDevice documentation page, explaining how a soft reset will cause the factory default USB descriptor to reemerge. It suggests that putting the code in boot.py will fix that, maybe -- or maybe I'm misreading the text on that point.

I think the USBDevice documentation could be clearer on this as well, but when MicroPython resets in "raw REPL mode" (which is the mode mpremote uses) then it executes boot.py but it doesn't execute main.py - as the assumption is that mpremote is about to send some commands so there's no need to run main.py. This is most likely the sequence of events that leads to the "default" USB CDC config showing up unexpectedly.

the DIP switches that hold the module's ID value (0 through 20-some modules) are behind an I2C I/O port expander. [...] Advice to use main.py vs documentation saying use boot.py aside, I don't think it's a good idea to be instantiating an I2C interface and the associated TCA9554 driver class inside boot.py

Ah yeah, this is a bit awkward. I guess it's a case of which trade-off you prefer. If you can minimise the I2C init code (and maybe place it in an exception handler block so failure will still continue execution) then I'd probably take the approach of putting that into boot.py, but it's definitely a trade-off.

But it has a resume option that prevents the soft reset. That seems to work for me, although I hold out hope that later on I will learn how to do this task (assign custom USB descriptor strings) correctly.

At this point, I think the code you posted up-thread (placed in either main.py or boot.py as discussed) and the resume option are the best and most "correct" approach, even though custom descriptors weren't really a use-case that was on the radar when designing the . The alternative, as you flagged, is to patch MicroPython to generate this string itself and pass those values to TinyUSB via C code. I guess you will encounter the same frustration about needing to initialise I2C before you can know the ID, though.

[thestumbler]

Thanks for taking the time to look at my problem @projectgus For the moment it's comforting to know that I've stumbled on the most correct solution.

I would like to suggest that consideration be given to the use case of custom descriptor strings only. It seems that this is something lots of people struggle with when connecting to USB serial ports of all sorts of embedded devices, not just MicroPython. My case with a system of 20-some modules is an extreme example, but being able to clearly identify one's project by customizing USB descriptors is a common enough request from what I've seen, and provides real benefits to the users.

I am not well versed in the bowels of the USB specification and its implementation here, but based on my limited experimentation the last few days, it sure feels like implementing this feature wouldn't be a difficult thing.

Whether or not dynamic descriptor strings can be implemented in an easier way, I think there is also some benefit to making it easier to change the default strings upon building the firmware, just by adding entries to the mpconfigboard file. Currently it's a dance (though well explained by user Hippy mentioned way at the top of this discussion) requiring tweaking the main CMake file and copy / editing a couple of tinyusb modules.

[thestumbler]

Oh -- one more observation. When I used the direct USBDevice() dynamic method, or "old style" building custom MicroPython firmware per module serial number, the resulting serial ports on my system were,let's say, "normal".

/dev/cu.usbmodem1101

If I make the port, well, pair of ports, using the mip-installed usb library, the usb serial port shows up as

/dev/cu.usbmodemmodule_141
/dev/cu.usbmodemmodule_143

I note that the text "module_14" is a place holder I put in the descriptor string for serial number. But on the other custom ports above, the serial number, custom or not, doesn't get appended to the PC name for the /dev serial port. I don't really care one way or another, I just thought it was interesting that they behaved differently