iSerialNumber not populated on USB Serial port in Windows

[heatsink99]

I have several Pi Pico W running code that was complied about 2 years ago that can connect to a node.js app on Windows via either Serial or websocket. The node app needs to be able to identify the Pico uniquely so that it knows which device is connected regardless of whether its connected via serial or websocket. Sometimes the Pico might be connected by both methods at the same time and the Node application needs to know its talking to the same device. The application does this using the Pico unique serial number from flash.
The Pico Serial number was available when enumerating serial ports in Node.js (SerialPort.list()).

function detectSerialTimer() {
    SerialPort.list().then(
        ports => {
            const vendors = {"2E8A": "Pi Pico", "2341": "Uno"}
            ports.forEach((port) => {
                if (Object.keys(vendors).includes(port.vendorId)) {
                    // Found serial device of correct type.  Check if we have seen it before
                    let trackId = getTrackIdFromSerialNo(port.serialNumber)
                    if (!trackId) trackId = createNewTrack(port.serialNumber) // track not known
...

and I passed the serial number in extra headers with SocketIO:

  // Connect to SocketIO server
  pico_get_unique_board_id_string(id, PICO_ID_LENGTH);   
  sprintf(msg, "client_type: track\r\nserial_no: %s",id);
  socketIO.setExtraHeaders(msg);
  socketIO.begin(settings.serverIp, 80, "/socket.io/?EIO=4"); // server address, port and URL

This has worked well for 2 years. However I need to make some updates to the Pico code and have struck a problem I have been unable to resolve after hours of research. When I compile the same original code now (using current arduno-pico library versions), the USB serial port is presented differently in Windows. Previously the Pico serial port appeared as a single "USB Serial Device". This device has the Serial Number property set so it can be accessed from the javascript SerialPort module as shown above.

Now the serial port shows up as a pair of usb devices: a parent "USB Composite Device" and the associated "USB Serial Device". The Pico serial number appears on the USB Composite Device, but not the USB Serial Device.

This results in Windows providing the USB Container ID as the Serial Number to the Node SerialPorts library, rather than the actual serial number. The value "7&2391C344&7&0000" in the JSON below used to be the Pico Serial number, now is is the Container ID.
{"path":"COM13","manufacturer":"Microsoft","serialNumber":"7&2391C344&7&0000","pnpId":"USB\\VID_2E8A&PID_F00A&MI_00\\7&2391C344&7&0000","locationId":"0000.0014.0000.010.004.000.000.000.000","friendlyName":"USB Serial Device (COM13)","vendorId":"2E8A","productId":"F00A"}
Is apparently possible to walk the USB tree from the serial port to the container device to get the serial number, but this is quite involved and I don't want to go there.

I'm guessing this is due to changes to the USB stack in arduino-pico, but scanning the release notes I couldn't see any reference to a major change that would trigger this.
Is it possible to make the Serial port appear as a simple single USB Serial Device like it used to (with the serial number)? Does anyone know what caused this change in behaviour? Or is there a way to have the serial number appear on the USB Serial Device when its a child of a USB Container Device?
I know a work around would be to have the node.js app open the serial port and query the Pico to gets its serial number, but I want to avoid that because its constantly scanning for new devices. The way the code was working, the node app could recognise devices without having to open the port.

[earlephilhower]

I'm not at my computer right now, but from memory this new configuration came into play about a year or so ago when there was a complete rewrite of the USB stack management.

I'd just added MIDI over USB and the special casing (we're talking the most convoluted series of if 5 level deep ifs that needed to be replicated in several spots) was brutal. Basically every possible permutation of 5 or 6 different USB devices we support needed to be coded around. A nightmare to keep correct-ish.

Check the release notes for the USB stack rewrite, it's definitely the prominent notice because it added things like disabling the serial port and coming up as just a keyboard, and then swapping back to a pure serial device at runtime.

I'm any case, the old unmaintainable hardcoded logic had a special case for only serial (because that's how the core started, only CDC Serial support). Otherwise there was a composite definition (requiring another different copy of the serial description). Now, it's always a composite device with the components added under program control.

The USB serial number, as you found, is a property only of the main descriptor, not the compound components. That just the way USB is defined.

I have a feeling it's possible to query Windows for the parent of the com device in this case, and from there grab the serial. I've done a lot of work on the storage stack in Win32, but not USB, though, so sadly can't suggest a specific API. PowerShell is probably your friend here.

It's rather non trivial to go back to a single device description when only one USB object registers because of the binary format of the descriptor. You might be able to single case something special but that way madness lies.

You might also try Adafruit TinyUsb, but I personally don't know if they do special cases either (since they've always allowed runtime creation of devices).

[heatsink99]

Thanks for your quick reply.
The Adafruit TinyUsb stack seems to behave the same.
Both Claude and CoPilot seem convinced it is possible to get the serial number to appear on the USB Serial Device by overriding the tud_descriptor_device_cb and tud_descriptor_string_cb callbacks.
However as far as I can see, in your code (USB.cpp) you are already setting the serial number in tud_descriptor_device_cb as per their examples.
I found a way to find the parent USB container and serial number using powershell. Here it is in case its useful to someone else:
((Get-PnpDeviceProperty -InstanceId (Get-CimInstance Win32_PnPEntity | Where-Object { $_.DeviceId -like '*<container id from port.serialNumber>' }).DeviceId -KeyName "DEVPKEY_Device_Parent").Data -split '\\')[-1]

However given the overhead of doing this I've decided I might as well change how I identify devices and just query the Pico for its ID.

By the way, many thanks Earle (and all contributors) for all your work on arduino-pico core. My application is a scheduling and timer system for Scouts Pinewood Derby model car races. Over the years arduino-pico has indirectly facilitated thousands of youth in New Zealand learn some wood working skills, think about the physics of what makes a model car go fast, and enjoy the racing thanks to the race timer Pi Picos your core has helped me build.

[earlephilhower]

Thanks, that's a really cool use case! I'm always amazed at the neat stuff folks use the cores for. TBH, I only just deployed my first Pico (after 5+ years of writing the core!) to put my pool controller on the network after my full-fat R.Pi board ate it's 4th SD card.

...Both Claude and CoPilot seem convinced...

Thx for the PowerShell magic. Hopefully it's useful to others.

The USB descriptor isn't a self-describing format like ASN.1 or protobuffers, and the descriptor format has a single predefined serial string reference ID (hence the incorrect suggestion to override the string CB) right in the header of the predefined struct. Your code can set that serial to whatever you want, but it's gonna be a property of the USB composite device, not the CDC interface.

Depending on the number of boards you're talking about and any physical setup, you could do something as simple as adding a Keyboard device and have it type the serial # when you hit BOOTSEL, for example (i.e. like a bar code scanner).

It's also possible to make your own custom USB devices, but under Windows actually accessing them might be a pain. Under Linux it's not a big teal w/libusb or even shell scripts to read stuff from the devices w/o a custom device driver.

Best of luck!