Coordinator backup

[Shulyaka]

I'd like to start a discussion on how to back up the network key on XBee coordinators.
The issue is that on XBee devices the network key is write-only, so we have to remember it from the moment we set it. Luckily, we have a backup mechanism that we can use. Currently, however, we only use the last backup to restore the state if the device is not already configured, and optionally for verification.
Current flow:

1. For a new network, we generate a new random key and write it to the device
2. The key is kept in memory until the next restart, and will be saved in the backup once it is created.
3. After the restart, we set the key to the default value (unknown), and because we can't get the key from the device, it remains unknown.

Suggestions:

1. Enhance zigpy ControllerApplication.initialize() function to use last_backup to set the initial network_info (instead of default) before trying to load it from the device. Something like:

     async def initialize(self, *, auto_form: bool = False) -> None:
         """Starts the network on a connected radio, optionally forming one with random
         settings if necessary.
         """

         last_backup = self.backups.most_recent_backup()

+        if last_backup:
+            self.state.network_info = zigpy.state.NetworkInfo.from_dict(last_backup.network_info.as_dict())
+
         try:
             await self.load_network_info(load_devices=False)
         except zigpy.exceptions.NetworkNotFormed:

So the radio library would only update what it can, and the rest will remain from the backup.

2. Pass last_backup as a new optional parameter to ControllerApplication.load_network_info and let the radio library (zigpy-xbee`) handle it as it wishes.

         try:
-            await self.load_network_info(load_devices=False)
+            await self.load_network_info(load_devices=False, last_backup=last_backup)
         except zigpy.exceptions.NetworkNotFormed:

3. To avoid changes to zigpy, overload ControllerApplication.initialize() function in zigpy_xbee.zigbee.ControllerApplication, load last backup inside it and restore the network key in memory there:

    async def initialize(self, *, auto_form: bool = False) -> None:
        """Overloaded initialize() to restore unreadable info from backup."""

        last_backup = self.backups.most_recent_backup()

        if last_backup:
            self.state.network_info = zigpy.state.NetworkInfo.from_dict(last_backup.network_info.as_dict())

        await super().initialize(auto_form=auto_form)

4. Do not try to read the key from backup, but perform implicit key rotation when making a new backup and the network key is not known. We can simply generate a new network key and write it to the device (and the backup), and it will be distributed to all devices in the network. Might be a bit tricky to implement it to do it only when the backup is created, and I also don't really like the idea of doing implicit actions like rotating the key when a user might not expect it.

5. There is also a native backup functionality in newer firmwares (the 'BK' AT command), it requires an additional investigation and is not available for legacy modules.

[puddly]

If the key is unknown, I think it should be treated as unknown. Implicit key rotation or anything that can permanently affect a production network isn't something I think we should do.

My suggestions:

1. Implement BK for newer coordinators.
2. Fix zigpy's form_network (or restore_backup's creation of a new backup) so that it merges the newly-created network backup with the the in-memory generated initial network settings, filling in the unknown network key's value from that. The fact that zigpy doesn't save the initially-created backup is a bug, especially for platforms like XBee or ZiGate.

Restoration for the XBee isn't possible either, since it is not possible to use a specific PAN ID. Is this implemented with newer firmwares?

[Shulyaka]

It is possible to restore the PAN ID using ID command. See

zigpy-xbee/zigpy_xbee/zigbee/application.py

Line 149 in 2f2e1ae

await self._api._queued_at("ID", epid)

What wasn't possible before is restoring IEEE. It is possible on newer firmwares, but is tricky (requires you to set an encryption key and use the encrypted backup file that is stored in internal flash and can be accessed using YMODEM protocol).

[Shulyaka]

Oh, wait, I confused PAN ID with Extended PAN ID!

[trunet]

@Shulyaka have you tested using Centralized trust center backup mentioned on https://www.digi.com/resources/documentation/digidocs/pdfs/90001539.pdf ?

[Shulyaka]

Yes, I have, but there are two caveats:

1. The backup is only possible in command mode, and so no communication with network is possible during that process (i.e. every time network info is requested from the library)
2. The backup file is encrypted. Even though I know the encryption key (I set it myself with KB command) and the algorithm (256-bit AES-CTR according to the specs), I was not able to decrypt it. Which means the backup will only be applicable to an another XBee device and not interchangeable with other stacks. Digi declined my request to provide the specification of the backup file.

Here is a file from my test network if you want to try it: backup_TC41A06E60.zip. The encryption key is 5A69 6742 6565 416C 6C69 616E 6365 3039 5A69 6742 6565 416C 6C69 616E 6365 3039 (ZigBeeAlliance09ZigBeeAlliance09)

[puddly]

@Shulyaka Could you change the encryption key to all null bytes and post another backup? I don't have an XBee 3 to test with, unfortunately, and it doesn't look like the S2C supports trust center backups.

The protocol to read and write the encryption key (in addition to the actual backup data) seems to be done through XCTU, which might use undocumented serial commands and transform the key before actually writing it to the device.

[Shulyaka]

Sure, will test it

[trunet]

Probably you already know that, I'll say it anyway if not.

A good reverse engineering technique is to change known small changes (like change the EE from 2 to 1 and stuff like that) and compare. Probably also, changing the KY trust center encryption key and changing nothing else, will give us a perspective on where the key is located and what's located where.

If I understood correctly the digi forum discussion, the file doesn't fit on AES block size, probably meaning a decryption will need to happen in portions of the backup file. Some things in that file are not related to its actual backup content, but some kind of metadata/headers/nonce/...

My xbee series 3 is being used at the moment, and I only have one usb serial connected to it, so I can't use it to test. If you could upload a couple of these backups, including its changes, I can try to help.

[trunet]

From your file, looks like the nonce first 4 bytes are the counter, and the 16 bytes left are the IV. The Version I guess is your firmware version, 1012. So the rest should be the encrypted file and we could try to extract its bytes and actually decrypt because the IV is known now.

IV: 1E4C92ED83E51CA5B75664C5D1DD1A15

[puddly]

I think the first four bytes of each <Version>|<Nonce><end> pair are the size of the contents of each.

<Version>[uint32 size][2 byte fw ver]<end><Nonce>[uint32 size][16 byte nonce]</end>[trailing data]

I haven't had much luck bruteforcing the format, unfortunately. The nonce being 128 bits makes it seem like it's the counter's initial value but no combination of endianness and no offset within the file seems to produce anything useful:

from pathlib import Path

# pip install pycryptodome
from Crypto.Cipher import AES
from Crypto.Util import Counter


KEY = b'ZigBeeAlliance09ZigBeeAlliance09'

with pathlib.Path('~/Downloads/backup_TC41A06E60.xbee').expanduser().open('rb') as f:
    # Version
    assert f.read(9) == b'<Version>'
    tag_size = int.from_bytes(f.read(4), 'little')
    version = f.read(tag_size)
    assert f.read(5) == b'<end>'
    print('Version', version)

    # Nonce
    assert f.read(7) == b'<Nonce>'
    tag_size = int.from_bytes(f.read(4), 'little')
    nonce = f.read(tag_size)
    assert f.read(5) == b'<end>'
    print('Nonce', nonce)

    rest = f.read()


for temp_key in (
    KEY,
    KEY[::-1],
    b'\x00' * 32,
):
    for temp_nonce in (
        int.from_bytes(nonce, "little"),
        int.from_bytes(nonce, "big"),
        0,  # Just in case :)
    ):
        for start_offset in range(len(rest)):
            for little_endian in (True, False):
                ciphertext = rest[start_offset:]

                ctr = Counter.new(128, initial_value=temp_nonce, little_endian=little_endian)
                aes = AES.new(temp_key, AES.MODE_CTR, counter=ctr)

                plaintext = aes.decrypt(ciphertext)

                if bytes.fromhex('41A06E60') in plaintext or bytes.fromhex('41A06E60')[::-1] in plaintext:
                    print(plaintext)

[trunet]

I just noticed your code while was working on something similar:

import binascii

from Crypto.Cipher import AES
from Crypto.Util import Counter

with open('backup_TC41A06E60.xbee', 'rb') as f:
    iv = f.read(52)[31:-5] # all between <nonce><end>
    ciphertext = f.read()

enckey = b'ZigBeeAlliance09ZigBeeAlliance09'

print(f'KEY: {enckey.hex(" ")} / Size: {len(enckey)}')
print(f'IV: {iv.hex(" ")} / Size: {len(iv)}')
#print(f'CIPHERTEXT: {ciphertext.hex(" ")} / Size: {len(ciphertext)}')

iv_int = int(binascii.hexlify(iv), 16)

ctr = Counter.new(AES.block_size * 8, initial_value=iv_int)
cipher = AES.new(enckey, AES.MODE_CTR, counter=ctr)

print(f'\nDECRYPTED: {cipher.decrypt(ciphertext).hex(" ")}')

We need to know the AT parameters values so we can compare with the output and try to come with something. And some other backup files with one or other parameter changed.

[Shulyaka]

Here are two backup files from the same device with the backup key set to all zeroes (b'\x00' * 32):
backup_TC41A06E60.xbee.0.zip
backup_TC41A06E60.xbee.1.zip

[puddly]

Could you also include the network key? Just so that we have some concrete bytestring to search for to test if decryption was successful.

[puddly]

I played around with alternate block cipher modes, nonstandard constructions, and various CTR schemes. Unfortunately, I haven't had any luck decrypting it. I think there may be a key derivation function involved somewhere.

It would be useful to MITM the serial traffic to see what exactly is being sent to the coordinator during backup/restore, as the KDF could be happening within the application itself and not on the MCU.

[Shulyaka]

I am thinking if it would be possible to disassemble the firmware...