fleetsie

Simple shell tools to provision and manage a fleet of linux-based iot devices.

WTAF?? (why think about fleetsie)

• headless provisioning of headless devices by untrained personnel
• OS image has no secrets - distribute / burn freely
• fleet admin doesn't handle SD cards, USB drives, or device hardware
• admin decides who gets to provision how many devices
• admin distributes batches of fleet permits to provisioners
• each permit allows one device to join fleet
• unused permits can be revoked
• provisioner puts SD card (with OS image) and USB disk (with batch of fleet permits) into device and powers up
• when provisioning completes, SD card stays in device and USB disk is removed and used to provision more devices
• if necessary, a provisioning technician can edit plain-text WiFi credentials file on USB disk, before supplying it to untrained provisioners

1. Admin uses fleetsie_mod to convert a standard OS disk image file to a modified OS disk image file. So far, only Raspberry Pi OS images are supported.
2. Admin uses the fleetsie_gen script to create the .zip archive for USB provisioning disks, and allocate devices on the server.
3. Technician writes modified OS image to SD cards, e.g. with rpi-imager
4. Technician installs these SD cards into the devices.
5. Technician attaches USB provisioning disk to device, powers up device, then waits until LED flashing pattern shows provisioning is complete.
6. During provisioning, each device connects to the fleet server via ssh (with fleet credentials and a one-time password supplied on the USB disk) and runs fleetsie_auth to obtain device credentials. Upon success, the device will appear on the server dashboard.
7. Technician plugs headphones into the audio jack on the device to hear a voice repeating the device ID. Technician attaches a physical label to the device and writes the ID on it. The USB disk can now be removed and used to provision another device.

Parts

fleetsie_mod: modifies a stock OS image for use with fleetsie

fleetsie_provision: script which provisions the system by using the code, configuration and credentials supplied on an attached USB disk. The script is intended to be run once, at provisioning time.

fleetsie-provision.service: systemd service that runs fleetsie_provision when the system is stable. The meaning of stable might depend on the OS. Disables itself after a successful run of fleetsie_provision.

fleetsie_auth: server-side script that provides credentials to a device that logs in with a one-time password during provisioning

fleetsie_srv: set up a server for use with fleetsie

fleetsie_gen: generate a USB provisioning disk and the server-side inventory for a fleet

fleetsie_gen_server: script run on the fleet server by fleetsie_gen

fleetsie_login: script run on the server when a provisioned fleet device connects via ssh. Maintains a hierarchy of connected devices in /run/fleetsie/dev/FLEET/ on the server, and sends a connect/disconnect event to the zabbix server, so that the device's connection status is visible on the fleet dashboard.

fleetsie_mod - Manage Creation of a Modified OS Image

Requirements

The host where fleetsie_mod is used to modify an OS image requires these packages:

sudo apt install xz-utils kpartx awk unionfs-fuse

Commands:

  fleetsie_mod init PATH/OSNAME.img.xz

• decompresses the image using xz and writes it to file OSNAME.img in the current directory
• mounts all partitions in the image, as is done by the fleetsie_mod mount command below
• creates a symlink from .image -> OSNAME.img
• initializes a git repo in my_work, where overlays for each of the partitions in OSNAME.img will be created, so that you can use version control to track your work.

  fleetsie_mod mount

• mounts each partition in OSNAME.img read-only to directory original/part_N
• creates an overlay mount for each partition; the merged mount is in part_N, where N is 1, 2, ...
• changes to the underlying image will be reflected in directories .new_N; these are created if they do not already exist.
• symlinks are created from the underlying partition labels to part_N; e.g. bootfs -> part_1

  fleetsie_mod unmount

• unmounts the merged and original filesystems
• changes made remain available in the .new_N directories and will be automatically restored the next time fleetsie_mount OSNAME.img is run

  fleetsie_mod install [OSTYPE]

• uses instructions customized for OSTYPE to install fleetsie in the overlain OS image.\
• OSTYPE defaults to Raspberry Pi OS; do fleetsie_mod install --help to list others.
• after this command, you can make any further customizations to the image by manipulating files in the my_work/part_N directories
• in particular, these files are treated specially:
  • my_work/part_2/opt/fleetsie/custom_pre is a folder which gets copied onto the SD card so that the running system will see it in /opt/fleetsie/custom_pre The fleetsie_provision script uses it early in the provisioning process, before even trying to find the USB provisioning disk.
  • my_work/part_2/opt/fleetsie/custom_pre/setup - if this exists, it is a script which will be run early by the fleetsie_provision script
  • my_work/part_2/opt/fleetsie/custom_pre/packages - if this folder exists, any .deb packages it holds will be installed early by the fleetsie_provision script (after setup is run, but before overlay is extracted). This allows installing extra packages onto the SD card before network connectivity is established.
  • my_work/part_2/opt/fleetsie/custom_pre/overlay.tar.xz - if this exists, it is an archive which will be extracted to "/" as user root early by the fleetsie_provision script, right after running the setup script mentioned previously. This is another way of modifying the SD card before network connectivity; it is meant for files and changes which you haven't had time (or don't intend) to package.
  • my_work/part_2/opt/fleetsie/custom_pre/cleanup - if this exists, it is a script which will be run early by the fleetsie_provision script, right after extracting the overlay image just mentioned
  • my_work/part_2/opt/fleetsie/custom_pre/pronunciation.txt - if this exists, it is used to help the text-to-speech system pronounce words or phrases associated with your project, by providing phonetically-spelled replacements. The file is treated as consecutive pairs of lines, with the first line in the pair giving a from expression, and the second line, a to expression. These pairs will be applied by bash in order when modifying a string s before speaking it. Specifically, each pair causes bash to do s="${s//$from/$to}". For example, the two lines:

       sqlite
	   ess-cue-light

cause the TTS system to pronounce `sqlite` as `ess-cue-light`.

• do fleetsie_mod save to create a new installable image that includes fleetsie and your changes

  fleetsie_mod save [NEW_IMAGE_NAME]

• writes the filesystem with your changes to a new, xz-compressed image
• if NEW_IMAGE_NAME is omitted, it defaults to OSNAME_fleetsie.img.xz,
• normally, you would only do fleetsie_mod save after doing fleetsie_mod install and, optionally, making further changes to the part_N directories

  fleetsie_mod updatesd DEVPART1 DEVPART2 ...

• updates the filesystem partitions on an attached SD card from the current image
• DEVPART1 is updated from the partition 1 (at merged/part_1)
• DEVPART2 is updated from the partition 2 (at merged/part_2)
• and so on...
• don't include the /dev/ in DEVPARTN
• example: fleetsie_mod updatesd sdc1 sdc2

This can be useful for testing the provisioning code, by avoiding having to completely rewrite the SD card each time.

fleetsie.service

• installed and enabled by fleetsie_mod on the OS image
• once the system is stable, runs the fleetsie script, which is also installed by fleetsie_mod
• if fleetsie exits with success, fleetsie.service disables itself.

fleetsie_provision

• installed into /usr/bin by fleetsie_mod on the OS image

• run by the fleetsie-provision.service which is also installed and enabled there

• only runs when the system is stable

• is disabled from running after the first successful run

• searches for a script to run on an attached USB disk.

• the script must be called setup and must be in a top-level directory called fleetsie on the USB disk

• after switching to the top-level fleetsie directory on the disk, the script is run as root

• if the script run is successful, fleetsie-provision.service will disable itself.

• when the device registers, the fleet server will supply an extra file of (presumably) secrets; this is saved as /opt/fleetsie/fleet_extra, and can be used by custom setup scripts (see custom_post below)

ssh-tunnel service

fleetsie uses ssh to connect devices to the fleet host. During provisioning, each device is assigned a unique set of ssh keys for logging into the fleet host, and a unique tunnel port number. The ssh-tunnel service will maintain a reverse tunnel from that port on the fleet host to its local ssh port (22).

fleetsie_gen

fleetsie_gen generates a USB provisioning disk and server inventory for a fleet of devices.

Usage:

   fleetsie_gen FLEET_NAME FLEET_HOST [NUM] [USB_PARTITION]

where

• FLEET_NAME is the name of the fleet. It should be short and composed of alphanumeric characters and underscores. Devices belonging to the fleet will be assigned the hostnames FLEET_NAME-1, FLEET_NAME-2, ... Also, a user called fleetsie_FLEET_NAME will be created on the server, and devices in the fleet will login as that user.

• FLEET_HOST is the server (e.g. whoflewby.org) where the fleet will be hosted. The user running fleetsie_gen must have ssh set-up on FLEET_HOST so that they can login to fleetsie@FLEET_HOST, and so that user has sudo privileges.

• NUM (optional) is the number of devices to pre-allocate for the fleet on the server. If this is zero or missing, no devices are pre-allocated. Otherwise, SIZE new devices are allocated for the fleet, adding to any which are already there. e.g. if the fleet already has 100 device allocated (from a previous use of fleetsie_gen), the new devices will be named FLEET_NAME-101, FLEET_NAME-102, ...

• USB_PARTITION (optional) is the name of the disk partition on the user's machine (e.g. sda2) where the fleetsie files used for provisioning devices will be installed. If missing, no provisioning files are installed anywhere; this allows you to just allocate new fleet devices on the server without creating a USB disk.

  If USB_PARTITION begins with a /, it is treated as a path to a directory, and fleetsie_gen will create or use a subdirectory there called fleetsie as the destination for installing files, rather than a disk partition. This can be used for testing.

fleetsie_gen creates files on a USB drive and on the fleet server, such that a number of devices can be provisioned (using the USB drive) with access to the server.

USB drive layout

fleetsie_gen creates this layout on the USB drive:

/fleetsie

• top-level folder

/fleetsie/password.txt

• this file can be used to set initial passwords for user 1000 (e.g. 'pi') and the root user on the device. These passwords are only valid until the device has been registered with the fleet host, which will supply new passwords to replace these. In most cases, the passwords from this file will not be used, but if the provisioning process fails to register the device with the fleet host, then these passwords will allow local access to the device via bluetooth (root) and/or ssh over ethernet (user 1000).

• if this file exists, its first line becomes the password for user 1000 (e.g. pi on a Raspberry Pi) and for the root user. By default, a first line with 'change-this-password' is created, and passwords will only be changed if you modify that line.

• to use a different password for the root user, add it as the second line of this file.

/fleetsie/wifi.txt

• file containing ESSIDs and passwords for wifi networks, one per line i.e. line 1 = ESSID1, line 2 = password1, line 3 = ESSID2, line 3 = password2, ... During provisioning, the device will attempt to connect to wifi using these credentials, one set at a time, until a connection succeeds.

/fleetsie/fleet.txt

• file containing the fleet hostname on line 1, and the fleet name on line 2

/fleetsie/provisioning.sqlite

• database containing unused fleet device IDs and one-time passwords for claiming them

/fleetsie/fleetsieauth.pub
/fleetsie/fleetsieauth

• ssh keys used by the device to login to the fleet server at provisioning time. On the server, ssh is configured so that logging in with these keys runs the server-side provisioning code. No other use for these keys is permitted by the server.

/fleetsie/ssh-tunnel.service
/fleetsie/ssh_tunnel

• systemd service and the script it runs to maintain a ssh connection to the fleet server, with a port mapped from the server back to the local ssh server. It also maintains a local port which maps to the zabbix data port on the fleet server.

/fleetsie/custom_post

• this folder is for any other files or directories you want to provide to the provisioning process. These files are used at the last stage of provisioning, after a network connection has been obtained and the device has been registered to the fleet.

There are 3 files with special meanings for fleetsie_provision:

   /fleetsie/custom/setup

• if this script is found, then it is run from within the fleetsie/custom directory after the preceding provisioning steps have succeeded.

   /fleetsie/custom/overlay.tar.xz

• if this file is found, then it is uncompressed and extracted to the root directory (/). This happens as user root, so all parts of the system can potentially be overwritten - use with care!

   /fleetsie/custom/cleanup

• if this script is found, then it is run from within the fleetsie directory run after all other provisioning steps, including setup and extraction of overlay.tar.xz

fleetsie_provision ignores all other files and subdirectories in the fleetsie/custom_post directory, so you can populated it with whatever is needed by your setup and cleanup scripts.

EOF exit 1 }

fleetsie_srv - set up a server for use with fleetsie_gen

fleetsie_srv is run on the fleet manager's PC like so:

fleetsie_srv USER@SERVER

where USER must either be root, or a user with sudo privileges on SERVER

This script will set up SERVER via ssh like so:

• install any of these packages which are missing:
  • sqlite3
• create user fleetsie with sudo privileges; you will be prompted to enter a password for this user
• create sqlite database /home/fleetsie/fleets.sqlite with this schema:

CREATE TABLE devices (
     id INTEGER UNIQUE PRIMARY KEY NOT NULL,  -- unique ID for device, across all fleets
     fleet VARCHAR NOT NULL,                  -- name of fleet device belongs to
     id_in_fleet INTEGER NOT NULL,            -- unique ID for device within fleet
     fleetuser VARCHAR NOT NULL,              -- name of user device uses for ssh to fleet server; typically, fleetsie_FLEET
     hostname VARCHAR NOT NULL,               -- hostname for device; typically FLEET-ID_IN_FLEET
     hwid VARCHAR,                            -- hardware ID of device; NULL means no device registered to this record yet
     otp VARCHAR NOT NULL,                    -- one-time password used by device to register
     ts_generated DOUBLE NOT NULL,            -- unix timestamp for when this device record was generated
     ts_registered DOUBLE,                    -- unix timestamp for when this device was registered; NULL means not registered yet
     tunnel_port INTEGER NOT NULL,            -- TCP port mapped on server back to device SSH server port
     user_pwd VARCHAR,                        -- plaintext password for user 1000 on device (yes, should be stored hashed, not plaintext!)
     root_pwd VARCHAR,                        -- plaintext password for root user on device (yes, should be stored hashed, not plaintext!)
     device_public_key VARCHAR NOT NULL,      -- public key which can be used to login as user 1000 on device
     device_private_key VARCHAR NOT NULL,     -- private key which can be used to login as user 1000 on device
     server_public_key VARCHAR NOT NULL,      -- public key which device will use to ssh into fleet server
     server_private_key VARCHAR NOT NULL,     -- private key which device will use to ssh into fleet server
     ip_provisioned_from VARCHAR              -- IP address from which request to provision this device originated
 );
CREATE UNIQUE INDEX devices_hwid ON devices(hwid);
CREATE UNIQUE INDEX devices_fleet_otp_hwid ON devices(fleet, otp, hwid);
CREATE UNIQUE INDEX devices_fleet_id_in_fleet ON devices(fleet, id_in_fleet);

This table, initially empty, holds pre-allocated device records for one or more fleets. Entries in this table will be created by fleet administrators using fleetsie_gen. When entries are created, these fields are left NULL:

ts_provisioned
ip_provisioned_from
hwid

The NULL fields get set during device provisioning when a physical device presents an unused OTP password to register itself with the server. It is not known in advance which physical device will end up registered to each pre-allocated device, but fleetsie_auth ensures that no two physical devices get registered to the same device record.

fleetsie_gen_srv

This script runs the server-side of the fleetsie_gen process. It runs as user fleetsie.

It manages three versions of fleet database:

Main

• contains records for all allocated devices for all fleets
• lives in /home/fleetsie/fleets.sqlite
• new device records are created here because the set of ssh tunnel ports is shared across fleets on one server.

Fleet

• contains device records for a single fleet, FLEET
• lives in /home/fleetsie/fleets/FLEET/fleet.sqlite
• owned by user fleetsie_FLEET
• new device records get copied here when generated in Main
• new registrations are copied from here to Main whenever fleetsie_gen_srv is run for fleet FLEET
• fleetsie_auth uses this database when a new device is provisioning

Provisioning

• generated as /home/fleetsie/fleets/FLEET/provisioning.sqlite
• only contains the id_in_fleet and otp columns
• owned by user fleetsie_FLEET
• copied to the USB provisioning disk for a fleet by fleetsie_gen
• only contains records for the unclaimed devices in a fleet
• each time it is used to provision a device, the corresponding record is removed from the DB on the USB disk, and from /home/fleetsie/fleets/FLEET/provisioning.sqlite on the fleet server

whenever fleetsie_gen_srv is run for fleet FLEET, it does this:

• if the Fleet database for FLEET exists, any new registrations are copied from it to the Main database.

• if requested, new devices are allocated in the Main database

  • these are copied to the Fleet database
  • the reduced records (id_in_fleet, otp) are copied to the Provisioning database