Add Apptainer remote sensor node deployment

[nigeldaniels]

Summary

• add a remote Sensor Node deployer that keeps the existing SSH, host inspection, privilege, transfer progress, and headless service flow
• build a SensorNode SIF locally or deploy a supplied SIF as separate actions
• reuse the shared Installer Apptainer package path and the existing SensorNode install mode instead of maintaining a second dependency list
• keep configuration, certificates, source, plugins, UUID, logs, artifacts, recordings, and other mutable runtime data on the host outside the SIF
• validate normal deploys from service startup without requiring Dashboard or HIPRFISR; keep heartbeat validation as an explicit --health-only diagnostic
• add focused config, image, restart, cleanup, plugin sync, and source sync operations

Reviewer feedback addressed

• preserved the existing remote orchestration and limited the first pass to the online deployment path
• detect and reuse a compatible Apptainer installation, or install it through the shared Installer script when it is missing
• build through the existing Installer/Modes SensorNode path and retain the upstream setuid Apptainer package behavior
• split local --build from remote --deploy so a supplied SIF can be deployed without rebuilding it
• removed the rollback, release-listing, and release-pruning interface and its automatic rollback behavior
• removed the hard Dashboard/HIPRFISR dependency from deploy and restart checks
• added --hiprfisr-address so the intended endpoint can be supplied explicitly
• externalized mutable Sensor Node state instead of relying on a general writable image overlay
• preserve plugins and source by default with non-destructive --sync-plugins and --sync-source actions
• added lightweight --update-config, --update-image, --restart, and category-specific --clear-data actions
• kept the systemd unit as the thin headless lifecycle boundary while the existing autostart question remains open

Manual test plan

Review-driven CLI options

• --build builds locally without opening an SSH connection.
• --deploy deploys a supplied SIF without starting a local build.
• --update-config changes only the active configuration and preserves mutable state.
• --update-image changes only the active SIF and preserves configuration and mutable state.
• --restart restarts and validates the existing service without changing deployed files.
• --clear-data=logs removes only core and plugin logs.
• --clear-data=artifacts removes only artifact data.
• --clear-data=recordings removes only archive and recording data.
• --sync-plugins adds and updates plugins without deleting remote-only files.
• --sync-source updates the host source tree without deleting remote-only files or copying excluded state.
• --hiprfisr-address writes the requested endpoint into the deployed Sensor Node configuration.
• Mutually exclusive review-driven action flags are rejected before SSH work begins.

Fresh deployment

• Start with a disposable supported target and confirm SSH and root or sudo access.
• Record whether Apptainer is absent or already installed.
• Build a SIF with --build; confirm no SSH connection is made and apptainer test passes.
• Confirm the build uses the existing SensorNode install mode and shared Apptainer installer when needed.
• Deploy the SIF with --deploy and an explicit --hiprfisr-address while Dashboard and HIPRFISR are stopped.
• Confirm compatible remote Apptainer is reused, or the shared Installer path installs it when missing.
• Confirm the systemd service is installed, enabled, active, and uses the requested HIPRFISR address.
• Confirm configuration, certificates, source, plugins, UUID, logs, artifacts, and recordings are outside the SIF.
• Start HIPRFISR and confirm --health-only observes a fresh heartbeat.
• Restart and reboot the target; confirm the service reconnects with the same Sensor Node UUID.
• Create state sentinels, redeploy, update config and image, and confirm state is preserved.
• Sync plugins and source; confirm updates arrive and remote-only sentinels remain.
• Run each --clear-data category and confirm only the selected data is removed.
• Uninstall the deployment; confirm the service and deployment directory are removed while Apptainer remains installed.

Validation

• deployment Python modules compile successfully
• the rendered systemd unit passes systemd-analyze verify
• remote lifecycle and Apptainer installer scripts pass bash -n
• focused deployment tests pass locally; test files are not included in the PR
• manual image build and SensorNode startup were exercised against a remote node
• HIPRFISR heartbeat receipt was verified from the local FISSURE event log

Follow-ups

• certificate generation and enrollment refinement will be handled separately
• official SIF download and offline deployment support remain later work
• the existing autostart mechanism versus the thin systemd unit is still awaiting reviewer direction

Notes

• the generated SIF is intentionally not committed
• the manual checklist remains unchecked until it is exercised on a fresh target

[cpoore1]

Thanks for putting this together. Here's my quick AI response to all this without fully digging into everything. The things written in here are not fully where I stand on this, but I think it's a good starting point for conversation/modifications. Please respond with questions or comments and let me know if you need help developing any of this or if you want to keep working away on your own...

I spent some time working through what the intended Sensor Node deployment model should be and how this should fit with the installer work already in FISSURE.

The remote deployment capability is valuable, particularly the SSH orchestration, privilege handling, host inspection, file transfer, progress reporting, and ability to provision a headless machine without requiring someone to manually log in and perform each step. I would like to preserve that work.

The main concern is avoiding multiple places that define how a Sensor Node is built and installed. The existing FISSURE installer and the files under Installer/Modes should remain the source of truth for component selection, dependencies, Apptainer installation, and autostart behavior. Some of those modes were incomplete or had only recently been added when this work began, so it was reasonable to narrow the installation independently, but I do not want to maintain two separate Sensor Node installation paths going forward.

Ideally, this utility should become a remote orchestration layer around the existing installer rather than a separate deployment architecture. It should invoke the existing Sensor Node mode and reuse the current Apptainer and autostart logic wherever possible. I would prefer small refactors to expose that functionality rather than a major redesign of the installer.

For a new remote machine, I do want the deployer to be capable of preparing a mostly blank host. It should detect what is already installed, reuse compatible software, and install anything that is missing. Internet-based installation can be the easiest/default path when connectivity is available, but it would be useful to support offline packages later for systems that cannot or should not reach the internet. It should also be possible to skip host preparation when the host and hardware support are already configured.

For the SIF, the normal long-term order will probably be:

1. Use a supplied or previously downloaded SIF.
2. Download an official Sensor Node SIF when those are available.
3. Build one locally when no suitable image exists or the user needs a customized installation.

At the moment, building still needs to be supported because I do not yet host official images. The official Sensor Node SIF will likely contain some extra user-space software rather than being aggressively hardware-specific, since most hardware integration is handled at the host level and the additional container packages are usually not the largest problem.

SSH deployment should be treated as one provisioning method, not the required installation model for all Sensor Nodes. For many field systems, I will still prepare a known-good installation locally, clone or transfer an SD card, and then adjust the configuration. The SSH workflow is useful for truly remote, network-reachable computers, but deployment should not assume the node has permanent internet access or that a multi-gigabyte SIF will always be transferred over the operational network.

The deployment process also should not depend on a running Dashboard or HIPRFISR instance. A node should be able to be installed and staged independently, and then automatically connect to its configured HIPRFISR endpoint when both sides are available. A heartbeat check could be an optional diagnostic when HIPRFISR is running, but I would not make it a required deployment success condition. Seeing the node appear later in the Tactical tab or node-selection interface is sufficient operational confirmation.

The HIPRFISR address should be explicitly configurable. Inferring it from the SSH connection can be a useful default for a simple LAN, but the SSH management route and the FISSURE communications route may be different in VPN, cellular, mesh, NAT, or multi-hop deployments.

I would also prefer the deployer to use the existing FISSURE autostart implementation rather than introduce a separate authoritative service configuration. I test the current autostart behavior across several types of systems, and remote Sensor Node installations generally should enable it by default.

Mutable FISSURE data should remain on the host rather than being written into a general persistent overlay. This includes:

• Plugins
• artifacts and recordings
• logs
• configuration
• certificates
• source files or runtime overrides that are intentionally updated outside the SIF

The SIF should remain replaceable without losing those files. For plugin updates, I would avoid trying to intelligently merge conflicting files. The utility can preserve the existing remote Plugins directory by default and provide explicit replace or synchronize behavior when requested.

It would also be useful for the same command-line utility to support lighter operations after the initial installation, such as:

• pushing a new configuration file
• replacing or synchronizing the Plugins directory
• replacing or updating the FISSURE source directory
• transferring a new SIF
• restarting the Sensor Node
• optionally clearing artifacts, recordings, or logs

Existing user data should be preserved by default, with explicit cleanup options. Timestamped artifacts and recordings are straightforward to remove when requested. Plugins and source directories should only be replaced when the operator explicitly selects that behavior.

Certificate handling also needs some refinement. Keeping certificates outside the SIF is correct, but copying the same certificate directory to every machine is only a temporary solution. The installation process should use the existing FISSURE certificate-generation approach and eventually perform a secure exchange or enrollment so each node can receive the correct keys without transferring unnecessary private key material.

I am not looking to add a full release-management and rollback framework at this stage. FISSURE does not currently maintain formal releases, migrations, or compatibility rules across the host software, SIF, source, plugins, and configuration, so transactional rollback could become difficult to maintain. A simple backup of the previous SIF, config, or plugin directory is fine if it remains isolated and low-maintenance, but I would not make versioned releases and automatic rollback central to the deployment model.

In summary, I think the remote deployment work is useful and worth continuing. The desired direction would be:

• preserve the SSH, host inspection, privilege handling, transfer, and headless orchestration work
• use Installer/Modes and the existing installer as the single source of truth
• reuse the existing Apptainer and autostart implementation
• support preparing a blank host while preserving existing compatible installations
• support online installation first and offline packages later
• allow supplied, downloaded, or locally built SIFs
• remove the hard dependency on a running Dashboard/HIPRFISR
• make heartbeat validation optional
• keep mutable data in host directories
• support lightweight config, source, plugin, and SIF updates
• avoid building a separate release-management architecture

That should let this become a useful remote frontend for the existing FISSURE installation process without creating parallel systems that have to be maintained separately.

[nigeldaniels]

I would also prefer the deployer to use the existing FISSURE autostart implementation rather than introduce a separate authoritative service configuration. I test the current autostart behavior across several types of systems, and remote Sensor Node installations generally should enable it by default.

I agree with everything else, but I would like to push back slightly on this point.

For a remotely deployed, headless Sensor Node, I think the host’s init system should provide the user-facing process management interface. Once the installation succeeds, the operator’s next questions are usually: Is the node running? How do I stop or restart it? Will it start after a reboot? Where are its logs? On a systemd-based distribution, a unit file provides standard and immediately discoverable answers through systemctl and journalctl.

I understand the concern about creating two authoritative startup configurations, and I do not think the deployer should independently reimplement FISSURE’s startup behavior. A better approach may be for the deployer to install a thin systemd unit that invokes the existing FISSURE autostart entry point or uses configuration generated by the existing implementation. FISSURE would remain authoritative for application startup configuration, while systemd would handle operating-system-level supervision and lifecycle management? What do you think ?

My main concern is that a remote headless installation should expose a conventional service-management interface and enable it by default, rather than requiring the operator to understand FISSURE’s internal startup mechanism.

[nigeldaniels]

Allow supplied, downloaded, or locally built SIFs.

I may be missing part of this requirement. --image allows the user to provide any existing SIF, whether it was supplied, downloaded, or built locally. --output-image
image controls where the deployer writes a SIF that it builds.

What I have not provided is a way to specify a custom Apptainer definition file. Is that what you intended?, or does the existing SIF selection cover this
requirement?

[nigeldaniels]

Certificate handling also needs some refinement. Keeping certificates outside the SIF is correct, but copying the same certificate directory to every machine is only a temporary solution. The installation process should use the existing FISSURE certificate-generation approach and eventually perform a secure exchange or enrollment so each node can receive the correct keys without transferring unnecessary private key material.

This makes sense I made an issue for this that proposes a first pass solution for this.

[cpoore1]

Thanks for the updates. I went back through the current implementation and your responses, and I think this is getting very close to something I am ready to merge and start testing.

The overall direction looks good now. I like the SSH/SCP deployment flow, persistent state handling, source/plugin sync, update operations, health checks, and the use of systemd for a headless Sensor Node. I would like to try the systemd approach on both a normal Ubuntu system and an Ubuntu-based Raspberry Pi or similar ARM SBC. If it works reliably on both, I am comfortable moving toward that as the normal service model.

There are a few things I would like to tighten up before merging:

1. Keep the Sensor Node SIF build tied to the existing FISSURE installer.

   After looking more closely at remote_sensor_node_apptainer.def.j2, I see that it is providing a specialized headless Sensor Node container recipe while still calling the existing FISSURE installer with the SensorNode mode. I am fine with that approach.

   What I want to avoid is having two separate sources of truth for the actual Sensor Node installation logic. The existing ./install/SensorNode mode should continue to determine what gets installed inside the image.

   Where practical, I would also like common Apptainer setup/build logic reused rather than independently maintained between the existing installer and the remote builder.

2. Reuse the existing host hardware preparation for remote deployments.

   The current remote preflight handles Apptainer, but the existing Apptainer installer also contains host-side setup for things such as UHD/USRP, HackRF, RTL-SDR, Wi-Fi, udev rules, firmware/images, permissions, and other hardware-specific requirements.

   For an online remote deployment, I would like the deployment utility to invoke or reuse that existing host preparation rather than create another implementation of those installation steps in the remote Python code.

3. Let the remote deployment select its hardware requirements and pass those selections into the existing installer.

   I am fine with the local installer and remote deployment utility each having their own hardware-selection interface since they serve different workflows.

   For a remote deployment, however, the choices made by the remote utility should be passed into the existing installation routines as explicit parameters and should override or bypass the local installer's normal interactive selections. We should not select B210, HackRF, Wi-Fi, etc. remotely and then have the underlying installer prompt for those choices again.

   The goal is to reuse the existing installation routines while allowing the remote deployment utility to control them non-interactively.

4. Support the direction of an offline host-support package.

   I think the clean model is to keep host-side installation files separate from the SIF. A deployment could consist of a matching Sensor Node SIF plus a host-support package containing the Apptainer installer/packages, debs, wheels, firmware, udev files, drivers, and whatever else is required by the selected hardware.

   We already have online installation logic for most of this, so ideally the same installer routines can eventually work in two modes: download the required files when online, or consume them from a local host-support package when offline.

   I do not expect every supported device to work offline in this PR. A simple proof of the workflow with one representative dependency or radio would be enough to establish the pattern, and we can build out additional hardware support over time. If you would rather leave the offline implementation for me, that is also fine, but please avoid making the host-preparation path assume Internet access will always be available.

I am not looking for another major redesign. Most of the remote deployment subsystem looks useful and I think you are on the right track.

The main thing I want before merging is to make sure the remote workflow cleanly reuses the FISSURE installation and host-preparation logic we already have, while keeping the genuinely new remote orchestration pieces separate.

Once those boundaries are cleaned up, I am basically ready to merge this and start exercising it on real Ubuntu systems and ARM hardware.