Skip to content

Conversation

ppawlowski
Copy link
Contributor

Description

This pull request introduces the FlowFuse Device Agent Installer scripts. Written in Golang, on first iteration it aims to provide smooth installation experience for installing the Device Agent package on Linux systems.

Related Issue(s)

#382

Checklist

  • I have read the contribution guidelines
  • Suitable unit/system level tests have been added and they pass
  • Documentation has been updated
    • Upgrade instructions
    • Configuration details
    • Concepts
  • Changes flowforge.yml?
    • Issue/PR raised on FlowFuse/helm to update ConfigMap Template
    • Issue/PR raised on FlowFuse/CloudProject to update values for Staging/Production

Labels

  • Includes a DB migration? -> add the area:migration label

@knolleary
Copy link
Member

knolleary commented Apr 28, 2025

Why is this in golang rather than a native script? I may have missed if that choice had been discussed somewhere.

If we're asking users to download and run a script from the internet, it's generally better for that to be a script rather than a closed binary they can't check before it is run.

I'm not familiar with golang, so a code review is going to take some time to learn what/how its working.

@knolleary knolleary requested a review from Copilot April 28, 2025 10:32
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This pull request introduces the FlowFuse Device Agent Installer scripts written in Golang to enable a streamlined installation process for the Device Agent package on Linux systems. Key changes include new utilities for permissions and directory creation, systemd service configuration and management, and Node.js-based package installation and configuration.

Reviewed Changes

Copilot reviewed 10 out of 12 changed files in this pull request and generated no comments.

Show a summary per file
File Description
installer/go/pkg/utils/utils.go Added Unix-specific permission and directory management functions
installer/go/pkg/service/templates.go Introduced systemd service template
installer/go/pkg/service/service.go Added service wrapper functions for install, start, stop, etc.
installer/go/pkg/service/linux.go Implemented Linux-specific service installation and uninstallation
installer/go/pkg/nodejs/deviceagent.go Added Node.js integration for installing, uninstalling and configuring the agent
installer/go/pkg/logger/logger.go Added a configurable logger with file and console outputs
installer/go/pkg/config/config.go Implemented installer configuration save/load functionality
installer/go/main.go Main entry point that parses flags and drives installation/uninstallation
installer/go/cmd/install.go High-level installation and uninstallation orchestration
Files not reviewed (2)
  • installer/go/build.sh: Language not supported
  • installer/go/go.mod: Language not supported

@ppawlowski
Copy link
Contributor Author

Why is this in golang rather than a native script?

To have a proper cross-platform solution. With native scripts, we will have to maintain 2 separate scripts - one for Unix systems (and for sure there will be plenty of case/if functions to have support for both Linux and macOS), and second, in Powershell, for Windows os.
With golang we have a single codebase for every environment. Furthermore, the outcome is a single binary that can be distributed and executed on every supported OS.

I may have missed if that choice had been discussed somewhere.

I mentioned it a couple of times during our weekly meetings, with similar clarification, as above, on why we should go this way instead of regular shell scripts. No objections were made.

If we're asking users to download and run a script from the internet, it's generally better for that to be a script rather than a closed binary they can't check before it is run.

The code remains open, as a part of this public repo, and can be verified anytime by anyone.

I'm not familiar with golang, so a code review is going to take some time to learn what/how its working.

I left proper (I hope) descriptions and comments to improve readability.

@knolleary
Copy link
Member

My apologies for missing it when you brought it up previously.

With golang we have a single codebase for every environment. Furthermore, the outcome is a single binary that can be distributed and executed on every supported OS.

That makes sense - and actually, is a good reason to go this way.

@knolleary knolleary requested a review from Steve-Mcl April 30, 2025 09:38
@ppawlowski
Copy link
Contributor Author

the outcome is a single binary that can be distributed and executed on every supported OS

@knolleary to clarify the above sentence - we will have a single binary for each system/architecture variation, so the result of of "build and release" pipeline will be similar to this release: https://github.com/FalcoSuessgott/golang-cli-template/releases/tag/v0.3.0 .

@Steve-Mcl
Copy link
Contributor

Steve-Mcl commented Apr 30, 2025

Initial review conducted with Piotr.

Points of note

  • only systemd is supported
    • In my experience, systemd is not present on some mfg devices e.g. Siemens IOT2040 uses init.d
  • Service file is written with Environment="NODE_OPTIONS=--max_old_space_size=512" should we consider leaving this out (use default)
  • The installer installs a copy of nodejs locally in /opt/flowfuse-device
    • I have concerns about concerns about exec or contribs using the global install of nodejs
  • The installer did not prompt for sudo
  • Use of single dash cli args for long form. In all of our existing applications, we use single letter form with - and long form with double dash --
  • No docs on usage or what it does
  • No uninstaller or uninstall instructions
  • Does not offer to install the "necessary tools" for compiling binary contribs

Try out failures noted (installing on arm (PI5))

steve@rpi5:~ $ ./xxx -otc year-bit-plates -url http://192.168.86.46:3000
Logger initialized.
****************************************************************
*            FlowFuse Device Agent Installer                   *
*                                                              *
* This installer will set up the FlowFuse Device Agent on your *
* system and configure it to run as a system service.          *
*                                                              *
****************************************************************
Installing FlowFuse Device Agent...
This installer requires sudo access for some operations.
You will be prompted for your password when needed.
Checking Node.js installation...
Installing Node.js 20.19.0...
[ERROR] Node.js setup failed: failed to set permissions for node executable: exit status 1
Output: chmod: cannot access '/opt/flowfuse-device/node/bin/node': No such file or directory
[ERROR] Installation failed: node.js setup failed: failed to set permissions for node executable: exit status 1
Output: chmod: cannot access '/opt/flowfuse-device/node/bin/node': No such file or directory
steve@rpi5:~ $ ./xxx -otc year-bit-plates -url http://192.168.86.46:3000 -debug
Logger initialized.
[DEBUG] Debug logging enabled - detailed logs will be written to /tmp/flowfuse-device-installer-20250430-143915.log
[DEBUG] Command line arguments: node=20.19.0, agent=latest, user=flowfuse, url=http://192.168.86.46:3000, debug=true
****************************************************************
*            FlowFuse Device Agent Installer                   *
*                                                              *
* This installer will set up the FlowFuse Device Agent on your *
* system and configure it to run as a system service.          *
*                                                              *
****************************************************************
Debug mode enabled. Logs will be written to: /tmp/flowfuse-device-installer-20250430-143915.log
Installing FlowFuse Device Agent...
[DEBUG] ENTER: Install map[agentVersion:latest installerDir:/home/steve nodeVersion:20.19.0 otc:year-bit-plates url:http://192.168.86.46:3000]
[DEBUG] Checking permissions...
This installer requires sudo access for some operations.
You will be prompted for your password when needed.
[DEBUG] Creating working directory...
[DEBUG] Service user flowfuse already exists
[DEBUG] Setting ownership of /opt/flowfuse-device to flowfuse...
[DEBUG] Working directory created at: /opt/flowfuse-device
Checking Node.js installation...
[DEBUG] Node.js base dir: /opt/flowfuse-device/node
[DEBUG] Node.js path: /opt/flowfuse-device/node/bin/node
[DEBUG] NPM path: /opt/flowfuse-device/node/bin/npm
Installing Node.js 20.19.0...
[DEBUG] Creating directory /opt/flowfuse-device/node (requires sudo)...
[DEBUG] Downloading Node.js from https://nodejs.org/dist/v20.19.0/node-v20.19.0-linux-armv7l.tar.gz
[DEBUG] Extracting Node.js...
[DEBUG] Moving extracted files to /opt/flowfuse-device/node (requires sudo)...
[DEBUG] Setting execute permissions for Node.js binaries...
Node.js installed successfully!
[DEBUG] Node.js check/installation successful
Installing FlowFuse Device Agent package...
[DEBUG] Node.js base dir: /opt/flowfuse-device/node
[DEBUG] Node.js path: /opt/flowfuse-device/node/bin/node
[DEBUG] NPM path: /opt/flowfuse-device/node/bin/npm
[DEBUG] Installing @flowfuse/device-agent as user flowfuse...
[DEBUG] Install command: /usr/bin/sudo --preserve-env=PATH -u flowfuse /opt/flowfuse-device/node/bin/npm install -g @flowfuse/device-agent
[ERROR] Device Agent package installation failed: failed to install device agent: exit status 243
Output: npm error code EACCES
npm error syscall rename
npm error path /usr/lib/node_modules/@flowfuse/device-agent
npm error dest /usr/lib/node_modules/@flowfuse/.device-agent-UkeQNKPe
npm error errno -13
npm error Error: EACCES: permission denied, rename '/usr/lib/node_modules/@flowfuse/device-agent' -> '/usr/lib/node_modules/@flowfuse/.device-agent-UkeQNKPe'
npm error     at async Object.rename (node:internal/fs/promises:779:10)
npm error     at async moveFile (/opt/flowfuse-device/node/lib/node_modules/npm/node_modules/@npmcli/fs/lib/move-file.js:30:5)
npm error     at async Promise.allSettled (index 0)
npm error     at async [reifyPackages] (/opt/flowfuse-device/node/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/reify.js:325:11)
npm error     at async Arborist.reify (/opt/flowfuse-device/node/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/reify.js:142:5)
npm error     at async Install.exec (/opt/flowfuse-device/node/lib/node_modules/npm/lib/commands/install.js:150:5)
npm error     at async Npm.exec (/opt/flowfuse-device/node/lib/node_modules/npm/lib/npm.js:207:9)
npm error     at async module.exports (/opt/flowfuse-device/node/lib/node_modules/npm/lib/cli/entry.js:74:5) {
npm error   errno: -13,
npm error   code: 'EACCES',
npm error   syscall: 'rename',
npm error   path: '/usr/lib/node_modules/@flowfuse/device-agent',
npm error   dest: '/usr/lib/node_modules/@flowfuse/.device-agent-UkeQNKPe'
npm error }
npm error
npm error The operation was rejected by your operating system.
npm error It is likely you do not have the permissions to access this file as the current user
npm error
npm error If you believe this might be a permissions issue, please double-check the
npm error permissions of the file and its containing directories, or try running
npm error the command again as root/Administrator.
npm notice
npm notice New major version of npm available! 10.8.2 -> 11.3.0
npm notice Changelog: https://github.com/npm/cli/releases/tag/v11.3.0
npm notice To update run: npm install -g [email protected]
npm notice
npm error A complete log of this run can be found in: /home/flowfuse/.npm/_logs/2025-04-30T13_39_30_411Z-debug-0.log
[DEBUG] EXIT: Install returned error: failed to install device agent: exit status 243
Output: npm error code EACCES
npm error syscall rename
npm error path /usr/lib/node_modules/@flowfuse/device-agent
npm error dest /usr/lib/node_modules/@flowfuse/.device-agent-UkeQNKPe
npm error errno -13
npm error Error: EACCES: permission denied, rename '/usr/lib/node_modules/@flowfuse/device-agent' -> '/usr/lib/node_modules/@flowfuse/.device-agent-UkeQNKPe'
npm error     at async Object.rename (node:internal/fs/promises:779:10)
npm error     at async moveFile (/opt/flowfuse-device/node/lib/node_modules/npm/node_modules/@npmcli/fs/lib/move-file.js:30:5)
npm error     at async Promise.allSettled (index 0)
npm error     at async [reifyPackages] (/opt/flowfuse-device/node/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/reify.js:325:11)
npm error     at async Arborist.reify (/opt/flowfuse-device/node/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/reify.js:142:5)
npm error     at async Install.exec (/opt/flowfuse-device/node/lib/node_modules/npm/lib/commands/install.js:150:5)
npm error     at async Npm.exec (/opt/flowfuse-device/node/lib/node_modules/npm/lib/npm.js:207:9)
npm error     at async module.exports (/opt/flowfuse-device/node/lib/node_modules/npm/lib/cli/entry.js:74:5) {
npm error   errno: -13,
npm error   code: 'EACCES',
npm error   syscall: 'rename',
npm error   path: '/usr/lib/node_modules/@flowfuse/device-agent',
npm error   dest: '/usr/lib/node_modules/@flowfuse/.device-agent-UkeQNKPe'
npm error }
npm error
npm error The operation was rejected by your operating system.
npm error It is likely you do not have the permissions to access this file as the current user
npm error
npm error If you believe this might be a permissions issue, please double-check the
npm error permissions of the file and its containing directories, or try running
npm error the command again as root/Administrator.
npm notice
npm notice New major version of npm available! 10.8.2 -> 11.3.0
npm notice Changelog: https://github.com/npm/cli/releases/tag/v11.3.0
npm notice To update run: npm install -g [email protected]
npm notice
npm error A complete log of this run can be found in: /home/flowfuse/.npm/_logs/2025-04-30T13_39_30_411Z-debug-0.log
[ERROR] Installation failed: device agent installation failed: failed to install device agent: exit status 243
Output: npm error code EACCES
npm error syscall rename
npm error path /usr/lib/node_modules/@flowfuse/device-agent
npm error dest /usr/lib/node_modules/@flowfuse/.device-agent-UkeQNKPe
npm error errno -13
npm error Error: EACCES: permission denied, rename '/usr/lib/node_modules/@flowfuse/device-agent' -> '/usr/lib/node_modules/@flowfuse/.device-agent-UkeQNKPe'
npm error     at async Object.rename (node:internal/fs/promises:779:10)
npm error     at async moveFile (/opt/flowfuse-device/node/lib/node_modules/npm/node_modules/@npmcli/fs/lib/move-file.js:30:5)
npm error     at async Promise.allSettled (index 0)
npm error     at async [reifyPackages] (/opt/flowfuse-device/node/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/reify.js:325:11)
npm error     at async Arborist.reify (/opt/flowfuse-device/node/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/reify.js:142:5)
npm error     at async Install.exec (/opt/flowfuse-device/node/lib/node_modules/npm/lib/commands/install.js:150:5)
npm error     at async Npm.exec (/opt/flowfuse-device/node/lib/node_modules/npm/lib/npm.js:207:9)
npm error     at async module.exports (/opt/flowfuse-device/node/lib/node_modules/npm/lib/cli/entry.js:74:5) {
npm error   errno: -13,
npm error   code: 'EACCES',
npm error   syscall: 'rename',
npm error   path: '/usr/lib/node_modules/@flowfuse/device-agent',
npm error   dest: '/usr/lib/node_modules/@flowfuse/.device-agent-UkeQNKPe'
npm error }
npm error
npm error The operation was rejected by your operating system.
npm error It is likely you do not have the permissions to access this file as the current user
npm error
npm error If you believe this might be a permissions issue, please double-check the
npm error permissions of the file and its containing directories, or try running
npm error the command again as root/Administrator.
npm notice
npm notice New major version of npm available! 10.8.2 -> 11.3.0
npm notice Changelog: https://github.com/npm/cli/releases/tag/v11.3.0
npm notice To update run: npm install -g [email protected]
npm notice
npm error A complete log of this run can be found in: /home/flowfuse/.npm/_logs/2025-04-30T13_39_30_411Z-debug-0.log

Untested

  • Operating behind a proxy
  • Running the agent and trying out contrib nodes that have compiled components (like serial port)
  • Does the directory permissions (and all existing files, if any) get correct permissions applied
  • If systemd is not found, what will happen?

Other points (for the memory banks when doing windows)

  • Later versions of windows do have curl but in my experience, the most typical means of command line install use winget - this provides things like an "uninstall" entry in the system installed list
  • Moreso, windows users are more accustomed to proper installers
  • Path precedence and existence of nodejs in path must be considered.

@Steve-Mcl
Copy link
Contributor

  • Service file is written with Environment="NODE_OPTIONS=--max_old_space_size=512" should we consider leaving this out (use default)

Probably yes - see this thread on discourse

Related https://github.com/FlowFuse/flowfuse/issues/5482

@ppawlowski
Copy link
Contributor Author

only systemd is supported
In my experience, systemd is not present on some mfg devices e.g. Siemens IOT2040 uses init.d

Added support for sysvinit

The installer did not prompt for sudo

Improved how sudo command validation is made

No docs on usage or what it does
No uninstaller or uninstall instructions

Added basic documentation

Does not offer to install the "necessary tools" for compiling binary contribs

This is not the scope of the install script. I've added basic development information to the readme file.

@Steve-Mcl
Copy link
Contributor

Does not offer to install the "necessary tools" for compiling binary contribs

This is not the scope of the install script. I've added basic development information to the readme file.

I think we have crossed wires. I am referring to the tools needed for any contrib nodes that have a binary element (like the serial port nodes). If a contrib node has a compile step it will fail likely fail due to not having build-essential (incl python, node-gyp etc)

@Steve-Mcl
Copy link
Contributor

Steve-Mcl commented Jun 6, 2025

Morning @ppawlowski I have tested latest version & can now import existing flows.
Once README is updated I will do final review.
Please (if you agree) consider adding info for where logs can be viewed (windows files/linux journal etc), make clean, make, winget requirements for windows etc

Copy link
Contributor

@Steve-Mcl Steve-Mcl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One minor nitpick (feel free to ignore)

Comment on lines 148 to 149
# Clone and navigate to the installer directory
cd installer/go
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need the git clone ... here (or are you leaving that implicit?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added explicit clone command.

@Steve-Mcl
Copy link
Contributor

Although approved, please note, I would like a 2nd set of eyes on (I have been very close to this, so may no longer be impartial ;) )

@ppawlowski
Copy link
Contributor Author

Although approved, please note, I would like a 2nd set of eyes on (I have been very close to this, so may no longer be impartial ;) )

@hardillb can I ask for another review, please?

@hardillb
Copy link
Contributor

hardillb commented Jun 9, 2025

@ppawlowski just checking, I'll need to build this myself to test it? (not a problem, just checking)

@Steve-Mcl
Copy link
Contributor

@ppawlowski just checking, I'll need to build this myself to test it? (not a problem, just checking)

Yeah Ben, need to build. (the readme has good info)

@hardillb
Copy link
Contributor

hardillb commented Jun 9, 2025

I ran a test build using the golang docker container

cd installer/go
docker run -it -v .:/usr/src golang
make build

And it appears to have build the linux variants in the out/linux directory, but it has also created an empty {linux,macos,windows} directory

@Steve-Mcl
Copy link
Contributor

I ran a test build using the golang docker container

cd installer/go
docker run -it -v .:/usr/src golang{linux,macos,windows}
make build

And it appears to have build the linux variants in the out/linux directory, but it has also created an empty {linux,macos,windows} directory

I pointed this out too but @ppawlowski has said, once all branches are merged, them too will be populated (the other PRs he has populate all dirs)

It might be wise to start your testing on a decendant branch e.g. PR #413 - which is based off this.

@ppawlowski
Copy link
Contributor Author

I ran a test build using the golang docker container

cd installer/go
docker run -it -v .:/usr/src golang
make build

And it appears to have build the linux variants in the out/linux directory, but it has also created an empty {linux,macos,windows} directory

That's expected - this PR contains only Linux-related stuff.

@hardillb
Copy link
Contributor

hardillb commented Jun 9, 2025

Doesn't work on debian 11 or ubuntu 20.04 (both are technically EoL for standard LTS)
Complains about gibc version.

Basic install appears to work correctly on Ubuntu 22.04

./flowfuse-device-installer-linux-amd64 --otc highway-satellites-helpful --url http://192.168.122.1:3000
Logger initialized.
****************************************************************
*            FlowFuse Device Agent Installer                   *
*                                                              *
* This installer will set up the FlowFuse Device Agent on your *
* system and configure it to run as a system service.          *
*                                                              *
****************************************************************
Installing FlowFuse Device Agent...
Checking Node.js installation...
Node.js version 20.19.1 found.
Installing FlowFuse Device Agent package...
FlowFuse Device Agent installed successfully!
Configuring FlowFuse Device Agent...
ℹ Starting Device setup...
✔ Success! This Device can be launched at any time using the following command:
  flowfuse-device-agent
✔ Import existing Node-RED flows? Yes
✔ Enter the path to your Node-RED directory (leave blank to cancel): 
Configuration completed successfully!
Configuring FlowFuse Device Agent to run as system service...
Installing service flowfuse-device-agent for linux
FlowFuse Device Agent installation completed successfully!
The service is now running and will start automatically on system boot.

Copy link
Contributor

@hardillb hardillb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basic testing done, looks good.

Will review other features in their branches

@ppawlowski
Copy link
Contributor Author

Doesn't work on (...) ubuntu 20.04

I was able to install the device agent, using this installer, on:

NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"

@hardillb
Copy link
Contributor

hardillb commented Jun 9, 2025

This was with building the install in the golang:latest docker container

hardillb@flowforge:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.6 LTS
Release:	20.04
Codename:	focal
hardillb@flowforge:~$ cd device-agent/
hardillb@flowforge:~/device-agent$ ls
CHANGELOG.md  CODE_OF_CONDUCT.md  CONTRIBUTING.md  docker  frontend  index.js  installer  lib  LICENSE  package.json  package-lock.json  README.md  service  test
hardillb@flowforge:~/device-agent$ cd installer/go/out/linux/
hardillb@flowforge:~/device-agent/installer/go/out/linux$ ./flowfuse-device-installer-linux-amd64 
./flowfuse-device-installer-linux-amd64: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./flowfuse-device-installer-linux-amd64)
./flowfuse-device-installer-linux-amd64: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./flowfuse-device-installer-linux-amd64)

@ppawlowski
Copy link
Contributor Author

ppawlowski commented Jun 9, 2025

I am not able to reproduce this error on Ubuntu 20.04 LTS. Build made on my macOS. I've created a separate issue to investigate this once we start build and release via a proper CI/CD pipeline.

@ppawlowski ppawlowski changed the title feat: Device Agent Linux installer feat(installer): Device Agent Linux installer Jun 25, 2025
@ppawlowski ppawlowski merged commit 89d0784 into main Jun 25, 2025
4 checks passed
@ppawlowski ppawlowski deleted the feat-device-linux-installer branch June 25, 2025 12:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants