Installation extremly broken (complete Journey)

[CourteousBullet]

Working now 2 days on the installation. 1 day for the domain setup and this hole day with the installation...

1. The driver installation briked the system. It tried to compilate the driver on the system instead of installing the driver direclty (GTX 970)
   -> manually installing the driver before attempting the olares installation fixed it

2. Now im stuck at the regular installation. It says all the time at another point of the installation XYZ failed. First time the InstallationAddon exec now the InstallOsSystem and it was unable to build some kubernetes objects from release manifest

Oh and also if i try the installation over ssh it is just not possible at this point. Shutdown and Start Up also show some Systemparts need more than one try to boot up or down.

Rebootet the System and tried "olares-cli start" didn´t do anything exept throwing errors.

Tried "olares-cli gpu enable" worked through fine.

Another "olares-cli start" -> "execute successfuly!!!" but the olares is unreachable!!!
Intern IP-Adress doesn´t do anything the server refuses the connection and over my costum domain it also doesn´t do anything exept showing that the host is unreachable.

Over the discover Olares feature in the App I am able to see System-error but nothing more. Right IP-Adress right System Version I´m trying to install 1.12.0 and a shiny Uninstall button I´m more than tempted to try at this point...

At least the reboot now works at this point again as intempted.

At boot down and boot up some "Job cri-containerd-XXXX.scope/stop" needs like 1.30min to do smthing (X stands for an anormus amount of numbers)

After reboot the behauvior in the App is the same. Funny to say but at shutting down it sayed install in that shiny button O.o

No there is the tempting uninstall button again...

"olares-cli start" is throwing again the error "gpu-scheduler in pod os-gpu/gpu-scheduler-n7vf5 is terminated" it tries its best but can´t start :,)

It starts now for like 5 min I think it´s more dead as alive...

Feels more like an 0.12 than a 1.12 at this point...

After 13min start-up it´s unreachable as expected...
App has it´s uninstall button and I`m going to shutdown the machine and go watch some TV.

Bevore I was able to shutdown it says "

[ 713.6XXXXX] IPVS: rr: UDP 10.233.0.3:53 - no destination available

"
maybe this is valuable to work this out. :)

And the "shutdown now" needs at this point, again, like 1min to push through. Also this cri conainerd programm needs another 1:30min to shutdown alone...

If you have any idea what can help in this nightmare thank you very much in advance.

PC-Specs:
OS: Ubuntu 24.04.3 LTS fresh install exept the nvidia driver 575 is manually added because of above
CPU: old i7 3 i don´t now but i can look it up if needed has 4 cores 8 threads
RAM: 16GB DDR3
GPU: GTX 970
2TB of decent HDD
1 Gig RJ45 directly plugged in the wall

And before someone asks i can´t provide any logs because the machine has to stand besides me because ssh doesn´t work at this time due to described behaviore.

[dkeven]

Hi @CourteousBullet , thanks for your interest in Olares and we appreciate the patience you've made into debugging through the journey of installation.

You mentioned that fetching logs is not possible at now, so we cannot take a thorough dig and inspection for the root cause of the installation failure. However, based on the randomness of failure points and your hardware setup, we highly suspect it's caused by a lack of performance in the HDD disk.

Olares consists of multiple components, most of which runs on Kubernetes, with ETCD as the backing database, and it has a strict requirement on disk performances, especially IOPS, you can refer to https://etcd.io/docs/v3.3/op-guide/hardware/ for more infos. You can also test the disk's performance by fio like fio --rw=write --ioengine=sync --fdatasync=1 --directory=. --size=40m --bs=4096 --name=test.

As Olares is being rapidly developed, the load it poses on ETCD has been gradually increasing, our doc only says a SSD is recommended, but this should now be changed to a hard requirement. We apologize for the doc not being updated on time.

That being said, if you can take the time to fetch a complete set of logs by sudo olares-cli logs, and provide it to us, we'd be glad to take a further look and maybe spot the root cause more precisely.

As for the olares-cli start/stop commands, their purpose is provide an easy way for user to shutdown/start various system services of Olares, like etcd/containerd/K8s, etc., in one shot. But for an incomplete installation, the start command alone is unable to boot up the entire Olares system.

[CourteousBullet]

First of all sorry for my really emotional issue description thats normally not my style.

P.S.: The installation itself did work out at the end. olares-cli info says version 1.12.0 is installed just the boot up of olares broke somewhere and is since that not possible completely. I think maybe the pod startup for the gpu is also broken in my case, since that is the only "error" i get with the startup and if I try to get the logs it also says pod exit status 1 as u see below.

Also I don´t expect a super performant outcome it is meant as a test if we should implement it on beefier hardware.

Got ssh to work it´s a little bit laggy, but it works.

After logging into my machine the load was at ~5,7 so I think also the CPU is lacking a little in performance.

I stoped olares with olares-cli stop and performed the fio test.
Result:
root@olares:~# fio --rw=write --ioengine=sync --fdatasync=1 --directory=. --size=40m --bs=4096 --name=test
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1
fio-3.36
Starting 1 process
test: Laying out IO file (1 file / 40MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=60KiB/s][w=15 IOPS][eta 00m:01s]
test: (groupid=0, jobs=1): err= 0: pid=40376: Sat Aug 16 11:48:00 2025
write: IOPS=5, BW=20.1KiB/s (20.5kB/s)(40.0MiB/2042114msec); 0 zone resets
clat (usec): min=11, max=504780, avg=86.47, stdev=5005.24
lat (usec): min=11, max=504781, avg=87.26, stdev=5005.24
clat percentiles (usec):
| 1.00th=[ 21], 5.00th=[ 26], 10.00th=[ 28], 20.00th=[ 29],
| 30.00th=[ 30], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 31],
| 70.00th=[ 31], 80.00th=[ 33], 90.00th=[ 37], 95.00th=[ 44],
| 99.00th=[ 52], 99.50th=[ 56], 99.90th=[ 66], 99.95th=[ 71],
| 99.99th=[31327]
bw ( KiB/s): min= 8, max= 152, per=100.00%, avg=26.45, stdev=18.25, samples=3094
iops : min= 2, max= 38, avg= 6.61, stdev= 4.56, samples=3094
lat (usec) : 20=0.83%, 50=97.63%, 100=1.51%
lat (msec) : 50=0.02%, 750=0.01%
fsync/fdatasync/sync_file_range:
sync (msec): min=22, max=3563, avg=199.35, stdev=322.22
sync percentiles (msec):
| 1.00th=[ 25], 5.00th=[ 40], 10.00th=[ 44], 20.00th=[ 51],
| 30.00th=[ 55], 40.00th=[ 59], 50.00th=[ 63], 60.00th=[ 71],
| 70.00th=[ 107], 80.00th=[ 313], 90.00th=[ 567], 95.00th=[ 810],
| 99.00th=[ 1670], 99.50th=[ 2089], 99.90th=[ 3037], 99.95th=[ 3171],
| 99.99th=[ 3473]
cpu : usr=0.01%, sys=0.06%, ctx=20876, majf=0, minf=13
IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,10240,0,0 short=10239,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: bw=20.1KiB/s (20.5kB/s), 20.1KiB/s-20.1KiB/s (20.5kB/s-20.5kB/s), io=40.0MiB (41.9MB), run=2042114-2042114msec

Disk stats (read/write):
dm-0: ios=19/92816, sectors=984/13417016, merge=0/0, ticks=2642/6034614, in_queue=6037256, util=99.86%, aggrios=19/73179, aggsectors=984/13417184, aggrmerge=0/25895, aggrticks=2642/5973182, aggrin_queue=7848171, aggrutil=81.91%
sda: ios=19/73179, sectors=984/13417184, merge=0/25895, ticks=2642/5973182, in_queue=7848171, util=81.91%

I tried to get the logs.
Result:
root@olares:~ olares-cli logs
collecting logs for service: k3s
collecting logs for service: containerd
collecting logs for service: olaresd
collecting logs for service: etcd
collecting logs for service: NetworkManager
collecting dmesg logs ...
collecting logs from kubernetes cluster...
2025/08/16 11:48:59 error: failed to collect kubernetes logs: failed to get pods: exit status 1
root@olares:~

(edit: deleted the # at the first and last line so that everything is readable)

Hope that helps :)

One reccomendation for the documentation is also to change the installation part for the custom domain.
The hole export part is complete useless if the normal installation asks for exact these parameters.

[dkeven]

Hi, based on the fio output, the disk performance is indeed very poor (with a max bandwidth~100KiB, one figure avg IOPS, and max sync time as high as 3s), also you mentioned that the CPU's load is pretty high, so now it's quite sure that the hardware setup do not satisfy the minimum requirement.

What specific disk are you using, if I may ask? I'm a little curious because that even for old HDDs or SD cards, It's a pretty bad performance. IDK, too many bad sectors, or is there any other IO-intensive services running in the background?

Also, the olares-cli logs command communicates with the kubernetes server to get cluster infos, if the kubernetes service is down, the command needs to be executed with the option --ignore-kube-errors

[CourteousBullet]

I fiddelt around in the BIOS, turned on XMP and turned secure boot off, to squeeze out every drop of power and redone the fio test.
First stopped olares to get the raw results, waited like 5 min with apt update && apt upgrade -y, then fio like you expresed earlyier.
CPU is btw an i7 3770 so the hardware, except the HDD is from 2012. So I think it is more of a Motherboard Problem than an HDD Problem :/ but the test was significantly faster than last time.

root@olares:~# fio --rw=write --ioengine=sync --fdatasync=1 --directory=. --size=40m --bs=4096 --name=test
test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1
fio-3.36
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=144KiB/s][w=36 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=33087: Mon Aug 18 14:17:53 2025
write: IOPS=34, BW=137KiB/s (141kB/s)(40.0MiB/298062msec); 0 zone resets
clat (usec): min=11, max=23585, avg=40.22, stdev=509.71
lat (usec): min=12, max=23595, avg=40.96, stdev=509.75
clat percentiles (usec):
| 1.00th=[ 18], 5.00th=[ 23], 10.00th=[ 28], 20.00th=[ 28],
| 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 29], 60.00th=[ 29],
| 70.00th=[ 29], 80.00th=[ 29], 90.00th=[ 31], 95.00th=[ 35],
| 99.00th=[ 48], 99.50th=[ 51], 99.90th=[ 420], 99.95th=[ 2180],
| 99.99th=[23462]
bw ( KiB/s): min= 8, max= 336, per=99.69%, avg=137.93, stdev=58.60, samples=593
iops : min= 2, max= 84, avg=34.48, stdev=14.65, samples=593
lat (usec) : 20=3.36%, 50=96.05%, 100=0.39%, 500=0.14%
lat (msec) : 4=0.01%, 50=0.05%
fsync/fdatasync/sync_file_range:
sync (msec): min=11, max=1314, avg=29.06, stdev=32.25
sync percentiles (msec):
| 1.00th=[ 12], 5.00th=[ 16], 10.00th=[ 20], 20.00th=[ 20],
| 30.00th=[ 23], 40.00th=[ 24], 50.00th=[ 24], 60.00th=[ 28],
| 70.00th=[ 28], 80.00th=[ 28], 90.00th=[ 28], 95.00th=[ 36],
| 99.00th=[ 201], 99.50th=[ 209], 99.90th=[ 359], 99.95th=[ 418],
| 99.99th=[ 776]
cpu : usr=0.05%, sys=0.40%, ctx=20994, majf=0, minf=13
IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,10240,0,0 short=10239,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: bw=137KiB/s (141kB/s), 137KiB/s-137KiB/s (141kB/s-141kB/s), io=40.0MiB (41.9MB), run=298062-298062msec

Disk stats (read/write):
dm-0: ios=4/24096, sectors=32/116528, merge=0/0, ticks=203/450576, in_queue=450779, util=99.52%, aggrios=4/23363, aggsectors=32/116576, aggrmerge=0/746, aggrticks=204/428105, aggrin_queue=707453, aggrutil=98.92%
sda: ios=4/23363, sectors=32/116576, merge=0/746, ticks=204/428105, in_queue=707453, util=98.92%

Do you think it makes sense to go further, install smb and get the log file, or is it a time waist and we wont get the system up and working?

[pengpeng]

I sincerely apologize for the inconvenience caused. I would recommend not proceeding with the installation on an HDD.

This is because Olares relies on Kubernetes, which in turn depends on etcd, and etcd requires an SSD.

Based on the feedback we’ve received from users, Olares has very low CPU requirements. Users have successfully completed the installation on devices from the 2010s after upgrading to an SSD.

We will update the hardware requirements in the documentation and explore the possibility of using Kine as an alternative to etcd, making it feasible to install Olares on devices using HDDs.

[CourteousBullet]

I´m giving up. Maybe I´m going to buy a SSD and give it another try.
Installation runs through relativly smoothly. I uninstalled with olares-cli uninstall --all and let the skript run through again.
Works fine untill InstallPlugin: InstallPlugin exec failed after 1 retires: context deadline exceeded

P.S.: @dkeven the HDD is an Seagate BarraCuda 2TB (ST2000DM008)