Document which packages MUST NOT be installed from upstream in a distribution

[renoirb]

Provided we run on a Pine64 a Longsleep packaged Linux distribution, some packages in Apt may get updated and affect which [kernel, initrd, ...] to install and break boot image.

It would be nice to get a pointer in the release notes either in the image download repository or the Pine64 forum thread about Ubuntu Xenial Image (BSP Kernel).

Maybe it's hidden somewhere in the forum, we could move this somewhere more authoritative, say maybe in a GitHub wiki in this project.

Problem

When we image a new SD card and make it up to date, we may get a package that should not be upgraded. After some time using image, when rebooting, the image becomes unusable.

Assumption is that maybe a package such as a kernel image, and an initrd hook superseeds kernel.

Reproduce

1. Image SD Card (see Makefile, image in question is xenial-pine64-bspkernel-20161218-1.img.xz)
2. Boot on an Pine64
3. Run pine64.sh (see pine64.sh)
4. Install a few packages, upgrade packages, reboot, iterate
5. Do not boot anymore (and/or IPv4 don't respond, video on HDMI port sends no signal)

I can't tell much right now. I don't have a Serial Port cable in hand to see what's happening.

Needs

• Confirmation on what to look for
• Hints on how to adapt our own image somewhere (wiki?)

Possible solution path

• List packages, per distribution, we hold (i.e. echo "linux-image hold" | dpkg --set-selections)

Related

(...) you can install and upgrade everything what is in the Ubuntu ports repository for arm64.
I do not provide packaging for Kernel, initrd and boot — so those cannot be upgraded through packaging.

Source

Talk on IRC

I'm renoirb both on Freenode and irc.pine64.xyz

[longsleep]

I am not exactly sure what you are looking for. There should be no packages which render the Pine unbootable on upgrade or dist-upgrade. Of course like with any additional package it might do things which could affect the system.

So the steps you provide above can you explain exactly what extra packages get installed which have an effect on initrd or kernel (assuming you do not install any base ubuntu repo kernel package).

[renoirb]

I think it's one package that adds modules to the kernel. Or triggers post scripts and rebuilds initrd.
Next time I'll keep a log of the packages and review which one may trigger what's breaking.
Problem is that I only know it broke when I reboot and it doesn't respond anymore.
I'll come back here as soon as I made another few rounds of testing.

[longsleep]

Even if a post-script would trigger a initrd rebuild it would not effect the initrd which is used for booting. I think its quite unlikely that a package is responsible.

It is much more likely that the issue is related to file system corruption resulting from under voltage or bad sd card.

If you can, please verify the boot of such a broken system with a UART serial console.

[renoirb]

Context: I'm obviously not a hardware person, rather a Linux automation geek

Maybe it's an hardware issue. But I may rule out power/voltage hardware as it's the same for all 5 nodes. It is a 4 port PDU (see quote below) and problem occurs on any of the 5 pines I have, until the "mysterious package of death" gets installed.

"Power input 9-40V and output 5V up to 24W max."
(source)

Next steps for me:

• Document heavily each package modification and keep log before rebooting
• Get an UART serial console cabling and access in place, dig from there

I'll keep it posted here.

[longsleep]

Ok cool, thanks for digging. If you can really find a package which renders my image unbootable after installation i am happy to provide a fix / workaround.

[renoirb]

PS: I'm fiddling around installing Docker. There MUST be something about asking Canonical upstream packages there.

I'll keep you posted

[longsleep]

PS: I'm fiddling around installing Docker. There MUST be something about asking Canonical upstream packages there.

Whats wrong with http://packages.ubuntu.com/xenial-updates/docker.io ?

[G07cha]

I also wasn't able to boot Pine64 running under xenial-pine64-bspkernel-20161218-1 image after executing sudo apt-get update && sudo apt-get upgrade and rebooting. Partition was resized uboot and kernel were updated via scripts provided in /usr/local/sbin successfully.

[renoirb]

I've been experimenting for a while now. Not as much as I'd like, but and I realized something I haven't thought: Pine64 hardware is flaky.

Even a good image, after installing packages, and reboot, may make a node NOT to reboot.

It's somehow annoying and not @longsleep’s work fault.

Things to do when you experience problems:

1. Run ALL your MicroSD cards against f3write/f3read.

2. If you trust your MicroSD

   • Remove power, power up that node and cross fingers.

If it doesn't work. Re-image.

I'm sure there's something. But during my process I haven't found anomalies. yet.

My process at the moment:

1. image
2. boot
3. upgrade software
4. install packages
5. cat a few commands to /var/log
6. archive /var/log
7. transfer elsewhere
8. reboot
9. cross fingers

That is all.

[G07cha]

@renoirb, it's sad to hear that, I've run f3 tests twice, SD card is perfect. And I agree, for the first time it's failed to boot after package updates as I mentioned earlier. But at second time it's failed to boot after a kernel update.
I've also tried Debian image, but it's didn't boot at all(at least no video output).

I'll continue testing in few hours and will let you know if I find anything!

[longsleep]

Most likely under voltage issue, so probably duplicate of #51, also see #16

[longsleep]

I was able to reproduce a boot-up panic with a specific USB device connected. PR longsleep/linux-pine64#56 seems to fix that. If you can please try if that change also fixes your particular issue.

[longsleep]

I am closing this as the initial issue is a non-issue. Crash on bootup discussion continues at #51