improve build scripts and documentation

Author: ading2210

README.md

@@ -37,7 +37,7 @@ Driver support depends on the device you are using shimboot on. This list is for
 - Webcam
 
 ### What Doesn't Work:
-- Audio (due to a firmware bug)
+- Audio (due to a firmware bug on `dedede`, this works just fine on `octopus`)
 - Suspend (disabled by the kernel)
 - Swap (disabled by the kernel)
 
@@ -59,8 +59,11 @@ Driver support depends on the device you are using shimboot on. This list is for
 - Finish Python TUI rewrite
 
 ### Long Term Goals:
+- Transparent disk compression
+- Full disk encryption
 - eliminate binwalk dependency
 - get audio to work
+- get kexec working
 
 ## Usage:
 
@@ -71,37 +74,45 @@ Driver support depends on the device you are using shimboot on. This list is for
 - An x86-based Chromebook
 
 ### Build Instructions:
+1. Find the board name of your Chromebook. You can search for the model name on [chrome100.dev](https://chrome100.dev/).
+1. Clone this repository and cd into it.
+2. Run `sudo ./build_complete.sh <board_name>` to download the required data and build the disk image.
+
+Alternatively, you can run each of the steps manually:
 1. Grab a Chrome OS RMA Shim from somewhere. Most of them have already been leaked and aren't too difficult to find.
 2. Download a Chrome OS [recovery image](https://chromiumdash.appspot.com/serving-builds?deviceCategory=ChromeOS) for your board.
-3. Unzip the recovery image and shim if you haven't done so already.
-4. Clone this repository and cd into it.
-5. Run `mkdir -p data/rootfs` to create a directory to hold the rootfs.
-6. Run `sudo ./build_rootfs.sh data/rootfs bookworm` to build the base rootfs.
-7. Run `sudo ./patch_rootfs.sh path_to_shim path_to_reco data/rootfs` to patch the base rootfs and add any needed drivers.
-8. Run `sudo ./build.sh image.bin path_to_shim data/rootfs` to generate a disk image at `image.bin`. 
+3. Unzip the shim and the recovery image if you have not done so already.
+4. Run `mkdir -p data/rootfs` to create a directory to hold the rootfs.
+5. Run `sudo ./build_rootfs.sh data/rootfs bookworm` to build the base rootfs.
+6. Run `sudo ./patch_rootfs.sh path_to_shim path_to_reco data/rootfs` to patch the base rootfs and add any needed drivers.
+7. Run `sudo ./build.sh image.bin path_to_shim data/rootfs` to generate a disk image at `image.bin`. 
 
 ### Booting the Image:
-1. Obtain a shimboot image by downloading a [prebuilt one](https://dl.ading.dev/shimboot/) or building it yourself. 
+1. Obtain a shimboot image by downloading a [prebuilt one](https://github.com/ading2210/shimboot/actions) or building it yourself. 
 2. Flash the shimboot image to a USB drive or SD card. Use the [Chromebook Recovery Utility](https://chrome.google.com/webstore/detail/chromebook-recovery-utili/pocpnlppkickgojjlmhdmidojbmbodfm) or [dd](https://linux.die.net/man/1/dd) if you're on Linux.
 3. Enable developer mode on your Chromebook. If the Chromebook is enrolled, follow the instructions on the [sh1mmer website](https://sh1mmer.me) (see the "Executing on Chromebook" section).
 4. Plug the USB into your Chromebook and enter recovery mode. It should detect the USB and run the shimboot bootloader.
 5. Boot into Debian and log in with the username and password that you configured earlier. The default username/password for the prebuilt images is `user/user`.
 6. Expand the rootfs partition so that it fills up the entire disk by running `sudo growpart /dev/sdX 4` (replacing `sdX` with the block device corresponding to your disk) to expand the partition, then running `sudo resize2fs /dev/sdX4` to expand the filesystem.
+7. Change the root password and regular user password by running `sudo passwd root` and `passwd user`.
 
 ## FAQ:
 
 #### I want to use a different Linux distribution. How can I do that?
 Using any Linux distro is possible, provided that you apply the [proper patches](https://github.com/ading2210/chromeos-systemd) to systemd and recompile it. Most distros have some sort of bootstrapping tool that allows you to install it to a directory on your host PC. Then, you can just pass that rootfs dir into `build.sh`.
 
 #### How can I install a desktop environment other than XFCE?
-You can pass another argument to the `build_rootfs.sh` script, like this: `sudo ./build_rootfs.sh data/rootfs bookworm "task-lxde-desktop"`. The third argument is a list of packages that will be installed in the place of XFCE. 
+You can pass another argument to the `build_rootfs.sh` script, like this: `sudo ./build_rootfs.sh data/rootfs bookworm custom_packages=task-lxde-desktop`. The `custom_packages` argument is a list of packages that will be installed in the place of XFCE. 
 
 #### Will this prevent me from using Chrome OS normally?
 Shimboot does not touch the internal storage at all, so you will be able to use Chrome OS as if nothing happened. However, if you are on an enterprise enrolled device, booting Chrome OS again will force a powerwash due to the attempted switch into developer mode.
 
 #### Can I unplug the USB drive while using Debian?
 By default, this is not possible. However, you can simply copy your Debian rootfs onto your internal storage by first using `fdisk` to repartition it, using `dd` to copy the partition, and `resize2fs` to have it take up the entire drive. In the future, loading the OS to RAM may be supported, but this isn't a priority at the moment.
 
+#### GPU acceleration isn't working, how can I fix this?
+If your kernel version is too old, the standard Mesa drivers will fail to load. Instead, you must download and install the `mesa-amber` drivers. Download this zip file (https://shimboot.ading.dev/mesa-amber.zip), extract it, and install all of the `.deb` files.
+
 ## Copyright:
 Shimboot is licensed under the [GNU GPL v3](https://www.gnu.org/licenses/gpl-3.0.txt). Unless otherwise indicated, all code has been written by me, [ading2210](https://github.com/ading2210).
 

build.sh

@@ -2,30 +2,17 @@
 
 #build the bootloader image
 
-set -e
-if [ "$DEBUG" ]; then
-  set -x
-fi
-
+. ./common.sh
 . ./image_utils.sh
 . ./shim_utils.sh
 
 print_help() {
   echo "Usage: ./build.sh output_path shim_path rootfs_dir"
 }
 
-if [ "$EUID" -ne 0 ]; then
-  echo "this needs to be run as root."
-  exit 1
-fi
-
-if [ -z "$3" ]; then
-  print_help
-  exit 1
-fi
-
-. ./common.sh
+assert_root
 assert_deps "cpio binwalk pcregrep realpath cgpt mkfs.ext4 mkfs.ext2 fdisk rsync"
+assert_args "$3"
 parse_args "$@"
 
 output_path=$(realpath -m "${1}")

build_complete.sh

@@ -1,28 +1,19 @@
 #!/bin/bash
 
-set -e
-if [ "$DEBUG" ]; then
-  set -x
-  export DEBUG=1
-fi
-
 . ./common.sh
 
-if [ "$EUID" -ne 0 ]; then
-  echo "This script must be run as root."
-  exit 1
-fi
-
-if [ -z "$1" ]; then
+print_help() {
   echo "Usage: ./build_complete.sh board_name"
   echo "Valid named arguments (specify with 'key=value'):"
   echo "  compress_img - Compress the final disk image into a zip file. Set this to any value to enable this option."
   echo "  rootfs_dir   - Use a different rootfs for the build. The directory you select will be copied before any patches are applied."
   echo "  quiet        - Don't use progress indicators which may clog up log files."
-  exit 1
-fi
+}
 
+assert_root
+assert_args "$1"
 parse_args "$@"
+
 needed_deps="wget python3 unzip zip git debootstrap cpio binwalk pcregrep cgpt mkfs.ext4 mkfs.ext2 fdisk rsync"
 if [ "$(check_deps "$needed_deps")" ]; then
   #install deps automatically on debian and ubuntu

build_rootfs.sh

@@ -9,12 +9,7 @@ fi
 
 . ./common.sh
 
-if [ "$EUID" -ne 0 ]; then
-  echo "This script must be run as root."
-  exit 1
-fi
-
-if [ -z "$2" ]; then
+print_help() {
   echo "Usage: ./build_rootfs.sh rootfs_path release_name"
   echo "Valid named arguments (specify with 'key=value'):"
   echo "  custom_packages - The packages that will be installed in place of task-xfce-desktop."
@@ -23,10 +18,11 @@ if [ -z "$2" ]; then
   echo "  username        - The unprivileged user name for the new rootfs."
   echo "  user_passwd     - The password for the unprivileged user."
   echo "If you do not specify the hostname and credentials, you will be prompted for them later."
-  exit 1
-fi
+}
 
+assert_root
 assert_deps "realpath debootstrap"
+assert_args "$2"
 parse_args "$@"
 
 rootfs_dir=$(realpath -m "${1}")

build_squashfs.sh

@@ -5,29 +5,23 @@
 # - FUSE kernel modules from the shim
 # - unionfs-fuse statically compiled 
 # - the main squashfs, compressed with gzip
-#this script is currently incomplete
 
 set -e
 if [ "$DEBUG" ]; then
   set -x
 fi
 
+. ./common.sh
 . ./image_utils.sh
 . ./shim_utils.sh
 
 print_help() {
   echo "Usage: ./build_squashfs.sh rootfs_dir uncompressed_rootfs_dir path_to_shim"
 }
 
-if [ "$EUID" -ne 0 ]; then
-  echo "this needs to be run as root."
-  exit 1
-fi
-
-if [ -z "$3" ]; then
-  print_help
-  exit 1
-fi
+assert_root
+assert_deps "git make gcc binwalk pcregrep"
+assert_args "$3"
 
 compile_unionfs() {
   local out_path="$1"
@@ -41,7 +35,7 @@ compile_unionfs() {
   git clone $repo_url -b master --depth=1 $working_path
   cd $working_path
 
-  env LDFLAGS="-static" make -j$core_count
+  env LDFLAGS="-static" CFLAGS="-O3" make -j$core_count
   local binary_path="$working_path/src/unionfs"
   cp $binary_path $out_path
   cd $original_dir
@@ -54,28 +48,15 @@ shim_path=$(realpath -m $3)
 shim_rootfs="/tmp/shim_rootfs"
 root_squashfs="$rootfs_dir/root.squashfs"
 modules_squashfs="$rootfs_dir/modules.squashfs"
-kernel_dir=/tmp/shim_kernel
 unionfs_dir="/tmp/unionfs-fuse"
 
 echo "compiling unionfs-fuse"
 compile_unionfs $unionfs_dir/unionfs $unionfs_dir
 
-echo "creating loop device for shim"
-shim_loop=$(create_loop "${shim_path}")
-kernel_loop="${shim_loop}p2" #KERN-A should always be p2
-
-echo "copying shim kernel"
-rm -rf $kernel_dir
-mkdir $kernel_dir -p
-dd if=$kernel_loop of=$kernel_dir/kernel.bin bs=1M status=progress
-
-echo "extracting initramfs from kernel (this may take a while)"
-extract_initramfs $kernel_dir/kernel.bin $kernel_dir $rootfs_dir
+echo "reading the shim image"
+extract_initramfs_full $shim_path $rootfs_dir
 rm -rf $rootfs_dir/init
 
-echo "removeing shim loop device"
-losetup -d $shim_loop
-
 echo "compressing old rootfs"
 mksquashfs $old_dir $root_squashfs -noappend -comp gzip
 

common.sh

@@ -1,10 +1,16 @@
 #!/bin/bash
 
+set -e
+if [ "$DEBUG" ]; then
+  set -x
+  export DEBUG=1
+fi
+
 check_deps() {
   local needed_commands="$1"
   for command in $needed_commands; do
     if ! command -v $command &> /dev/null; then
-      echo $command
+      echo " - $command"
     fi
   done
 }
@@ -28,4 +34,18 @@ parse_args() {
     local value="${argument:$key_length+1}"
     args["$key"]="$value"
   done
-}
+}
+
+assert_root() {
+  if [ "$EUID" -ne 0 ]; then
+    echo "this needs to be run as root."
+    exit 1
+  fi
+}
+
+assert_args() {
+  if [ -z "$1" ]; then
+    print_help
+    exit 1
+  fi
+}

image_utils.sh

@@ -68,7 +68,6 @@ safe_mount() {
   umount $2 2> /dev/null || /bin/true
   rm -rf $2
   mkdir -p $2
-  #try to mount multiple times
   mount $1 $2
 }
 

shim_utils.sh

@@ -2,6 +2,7 @@
 
 #utilties for reading shim disk images
 
+#extract the initramfs from a kernel image
 extract_initramfs() {
   local kernel_bin="$1"
   local working_dir="$2"
@@ -23,3 +24,30 @@ extract_initramfs() {
   rm -rf $output_dir
   cat $cpio_path | cpio -D $output_dir -imd --quiet
 }
+
+copy_kernel() {
+  local shim_path="$1"
+  local kernel_dir="$2"
+
+  local shim_loop=$(create_loop "${shim_path}")
+  local kernel_loop="${shim_loop}p2" #KERN-A should always be p2
+
+  dd if=$kernel_loop of=$kernel_dir/kernel.bin bs=1M status=progress
+  losetup -d $shim_loop
+}
+
+#copy the kernel image then extract the initramfs
+extract_initramfs_full() {
+  local shim_path="$1"
+  local rootfs_dir="$2"
+  local kernel_dir=/tmp/shim_kernel
+
+  echo "copying the shim kernel"
+  rm -rf $kernel_dir
+  mkdir $kernel_dir -p
+  copy_kernel $shim_path $kernel_dir
+
+  echo "extracting initramfs from kernel (this may take a while)"
+  extract_initramfs $kernel_dir/kernel.bin $kernel_dir $rootfs_dir
+  rm -rf $kernel_dir
+}

website/index.html

@@ -68,21 +68,25 @@
 After copying all the firmware from the recovery image and shim to the rootfs, we're able to boot to a mostly working XFCE desktop.
 
 <b id="prebuilt">Prebuilt Images:</b>
-Prebuilt images are available <a href="https://dl.ading.dev/shimboot/">here</a>. If there is not a prebuilt image for your board, you must manually build the shimboot image. 
+Prebuilt images are available <a href="https://github.com/ading2210/shimboot/actions">here</a>. If there is not a prebuilt image for your board, you must manually build the shimboot image. 
 
 For these images, the root password is "root". The name of the default user is "user" and its password is "user" as well. You should change these credentials as soon as possible.
 
 <b id="build">Build Instructions:</b>
+1. Find the board name of your Chromebook. You can search for the model name on <a href="https://chrome100.dev/">chrome100.dev</a>.
+1. Clone this repository and cd into it.
+2. Run `sudo ./build_complete.sh [board_name]` to download the required data and build the disk image.
+
+Alternatively, you can run each of the steps manually:
 1. Grab a Chrome OS RMA Shim from somewhere. Most of them have already been leaked and aren't too difficult to find.
 2. Download a Chrome OS <a href="https://chromiumdash.appspot.com/serving-builds?deviceCategory=ChromeOS">recovery image</a> for your board.
-3. Clone this repository and cd into it.
-4. Run `mkdir -p data/rootfs` to create a directory to hold the rootfs.
-5. Run `sudo ./build_rootfs.sh data/rootfs bookworm` to build the base rootfs.
-6. Run `sudo ./patch_rootfs.sh path_to_shim path_to_reco data/rootfs` to patch the base rootfs and add any needed drivers.
-7. Run `sudo ./build.sh image.bin path_to_shim data/rootfs` to generate a disk image at `image.bin`. 
-
-<b id="usage">Usage:</b>
-1. Obtain a shimboot image by downloading a <a href="https://dl.ading.dev/shimboot/">prebuilt one</a> or building it yourself. 
+3. Run `mkdir -p data/rootfs` to create a directory to hold the rootfs.
+4. Run `sudo ./build_rootfs.sh data/rootfs bookworm` to build the base rootfs.
+5. Run `sudo ./patch_rootfs.sh path_to_shim path_to_reco data/rootfs` to patch the base rootfs and add any needed drivers.
+6. Run `sudo ./build.sh image.bin path_to_shim data/rootfs` to generate a disk image at `image.bin`. 
+
+<b id="usage">Booting the Image:</b>
+1. Obtain a shimboot image by downloading a <a href="https://github.com/ading2210/shimboot/actions/">prebuilt one</a> or building it yourself. 
 2. Flash the shimboot image to a USB drive or SD card. Use the <a href="https://chrome.google.com/webstore/detail/chromebook-recovery-utili/pocpnlppkickgojjlmhdmidojbmbodfm">Chromebook Recovery Utility</a> or <a href="https://linux.die.net/man/1/dd">dd</a> if you're on Linux.
 3. Enable developer mode on your Chromebook. If the Chromebook is enrolled, follow the instructions on the <a href="https://sh1mmer.me">sh1mmer website</a> (see the "Executing on Chromebook" section).
 4. Plug the USB into your Chromebook and enter recovery mode. It should detect the USB and run the shimboot bootloader.