Skip to content

Unstructured firmware development notes

William Wong edited this page Jun 8, 2023 · 53 revisions

How to Build Your Software

(1) Find a ticket on JIRA to work on in the current Sprint
(2) Pick a repository/source directory (such as morepork-machine) that gets impacted by the code change.
(3) Check out the base branch that this feature/bug fix will be merged into (e.g. ReleaseX.Y or develop)
(4) Create a new branch based on the JIRA ticket number and brief description inside that repository/source of interest.
(5) Edit the files in the repository of interest.
(6) Build the Repository of Interest.
	cd ~/software/Toolchain-Release/Toolchain/morepork-machine
	../../build-repos.py --this-dir (cross-compiles source, cannot be tested)
	OR
	../../ssh_install.sh <IP> --this-dir (cross-compiles source and installs to the bot <IP>
(7) Commit and push the changes
(8) Open a Pull Request then for Code Review.

Preparing a method to allow the SSH family of utilities to connect

Fully documented at: https://github.com/makerbot/Birdwing-Host-Utils and quick guide: https://drive.google.com/file/d/1F-j7Gjs3hcBpKEHhyr3ZSlQQ2YFkqUZj/view?usp=sharing

So if you don't have a UI, and front USB is unavailable

You need to connect a TTL cable and run:

systemctl stop morepork_ui_new_display
systemctl restart auther

and then run the host driver install_ceskey script again

Installing A Firmware ZIP On the Bot

(1) Download the firmware ZIP from Jenkins, MB website or other means.
(2) scp over the FW to the bot's /home/firmware directory
	scp -i ~/.ssh/CESkey ~/Downloads/firmware_lava_1_4_0_335.zip root@<IP>:/home/firmware/
(3) Then SSH to the bot and then use Repl to do a brooklyn_upload:
	# python -m repl
	Importing Kaiten
	Inspecting kaiten for available methods
	Welcome to the onboard repl
	> brooklyn_upload("firmware/firmware_lava_1_4_0_335.zip",False)

After the Install and Verification of the firmware finishes, the UI may prompt to install unsigned FW. Click Install on that prompt to continue.
The bot then should reboot when finished.

Main Bot Log Locations

/home/logs/kaiten/kaiten.log
/home/logs/printer/machine_<latest>.log
/home/logs/printer/network.log

Set the date and time when remoting into a method with wildly inappropriate date/time

date -s "2020-01-17 09:59"; hwclock -wu

What To Do When One of the Projects Does not Build Successfully

cd ~/software/Toolchain-Release/Toolchain/<non-building-project>
rm -rf ./_build*

Complete Clean, Code Checkout and Build of your Source Code Repository

cd ~/software/Toolchain-Release
rm -rf Install
python for-each-repo.py git clean -dfx
python update-repos.py -t morepork
python build-repos.py --clean
OR
python build-repos.py --clean --no-test (if one of the tests are failing and rootfs is not getting built)

Debugging the Extruder (Incomplete 10/11/2018 10:09 AM EST)

How to Debug using SWD Debugger for the Extruder: NOTE: If the extruder is being debugged in the bot, connect the Discovery board to the extruder using the SWD pins to the extruder. Do NOT use the John Cena board with the extruder if it's in the bot. Use the John Cena only if it's there's no power going to the board, such as debugging it standalone on your desk.

sudo apt-get install libgtk-3-dev
cd ~/software/
git clone https://github.com/texane/stlink.git stlink-src
cd stlink-src
make release
cd build/Release; sudo make install
sudo ldconfig
pip install gdbgui --upgrade

Run the 'st-util' command from any directory and if it connects to the debugger properly, the result is below:

wolfgang@Kamakura:~$ st-util
st-util 1.4.0-50-g7fafee2
2018-10-10T13:41:57 INFO usb.c: -- exit_dfu_mode
2018-10-10T13:41:57 INFO common.c: Loading device parameters....
2018-10-10T13:41:57 INFO common.c: Device connected is: L0x3 device, id 0x10086417
2018-10-10T13:41:57 INFO common.c: SRAM size: 0x2000 bytes (8 KiB), Flash: 0x10000 bytes (64 KiB) in pages of 128 bytes
2018-10-10T13:41:57 INFO gdb-server.c: Chip ID is 00000417, Core ID is  0bc11477.
2018-10-10T13:41:57 INFO gdb-server.c: Listening at *:4242...

Next, from another console/Terminal window, run:

cd ~/software/Toolchain-Release/Toolchain/morepork-extruder-firmware
rm -rf ./_build_morepork_extruder
../../build-repos.py --this-dir --debug
gdbgui -g ../arm-gcc-embedded/gcc-arm-none-eabi-5_3-2016q1/bin/arm-none-eabi-gdb ./_build_morepork_extruder/morepork-extruder-firmware.elf

The gdbgui should open up inside your Internet browser, such as Mozilla Firefox. Type in the following commands in the gdbgui to

target ext :4242
load
break <src_file>:<line> (or click Fetch Source files in the top left corner and navigate to the source file and line of interest. Left click the line in the resulting editor and the breakpoint will show up in the Breakpoints section on the right panel)
c (or just type continue)

When the debugger hits a breakpoint, you should be able to hover over variables in the editor. You can also see the variables on the stack on the right panel.

TBD: Screenshots.

Programming Extruder Configurations

# python -m repl
write_extruder_eeprom(0,1,0x00000000) -- v1 model (mk14) in slot 1
write_extruder_eeprom(1,1,0x00010000) -- v1 support (mk14_s) in slot 2
write_extruder_eeprom(0,1,0x00020000) -- v2 model hot (mk14_hot) in slot 1
write_extruder_eeprom(1,1,0x00030000) -- v2 support hot (mk14_hot_s) in slot 2
write_extruder_eeprom(0,1,0x00040000) -- Labs Extruder (mk14_e) in slot 1
write_extruder_eeprom(0,1,0x00050000) -- Composites Extruder (mk14_c) in slot 1

Commanding the bunk fans via repl

The top bunk fans can be turned on with the repl command toggle_top_bunk_fans(True) and turned off with the repl command toggle_top_bunk_fans(False). By default this sets both fans up with an RPM target of 8000 rpm, but that can be changed at any point with the command set_top_bunk_fan_rpm.

Troubleshooting Tips for Debugging

TIP 1: If you're modifying toolhead code, and wondering why you're not seeing changes in behavior, it's because the bot won't reprogram the toolhead microcontroller with your changes, unless the whole bot is rebooted after your development firmware is installed.

TIP 2: If your extruder is connected to the bot and you try to flash the firmware via the SWD pins using st-link and the load gdb command or just st-flash write, then you may see a flash write error. This is because the toolhead is trying to program the firmware at the same time. Currently, there is no way to stop this programming, so the workaround is use ssh_install script from wherever you want to upload the firmware.

TIP 3: If you do a ssh_install and reboot, you may see messages like this when you connect to the bot via the tty.usbserial:

[    3.676106] EXT4-fs error (device mmcblk0p2): ext4_find_extent:900: inode #7407: comm systemd-udevd: pblk 249241 bad header/extent: invalid magic)
[    3.743845] Aborting journal on device mmcblk0p2-8.
[    3.768415] EXT4-fs (mmcblk0p2): Remounting filesystem read-only
[    3.774528] EXT4-fs error (device mmcblk0p2): ext4_find_extent:900: inode #7407: comm systemd-udevd: pblk 249241 bad header/extent: invalid magic)
[    3.889472] EXT4-fs error (device mmcblk0p2): ext4_find_extent:900: inode #7407: comm systemd-udevd: pblk 249241 bad header/extent: invalid magic)
[  OK  ] Started Journal Service.
[    3.932276] EXT4-fs error (device mmcblk0p2): ext4_find_extent:900: inode #7407: comm systemd-udevd: pblk 249241 bad header/extent: invalid magic)
[    3.975329] EXT4-fs error (device mmcblk0p2): ext4_find_extent:900: inode #7407: comm systemd-udevd: pblk 249241 bad header/extent: invalid magic)
[    4.015338] EXT4-fs error (device mmcblk0p2): ext4_find_extent:900: inode #7407: comm systemd-udevd: pblk 249241 bad header/extent: invalid magic)
[    4.055329] EXT4-fs error (device mmcblk0p2): ext4_find_extent:900: inode #7407: comm systemd-udevd: pblk 249241 bad header/extent: invalid magic)
[    4.095337] EXT4-fs error (device mmcblk0p2): ext4_find_extent:900: inode #7407: comm systemd-udevd: pblk 249241 bad header/extent: invalid magic)
[    4.135380] EXT4-fs error (device mmcblk0p2): ext4_find_extent:900: inode #7407: comm systemd-udevd: pblk 249241 bad header/extent: invalid magic)
[    4.175381] EXT4-fs error (device mmcblk0p2): ext4_find_extent:900: inode #7407: comm systemd-udevd: pblk 249241 bad header/extent: invalid magic)

This means that the filesystem was not properly shutdown (probably due to the journaling nature of ext4, it's sensitive to power being pulled while writing). The solution is just to run fsck and reboot on the offending partition (mmcblk0p2 from above messages):

# fsck -y /dev/mmcblk0p2

Then just reboot the bot using the power button on the front panel and it should be back in a working state.

Adding a Backtrace to Code (Without Debugger)

(1) If there is enough FLASH memory, you can print out the backtrace with the Stack Addresses:

#include <unwind.h> // GCC's internal unwinder, part of libgcc
...
static _Unwind_Reason_Code trace_fcn(_Unwind_Context *ctx, void *d)
{
    int *depth = (int*)d;
    printf("\t#%d: program counter at %08x\n", *depth, _Unwind_GetIP(ctx));
    (*depth)++;
    return _URC_NO_REASON;
}
...
<Inside your method you want a backtrace of>:
_Unwind_Backtrace(&trace_fcn, &depth);

(2) Add the -funwind-tables flag to your CMake Compile options.

Core Dump from Motherboard

  1. First enable the core dump by outputting a valid path for /proc/sys/kernel/core_pattern. Use the references below for further instructions.
  2. Copy the core dump of some program locally.
  3. Use gdb to analyze the core dump: ~/software/Toolchain-Release/Toolchain/linaro-linux-gnu/gcc-linaro-5.3.1-2016.05-x86_64_arm-linux-gnueabihf/bin/arm-linux-gnueabihf-gdb -c core_dump

References: [1] https://sigquit.wordpress.com/2009/03/13/the-core-pattern/ [2] https://jvns.ca/blog/2018/04/28/debugging-a-segfault-on-linux/

Logging

  • Location on Bot: /home/logs/
  • Types of Logs: kaiten, printer and diagnostics
  • Printer type Log Filenames: chamber_telem_.log, telemetry_.log
  • telemetry_.log Columns (Source Code is Morepork-Toolhead/driver/cpp/src/morepork_toolhead.cpp): YYYY-MM-DD HH:MM:SS.mmmmmm,?,ToolheadNum,ErrorCode,ToolheadAttached,IsProgramming,MeasuredTemperature,TargetTemperature,EncoderTicks,ActiveFanRPM,GradientFanRPM,HallEffectValue,HeaterPWM,MoveCounter,IsWaitingToSync
  • Debug telemetry for STEPPERMOVEAXIS: YYYY-MM-DD HH:MM:SS.mmmmmm, ?, axis, ticks, velocity, acceleration, direction, suspend_ticks, suspend_acceleration, command_index, is_final_move, use_end_stop, pause_ok, send_sync_pulse, parser_index

Turning on Telemetry Logging

# python -m repl
> set_config({"logging":{"telemetry": "info"}})
None

Some Constants In FW

  • SECONDS_PER_TICK = 10e-6
  • TICKS_PER_SECOND = 1.0 / SECONDS_PER_TICK
  • STEPS_PER_MM = [88.888888, -88.888888, -1259.84, 1, 1, -72.76, -72.76] (for each axis)
  • STEPS_PER_FSTEP = float.fromhex('1') * pow(2,-32) (i.e. 1.0 * 2^-32)
  • MM_PER_FSTEP = [(STEPS_PER_FSTEP / x) for x in STEPS_PER_MM] (for each axis)

preparing a linux computer with the Birdwing Host Utils:

See: https://github.com/makerbot/Birdwing-Host-Utils#downloading-the-driver

Flashing a Bot with All Latest Built Software from Git Develop

cd ~/software/Toolchain-Release
rm -rf Install
./build-repos.py
cd ~/software/
git clone git@github.com:makerbot/Birdwing-Host-Utils.git
cd ~/code/src/Birdwing-Host-Utils 
python update_driver.py --select # This only has to be done once
<From the first prompt, select Birdwing_Develop>
<From the second prompt, Select latest build number>
python brooklyn_upload.py --ip=10.1.15.245 ../Toolchain-Release/Install/per_machine/fire/firmware_fire_0_ <tab this>

DVT3 Firmware on DVT2 Hardware Steps

cd /usr/settings/
mv use_new_display.blank use_new_display.blank.backup
touch th_enable_low.blank

Editing settings files in /usr/settings

The Python REPL command is known to fill the firmware installation partition with temp files. The solution was to mount it read-only to keep things neat.

1. determine which partition was mounted:
	cat /proc/cmdline
should return something like this:
	console=ttyS0,115200n8 loglevel=4 lpj=6000000 vt.global_cursor_default=0 root=/dev/mmcblk0p1 rootfstype=ext4 rootwait

2. if the "root=" device name ends in "p1", run the following to remount with read-write permissions:
	mount -o remount,rw /dev/mmcblk0p1 /
otherwise, if the "root=" device name ends in "p2", use the below:
	mount -o remount,rw /dev/mmcblk0p2 /

3. After you've made your changes, you MUST remount the partition in read-only mode, or else python will be free to fill your firmware partition with temp files. for p1:
	mount -o remount,ro /dev/mmcblk0p1 /
and for p2:
	mount -o remount,ro /dev/mmcblk0p2 /

Troubleshooting Bot When Kaiten Does not Start and Brooklyn Upload Does not Work

First, try switching back to the previous partition (if there is software on there). See the Switching Partition that Boots Firmware Section.
Otherwise, if all else fails, you can flash the root filesystem manually using the following steps:
(1) Start up the bot as usual
(2) Log in using either SSH or TTL in if you do not know the IP address.
	SSH (requires IP address and CESkey.pub has already been generated and copied over to bot):
		ssh -i ~/.ssh/CESkey root@<ip address>
	TTL command (requires specialty cable connected from USB to motherboard TTL pins):
		sudo screen /dev/ttyUSB0 115200
(3) Copy over the firmware file zip of interest to copy (ideally from Jenkins or your own developer zip) using scp to /home/firmware
(4) Next, run the following commands for your firmware version/location. Note to change the dd output (of) device to the other partition than the current root file system
# cat /proc/cmdline 
console=ttyS0,115200n8 loglevel=4 lpj=6000000 vt.global_cursor_default=0 root=/dev/mmcblk0p2 rw rootfstype=ext4 rootwait
NOTE THE root variable above. If your current running system is booted from /dev/mmcblk0p2, as above, you will write the firmware to the other filesystem /dev/mmcblk0p1
# cd /home/firmware/
# unzip firmware_fire_0_53_0_208.zip 
Archive:  firmware_fire_0_53_0_208.zip
  inflating: manifest.json
  inflating: hash
  inflating: signature
  inflating: rootfs
# dd if=/home/firmware/rootfs of=/dev/mmcblk0p1 bs=32768 conv=sync,noerror
8202+1 records in
8203+0 records out
268795904 bytes (256.3MB) copied, 23.290456 seconds, 11.0MB/s
(5) Don't forget to echo a 0 or 1 to /var/bootpartnr, based on the destination of dd.

Finding Out Why Some Application or Service Does not Start or Crashes

Here's a real-life example that uses Kaiten to figure out why it did not start upon bootup:

# gdb --args python -m kaiten.main
GNU gdb (GDB) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-buildroot-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/bin/python -m kaiten.main
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".

Program received signal SIGILL, Illegal instruction.
0xb66ed428 in _armv7_tick () from /lib/libcrypto.so.1.0.0
(gdb) c
Continuing.
Kaiten starting up!
[New Thread 0xb54f0470 (LWP 1746)]
terminate called after throwing an instance of 'PRUInterface::FileIOException'
  what():  Problem with /dev/rpmsg_pru29: 2 No such file or directory

Thread 1 "python" received signal SIGABRT, Aborted.
0xb6b9d996 in ?? () from /lib/libc.so.6
(gdb) quit
A debugging session is active.

	Inferior 1 [process 1739] will be killed.

Quit anyway? (y or n) y
# 

Booting a Linux Root Filesystem from the U-Boot Prompt

You may end up here, where U-Boot complains that it failed to load the kernel:

U-Boot SPL 2015.07-gb4d3dee (Mar 15 2018 - 17:06:41)                                                       
SPL: Please implement spl_start_uboot() for your board                                                     
SPL: Direct Linux boot not active!                                                                         


U-Boot 2015.07-00062-g1b72e78 (Mar 29 2018 - 11:31:45 -0400)

I2C:   ready
DRAM:  512 MiB
MMC:   OMAP SD/MMC: 0, OMAP SD/MMC: 1
Card did not respond to voltage select!
** Bad device mmc 0 **
Using default environment

Net:   <ethaddr> not set. Validating first E-fuse MAC
ethaddr loaded
<eth1addr> not set. Validating second E-fuse MAC
eth1addr loaded
Registering cpsw
cpsw
Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc1(part 0) is current device
Found Macho man eMMC
2 bytes read in 9 ms (0 Bytes/s)
Booting from partition 2 ...
ext4fs_devread read outside partition 630252
** File not found /boot/zImage **
Failed to load kernel
U-Boot# 

The following recovery steps assume you have at least one working root filesystem. Also, the way the second command that starts with if load mmc is formatted here is all on one line. So, please paste it all on one line.
setenv bootpartnr 1 # For booting from /dev/mmcblk0p1
OR
setenv bootpartnr 2 # For booting from /dev/mmcblk0p2
if load mmc 1:${bootpartnr} ${loadaddr} /boot/zImage; then load mmc 1:${bootpartnr} ${fdtaddr} /boot/${fdtfile}; setenv bootargs console=${console} ${optargs} root=/dev/mmcblk0p${bootpartnr} rw rootfstype=ext4 rootwait; bootz ${loadaddr} - ${fdtaddr}; else echo Failed to load kernel; fi;

Change the Boot Partition: 1) Review the contents of the bootpartnr: cat /var/bootpartnr 2) Edit the file /var/bootpartnr and change the 1 to a 0 or 0 to 1 (to whatever is the opposite value in the file). This file should only contain this character (and possibly /r or /n)!! If you use Vim, make sure vim states 1L, 2C when you go to write it. echo 0 > /var/bootpartnr or echo 1 > /var/bootpartnr 3) Now, reboot the Bot

Running the Production Line Diagnostics Script on your Local Machine

Connect your Ubuntu developer machine to the Sombrero bot using the USB-A cable (not TTL). Then on your developer machine, run the following commands:

cd ~/software/
git clone https://github.com/makerbot/Birdwing-Production-Line.git
git clone https://github.com/makerbot/Test-Framework.git
cd Birdwing-Production-Line
git checkout sombrero
python update_driver.py
cd line_scripts
<Edit fire_integration_config.json, tars_writer_configuration.json and/or production_config.json to suit your needs, such as where the output TARS files go>
python integration.py -l <Your current Ethernet MAC address separated by colons>

# It might even be better to comment out the Mac address setting part

Important Git Commands

Cloning to a Local Repository:
	git clone <Some_Git_Url>
List all Branches
    git branch -a
Switching to a Particular Branch
    git checkout -b <Branch_Name>
Making all Source Code Go to Develop Branch and Pulling the Latest Code:
	cd ~/software/Toolchain-Release
python for-each-repo.py git checkout develop
Commit Local Changes to a New Branch:
    git checkout -b <Branch_Name> (ideally involves ticket number)
git add <Any_New_File> (optional, if you created a new file that doesn't exist in remote repository)
	git commit (to get a list of the files changed)
git commit <Changed_File1> <Changed_File2> -m '<Title_Description_of_Commit>'
git push -u origin <Branch_Name>
Reset Local Branch to Remote Branch:
git fetch origin
git reset --hard origin/master

(Mostly) Deprecated Git Command:

"Git pull does not handle submodules. Morepork Dragon is the only component that currently uses submodules, so in general you can use git pull for just updating individual repositories, but you really want to be using update-repos to update all repositories." --CM

python for-each-repo.py git pull

Modifying the TI AM437x Kernel

cd ~/software/Toolchain-Release
rm -rf Toolchain/morepork-linux/
./update-repos.py -r morepork_linux
cd Toolchain/morepork-linux/
python build.py -m (allows to configure options via Kernel config)
<Make your modifications to the files, commit and then push to GitHub>
../../ssh_install.sh <Bot_IP> -r morepork_linux
OR
../../ssh_install.sh <Bot_IP> -r morepork_linux --script-args="--dt-only" (if you only end up modifying the dts file)

DTS File

Common DTS File: ~/software/Toolchain-Release/Toolchain/morepork-linux/arch/arm/boot/dts/mb-method-common.dtsi
Gen1 DTS File: ~/software/Toolchain-Release/Toolchain/morepork-linux/arch/arm/boot/dts/mb-macho-man-revd.dts
Gen2 DTS File: ~/software/Toolchain-Release/Toolchain/morepork-linux/arch/arm/boot/dts/mb-method-rev1.dts

These files uses reg-th-consumer for the "consumer" nodes that control the regulators.

In the DTS file, you will see an entry like the following:

0x22C (PIN_INPUT_PULLUP | MUX_MODE7) /* (H24) uart3_txd.pr0_pru0_gpi19 LID-OR-DOOR */

where
0x22C = CTRL_CONF_UART3_TXD - CTRL_CONF_GPMC_AD0 = 0xA2C - 0x800 = 0x22C
(PIN_INPUT_PULLUP | MUX_MODE7) = Treat as GPIO Input

Refer to: am4378.pdf (i.e. AM437x Sitara Processors Datasheet Rev. E)

DTB File

cat /proc/device-tree/model (checks which DTB file Linux booted with from U-Boot)

Some Insight Into Linux Kernel

cat /proc/interrupts (Shows interrupts the kernel is aware of)
cat /sys/kernel/debug/pinctrl/44e10800.pinmux/pingroups (Shows the GPIOs)

Kernel Module Debugging

lsmod (Find the name of the module of interest)
echo 8 > /proc/sys/kernel/printk (Enable a higher level Kernel logging, for dev_dbg/printk)
cat /sys/kernel/debug/dynamic_debug/control | grep ft6236 (Find the name of the C file that corresponds to your module of interest)
echo "file drivers/input/touchscreen/ft6236.c +p" > /sys/kernel/debug/dynamic_debug/control (Enable dynamic debugging for this C file)
rmmod ft6236 (Stop the module of interest)
modprobe ft6236 dyndbg==pmf (Start the module of interest with dynamic debug, overriding the previous settings)

Refer to https://www.kernel.org/doc/html/latest/admin-guide/dynamic-debug-howto.html

Watch Live Kernel Interrupts

watch -n1 "cat /proc/interrupts"

Test Linux Kernel IOCTLs Quickly on the Bot

Use the following Python script

#!/usr/bin/env python
import os
import sys
from subprocess import Popen, PIPE
import fcntl
ioctl_dev = sys.argv[1]
ioctl_num = int(sys.argv[2], 16)
ioctl_arg = int(sys.argv[3])

try:
		f = open(ioctl_dev, 'w', os.O_WRONLY)
		fcntl.ioctl(f, ioctl_num, ioctl_arg)
except Exception as inst:
		print("An Exception has occurred:", inst)

Sample ADC on Motherboard

echo 1 > /sys/bus/iio/devices/iio:device1/scan_elements/in_voltage0_en
echo 1000 > /sys/bus/iio/devices/iio:device1/buffer/length
<Then fread bytes /dev/iio:device1>

Build Build-Root

cd ~/software/Toolchain-Release rm -rf ~/software/Toolchain/morepork-build-root/ python update-repos.py -r morepork_build_root -c cd ~/software/Toolchain-Release/Toolchain/morepork-build-root/ git checkout morepork python menuconfig.py ... cd ~/software/Toolchain-Release python build-repos.py -r morepork_build_root (it's probably a better idea to build buildroot on Jenkins into a FW and test it that way)

Toggle Filament Bay Power Commands

echo enabled > /sys/devices/platform/mb-v3_3_aa-consumer/state
echo disabled > /sys/devices/platform/mb-v3_3_aa-consumer/state
echo enabled > /sys/devices/platform/mb-v3_3_bb-consumer/state
echo disabled > /sys/devices/platform/mb-v3_3_bb-consumer/state

Bot Error Codes in Source

Toolchain-Release/Toolchain/morepork-mbcoreutils/birdwing_codegen/contexts/errors.json (has Error_Name)

Suppressing Errors on the Bot

SSH into the bot and run the following commands to start repl and use the error suppression command:
ssh -i ~/.ssh/CESKey root@<Bot_IP>
python -m repl
(Now inside Repl prompt)
> set_toolhead_error_visibility("<Error_Name>", True)
> get_system_information() (Verify that the error shows up under disabled_errors)

Use the following commands to disable the common errors during development:
set_toolhead_error_visibility("no_filament",True)
set_toolhead_error_visibility("no_filament_loaded",True)
set_toolhead_error_visibility("door_interlock_triggered",True)
set_toolhead_error_visibility("lid_interlock_triggered",True)
set_toolhead_error_visibility("filament_slip",True)

You can then push the filament into extruder until it hits the Filament switch and seat it just before it reaches the drive gears.

Clear Latched Errors

python -m repl
(Now inside Repl prompt)
> machine_action_command('try_clear_toolhead_errors', {'index': 0, 'errors': [`errs`]}, ignore_tool_errors=True)

Printing/Air Printing Without Waiting For Chamber to Heat Up

Edit /usr/scripts/start.json to have the following lines (note that this makes "$CHAMBER_TEMPERATURE" equal to zero, so it doesn't take time to warm up the chamber)

{
"environment": {},
"sequence": [
    ["wait_for_file"],
    ["enable_dual_chamber_heaters"],
    ["load_temperature_settings", [0, 0, 0]],
    ["heat"],
    ["home_z"],
    ["move_axis", 2, 20, 10, true],
    ["home_gantry"],
    ["change_tool", 0],
    ["move", ["$X_PURGE_END", "$Y_PURGE_END", 0, 0, 0, 0, 0], 100,
         [false, false, true, true, true, true, true]],
    ["wait_for_heaters_at_target", 30, [false, false, true]],
    ["disable_dual_chamber_heaters"],
    ["load_temperature_settings",
        ["$EXTRUDER_0_TEMPERATURE", "$EXTRUDER_1_TEMPERATURE", 0]],
    ["heat"],
    ["wait_for_heaters_at_target", 5, [true, true, false]],
    ["toggle_top_bunk_fans", true],
    ["set_position", 3, 0],
    ["set_position", 4, 0],
    ["move_axis", 2, "$Z_PURGE_END", 100, false]
]

}

Protobuf Messages

Brief: The sender of a Protobuf message will use the *_encode method whereas the receiver will use the *_decode method. Example: Sender: mb_pb_chamber_encode_status() in morepork-chamber-heater/chamber/src/chamber_manager.cpp Receiver: pb_decode(&istream, moreporkchamber_ChamberStatus_fields, &msg) in morepork-chamber-heater/driver/cpp/src/morepork-chamber.cpp

Debugging the PRU

Use either Code Composer Studio with the Skyhawk JTAG debugger. OR ~/software/Toolchain-Release/Toolchain/morepork-libstepper/utils/pru_dbg.py AND ~/software/Toolchain-Release/Toolchain/morepork-libstepper/utils/annotate_asm.py

If using the scripts above, to know which executor axis you're looking at, simply add a volatile register variable for it and add some number and subtract it to make it appear as one of the GPREGs changed.

Adding new materials

Steps for adding materials for internal use:

I. Kaiten
    a) settings/machine_settings.json.mp_dual
        1: material_types
          API names for each new material need to be added.
        2: material_linear_densities
          Appropriate density values need to be added for each new material, both lists need to be the same total length.
    b) kaiten/src/kaiten/machine_manager.py
        1: "hide_dev_materials"
          The new (and unreleased) material API name needs to be added to this list to prevent the UI from approving the
          material during material checks when loading and starting a print.
II. morepork-ui
    a) src/model/bot_model.cpp
        1: Add user facing name to this function for this material -- QString BotModel::getMaterialName(QString apiName)

III. morepork-machine
    a) include/moreporkmachine/machine_structs.hh
        1: ExtrusionDist
            The struct definition needs names and formats for each new material.
    b) python-binding/pytypes.h
        1: extrusion_dist
            The tuple being built and assigned needs slots, names, and values added for each new material.
    c) settings/extruder_settings.json
        1: Add a new entry for the material under "material profiles".
        2: Add a reference to that material for every extruder type that is allowed to use the material.
    d) src/extruder_tracker.cpp
        1: stats.extrusion_dist_mm and ext.odometer_mm
            Init values need to be retrieved for each material use stat.
        2: ext.current_material
            Needs hook for checking membership in enum list before assignment.
    e) src/extruder_tracker.hh
        1: #define NUM_MATERIAL_TYPES
            Needs updating to match the number of entries in the below list.
        2: enum MaterialType
            New entries need enumerating, then the total counted and entered in the declaration above.

Steps to expose materials to user (public releasing of the material), do these:

I. kaiten
    a) kaiten/src/kaiten/machine_manager.py
        1: "hide_dev_materials"
          Remove material API name from this list to actually allow users to use the material through the UI, both loading and printing.

II. morepork-ui
    a) src/qml/ExpExtruderSettingsForm.qml
        1: expMaterialsList
            Load and unload temperatures need to be added for each new material.
    b) src/qml/CleanExtrudersForm.qml
        1: nozzleCleaningTempList
            Cleaning temperatures for each new material that appears under Advanced > Clean Extruders
    c) src/qml/DryMaterialForm.qml
        1: dryingMaterialsListMethod
            Filament drying cycle temperatures for the Method/Fire
        2: dryingMaterialsListMethodX
            Filament drying cycle temperatures for the MethodX/Lava

Some Coding Patterns

  • Mostly follow the Google Style Guides for C/C++ and Python: https://google.github.io/styleguide/
  • A bunch of variables related to a feature/functionality can be all put into a struct (see morepork-machine/src/machine_impl.hh and HomeConfig)
  • Create a reference for long variable names that are used throughout a method
  • Don't make something a coroutine in morepork-machine/src/machine.cpp unless it really needs to be- instead just make it return a machine::Error.
  • Prefer command indexes usage over parser indexes usage.

Debugging Pro-Tips

When debugging an issue, here are some first basic things before delving into more complex code that one can do:

  • Check the Initialisation/Assignment of the item of interest. For example, if there is a problem communicating over a GPIO, has the GPIO been initialised properly in terms of the Device Tree files or whatever structures are being used to access it?

  • Check that the expected messages between the different components are also being sent and parsed/processed correctly.

  • Check that the methods that you're looking at to understand the flow/structure are actually being used, as there may be dead-code or the methods may be overloaded.

  • If any numbers or physical quantities are involved, check that the numbers fall within a reasonable range. For example, the effect of assinging a float to an unsigned integer.

Reflector Setting Name

kaiten.constants.reflector_url_default = "reflector-staging.makerbot.com"

PiFlasher Image

Refer to the "Sombrero pi flasher.docx" document. Link: https://docs.google.com/document/d/1BPIMFGc--L4oxKiILRigwO12x_Ono7XXxDszKczYye8/edit?ts=5ae013d4#heading=h.i6p70p3sjxy4

(1) Download the PiFlasher image from y.soft.makerbot.net/files/morepork_pi_flash/rev_d_flasher.img
(2) Use fdisk to note where each partition starts from in the image in order to mount each one separately.
	wolfgang@Kagayaku:~/proj/pi_flasher$ fdisk -l ./pvt_pi_flasher.img
	Disk ./pvt_pi_flasher.img: 1.4 GiB, 1468215808 bytes, 2867609 sectors
	Units: sectors of 1 * 512 = 512 bytes
	Sector size (logical/physical): 512 bytes / 512 bytes
	I/O size (minimum/optimal): 512 bytes / 512 bytes
	Disklabel type: dos
	Disk identifier: 0xc9c95335

	Device                Boot  Start     End Sectors  Size Id Type
	./pvt_pi_flasher.img1        8192  137215  129024   63M  c W95 FAT32 (LBA)
	./pvt_pi_flasher.img2      137216 2867608 2730393  1.3G 83 Linux

(3) To access the files, mount as a loop partition with a start offset. The start offset is Start*Sector_Size. The start sector for the first FAT32 partition above then is 8192*512=4194304 and the start sector then for the main Linux partition above is 137216*512=70254592.

	$ sudo mkdir /mnt/loop1
	$ sudo mkdir /mnt/loop2
	$ sudo mount -o loop,offset=4194304 ./pvt_pi_flasher.img /mnt/loop1/
	$ sudo mount -o loop,offset=70254592 ./pvt_pi_flasher.img /mnt/loop2/

Boot bot from a different partition

$ cat /var/bootpartnr
if return value of cat is 0
$ echo 1 > /var/bootpartnr
if return value of cat is 1
$ echo 0 > /var/bootpartnr

REPL move_axis gets interrupted by Error 83 ("Drawer out of filament") for no reasonable reason

If you want commands like move_axis to ignore unrelated errors like "Drawer out of filament", the best way to do this on newer firmware is with the REPL command:

errors off

This will only affect command coming through from the REPL and will not produce any persistent changes to normal printer behavior. (It does mean that you have to run this command every time you start up a new REPL session.)

For firmware 1.11 and older you can instead use the REPL command:

set_toolhead_error_visibility('drawer_no_filament', True)

This will ignore the drawer not having filament in ALL contexts, including when you are printing and run out of filament. This setting will persist until explicitly removed or until a reset to factory.

if you get errors regarding undefined reference to xyz::staticMetaObject when working with/adding something to Qt:

The file, "CMakeLists.txt" needs to have your new header and code files added to lists inside of it, before it knows to process your new additions through QT's "meta object compiler" (moc, or "moc tool").

handy Method Spool reprogramming tricks:

Location of Toolhead USB on Linux

sysfs: /sys/devices/platform/44000000.ocp/483c0000.omap_dwc3/483d0000.usb/xhci-hcd.1.auto/usb1/1-1/1-1.P/00D
dev: /dev/bus/usb/001/00D

P = Port number D = device number.

Example on my bot:

TH0 sysfs: /sys/devices/platform/44000000.ocp/483c0000.omap_dwc3/483d0000.usb/xhci-hcd.1.auto/usb1/1-1/1-1.2/006
TH0 dev: /dev/bus/usb/001/006
TH1 sysfs: /sys/devices/platform/44000000.ocp/483c0000.omap_dwc3/483d0000.usb/xhci-hcd.1.auto/usb1/1-1/1-1.3/007
TH1 dev: /dev/bus/usb/001/007

Source Files for Adding Commands to the Toolhead

~/software/Toolchain-Release/Toolchain/Morepork-Protobufs/protofiles/toolhead_commands.proto
~/software/Toolchain-Release/Toolchain/Morepork-Protobufs/include/toolhead_coding.h
~/software/Toolchain-Release/Toolchain/Morepork-Protobufs/src/toolhead_coding.c
~/software/Toolchain-Release/Toolchain/morepork-machine/python-binding/interface.json
~/software/Toolchain-Release/Toolchain/morepork-machine/include/moreporkmachine/machine.hh
~/software/Toolchain-Release/Toolchain/morepork-machine/src/machine.cpp
~/software/Toolchain-Release/Toolchain/morepork-machine/src/machine_impl.hh
~/software/Toolchain-Release/Toolchain/morepork-machine/src/morepork_transceiver.hh
~/software/Toolchain-Release/Toolchain/morepork-machine/src/morepork_transceiver.cpp
~/software/Toolchain-Release/Toolchain/Morepork-Toolhead/firmware/include/toolhead_interface.hpp
~/software/Toolchain-Release/Toolchain/Morepork-Toolhead/driver/cpp/include/moreporktoolhead/morepork_toolhead.hpp
~/software/Toolchain-Release/Toolchain/Morepork-Toolhead/driver/cpp/src/morepork_toolhead.cpp
~/software/Toolchain-Release/Toolchain/Morepork-Toolhead/firmware/include/toolhead.hpp
~/software/Toolchain-Release/Toolchain/Morepork-Toolhead/firmware/src/toolhead.cpp
~/software/Toolchain-Release/Toolchain/Morepork-Toolhead/firmware/src/command_parser.cpp

Toolhead Boot Loader Extruder Firmware Check Sequence (~/software/Toolchain-Release/Toolchain/Morepork-Toolhead/firmware/app/src/bootloader.cpp)

  1. kWaitForExpectedFw: Waits for data structure to be available set by TH driver's SetExpectedFwInfo message, which gives the four start and end bytes of Extruder FW expected
  2. kCheckStartBytes: Compare four start bytes of extruder FW (using ReadMemory) starting at 0x80000000 with expected bytes (from SetExpectedFwInfo message)
  3. kCheckEndBytes: Compare four end bytes of extruder FW (using ReadMemory) starting at 0x80000000 with expected bytes (from SetExpectedFwInfo message)

Power Cycle the toolheads and restart kaiten without having to power cycle the bot:

if you want to change bot settings or for ssh_install

systemctl kill kaiten
i2cset -y 2 0x27 0x03 0x93
i2cset -y 2 0x27 0x02 0xe0
systemctl restart kaiten

The restart can be replaced with ssh_install from your computer

Debugging QML on the printer

You can get Qt creator to show you a live tree view of all QML components and their properties from the actual UI application running on a printer.

To start, you first have to compile the UI application to allow QML debugging. To do this you need to add the commands:

add_definitions(-DQT_DECLARATIVE_DEBUG)
add_definitions(-DQT_QML_DEBUG)

to the src/CMakeLists.txt of morepork-ui, and add the argument -qmljsdebugger=port:3456,host:127.0.0.1 to the command that starts the UI application. Then run a clean build and install to a printer.

Next you need to set up port forwarding from your laptop to the printer because for some reason Qt Creator does not support connecting to a different system. Running the command ssh -i ~/.ssh/CESkey root@<bot IP> -L 3456:localhost:3456 will set up port forwarding to the port that the printer is listening on.

Finally, you just need to open up Qt Creator, go to the Debug menu, and under Start Debugging > select Attach to QML Port ... (you do not have to have any project loaded to do this). Enter 3456 for the port and hit OK. You then get a little window pane in the upper right corner showing the QML tree starting from the QQMLApplicationEngine (none of the other panes show any relevant information so generally you want to make that one pane as big as possible).

Activate (hidden) time lapse capture feature

At the REPL prompt on the targeted Bot, type the following command:

set_config({'capture_timelapse_images':True})

Then edit the /home/settings/printer/ui_settings.json file to add/update this setting:

"capture_timelapse_images": true,

Clone this wiki locally