cbmem -L output with TPM1.2 under coreboot 24.12 segfaults

[marmarek]

Issue was renamed from symptom "Regenerating HOTP fails with "Unable to unseal HOTP secret" to "cbmem -L output with TPM1.2 under coreboot 24.12 segfaults" cause.

Please identify some basic details to help process the report

A. Provide Hardware Details

1. What board are you using? (Choose from the list of boards here)

x230

2. Does your computer have a dGPU or is it iGPU-only?

   • dGPU (Distinct GPU other then internal GPU)
   • iGPU-only (Internal GPU, normally Intel GPU)

3. Who installed Heads on this computer?

   • Insurgo (Issues to be reported at https://github.com/linuxboot/heads/issues)
   • Nitrokey (Issues to be reported at https://github.com/Nitrokey/heads/issues)
   • Purism (Issues to be reported at https://source.puri.sm/firmware/pureboot/-/issues)
   • Novacustom (Issues to be reported at https://github.com/Dasharo/dasharo-issues)
   • HardnenedVault (Issues to be reported at https://github.com/hardenedvault/vaultboot/issues)
   • Other provider
   • Self-installed

4. What PGP key is being used?

   • Librem Key (Nitrokey Pro 2 rebranded)
   • Nitrokey Pro
   • Nitrokey Pro 2
   • Nitrokey 3 NFC
   • Nitrokey 3 NFC Mini
   • Nitrokey Storage
   • Nitrokey Storage 2
   • Yubikey
   • Other

5. Are you using the PGP key to provide HOTP verification?

   • Yes
   • No
   • I don't know

B. Identify how the board was flashed

1. Is this problem related to updating heads or flashing it for the first time?

   • First-time flash
   • Updating heads

2. If the problem is related to an update, how did you attempt to apply the update?

   • Using the Heads menus
   • Flashrom via the Recovery Shell
   • External flashing

3. How was Heads initially flashed?

   • External flashing
   • Internal-only / 1vyprep+1vyrain / skulls
   • Don't know

4. Was the board flashed with a maximized or non-maximized/legacy rom?

   • Maximized
   • Non-maximized / legacy
   • I don't know

5. If Heads was externally flashed, was IFD unlocked?

   • Yes
   • No
   • Don't know

C. Identify the rom related to this bug report

1. Did you download or build the rom at issue in this bug report?

   • I downloaded it
   • I built it

2. If you downloaded your rom, where did you get it from?

   • Heads CircleCi
   • Purism
   • Nitrokey
   • Dasharo DTS (Novacustom)
   • Somewhere else (please identify)

   Please provide the release number or otherwise identify the rom downloaded

3. If you built your rom, which repository:branch did you use?

   • Heads:Master
   • Other (please identify)

4. What version of coreboot did you use in building?
   { You can find this information from github commit ID or once flashed, by giving the complete version from Sytem Information under Options --> menu}

5. In building the rom, where did you get the blobs?

   • No blobs required
   • Provided by the company that installed Heads on the device
   • Extracted from a backup rom taken from this device
   • Extracted from another backup rom taken from another device (please identify the board model)
   • Extracted from the online bios using the automated tools provided in Heads
   • I don't know

Please describe the problem

Describe the bug

After Heads update (via the menu) regenerating HOTP secret fails with message that it fails to unseal the HOTP secret. But I'm regenerating it exactly because it (as expected) couldn't be unsealed after the update!

To Reproduce
Steps to reproduce the behavior:

1. Fetch heads-x230-hotp-maximized_usb-kb-v0.2.0-2731-gf572998.zip from CI.
2. Update Heads via the menu.
3. Reboot the system after update, see "TOTP Generation Failed" message
4. Choose "Generate new HOTP/TOTP secret"
5. Confirm ("Yes")
6. See QR code, confirm to "configure your HOTP USB Security dongle"
7. See the message "Error PCR mismatch from TPM_Unseal"

Expected behavior

New secret written to the USB dongle (after the admin pin prompt).

Screenshots

Additional context
Add any other context about the problem here.

[marmarek]

I tried also "Reset TPM" flow, with similar result.

[tlaurion]

Asked same question under Matrix thread of this issue

I cannot replicate this issue with x230-hotp-maximized (non-usb_kb), can you turn DEBUG output on under Options -> Change configuration settings -> Configure Heads informational output -> Debug

I am working on HOTP resealing/TPM TOTP resealing under #1935

But nothing should prevent you to reset TPM1.2 on x230, which would take reownership of TPM and reseal TPMTOP

[tlaurion]

My hypothesis is that unseal operations is not taking into consideration that usb HID driver is required in unsealing operation.
Will try to find time testing this in debug, thought this was tested as part of standard, non usb variant + usb keyboard enabled through configuration options?

[tlaurion]

My hypothesis is that unseal operations is not taking into consideration that usb HID driver is required in unsealing operation. Will try to find time testing this in debug, thought this was tested as part of standard, non usb variant + usb keyboard enabled through configuration options?

Nope. Can't replicate @marmarek

• Used Librem Key (equiv of nk2 pro, should be irrelevant anyway)
• I tested flashing heads-x230-hotp-maximized_usb-kb-v0.2.0-2731-gf572998.zip over 230-hotp-maximized internally, wiping settings
  • On reboot, TPMTOTP fails unsealing
    • Tested TPM Reset. Succeeds resealing TPMTOTP, generates QR code and then unsealing succeed when attempting to seal HOTP.
  • OEM Factory Reset (basically TPM Reset which wipes nvram region used to seal secrets), reboot
    • HOTP reseal works (Generate new HOTP/TOTP secret which seals TPMTOTP secret in TPM nvram, then generate Qr code, then attempts to reverse seal TPMTOTP secret through HOTP...).

Debug logs from /tmp/debug.log would help

• Options -> Change configuration settings -> Configure Heads informational output -> Debug : follow on screen instructions.
• mount-usb --mode rw
• cp /tmp/debug.log /media
• umount
• Provide /media/debug.log file here

[marmarek]

debug.log

The above is from executing the same flow as in the issue description.

[tlaurion]

debug.log

The above is from executing the same flow as in the issue description.

Sorry @marmarek but the log provided is related to

Xen 4.21-unstable

Not Heads. Try again :)

[marmarek]

Oops, wrong file, lets try again:

debug.log

[tlaurion]

Oops, wrong file, lets try again:

debug.log

My log

thierry-debug.log

diff -u marek-debug.log thierry-debug.log:

--- marek-debug.log	2025-04-16 14:22:30.397393800 -0400
+++ thierry-debug.log	2025-04-16 14:45:04.250000000 -0400
@@ -2,20 +2,20 @@
 TPM: Extending PCR[7] with filename /.gnupg/pubring.kbx and then its content
 TPM: Extending PCR[7] with hash 7ccf4f64044946cf4e5b0efe3d959f00562227ae
 LOG: exec stdout: New value of PCR[7]: 1beb88e70c9acf0fa187725205db206675dea4ab
-TPM: Extending PCR[7] with hash e6b956db64eb14ef1a22feef42576d18152d0c7d
-LOG: exec stdout: New value of PCR[7]: 8675167a245590244381acae46f90b9355b60e14
+TPM: Extending PCR[7] with hash 68affd0a2a8aa8b3c0385183a5e57ec41153065e
+LOG: exec stdout: New value of PCR[7]: 101de4a7cb48393c87f40dcb5a4cfc0ded37b21c
 Extracting CBFS file heads/initrd/.gnupg/trustdb.gpg into /.gnupg/trustdb.gpg
 TPM: Extending PCR[7] with filename /.gnupg/trustdb.gpg and then its content
 TPM: Extending PCR[7] with hash 7236ea8e612c1435259a8a0f8e0a8f1f5dba7042
-LOG: exec stdout: New value of PCR[7]: b6f378512ae1df0b71e105a983d6499aadcc6561
-TPM: Extending PCR[7] with hash 9c7fd882a5a3ec0322403a079ceca5dcc4a7f004
-LOG: exec stdout: New value of PCR[7]: 03acfe701eab6376abc594008bff0f6e3734f251
+LOG: exec stdout: New value of PCR[7]: ee4a9d98d87cbd9e14b936e4affe9c0a47fa7055
+TPM: Extending PCR[7] with hash ed593a95731274240266893865c6a6a7b220801b
+LOG: exec stdout: New value of PCR[7]: a6ab5b287f6ac0e7a6a5acf7a41dae5fafcbab5b
 Extracting CBFS file heads/initrd/etc/config.user into /etc/config.user
 TPM: Extending PCR[7] with filename /etc/config.user and then its content
 TPM: Extending PCR[7] with hash a0c02a9f82b98bc70224059ca53af19782ca0b28
-LOG: exec stdout: New value of PCR[7]: 42bef5f461984d203c7abcb8f9d0d5ec5c11747f
-TPM: Extending PCR[7] with hash 936e97235f2076c83dc362d8507604385ac766a9
-LOG: exec stdout: New value of PCR[7]: 88883d8ab62c642ef4d50527960695bd94f0bbe6
+LOG: exec stdout: New value of PCR[7]: 700dea51122158abb7a6b2881dbd7b31462cbe61
+TPM: Extending PCR[7] with hash bb52e739677822a892e4f3cb0417749a83684e13
+LOG: exec stdout: New value of PCR[7]: 4c5334aad32424a1969d4d5f11b202702f86e003
 TRACE: /init(102): main
 DEBUG: Full debug output enabled from this point: output both in dmesg and on console (equivalent of passing debug to kernel cmdline)
 DEBUG: NOTE: DO_WITH_DEBUG std_err and std_out will be redirected to /tmp/debug.log
@@ -92,14 +92,8 @@
 TRACE: /etc/functions(1046): is_gpt_bios_grub
 TRACE: /dev/sda1 is partition 1 of sda
 TRACE: /etc/functions(1023): find_lvm_vg_name
-TRACE: /etc/functions(1109): mount_possible_boot_device
-DEBUG: Partition /dev/sda1 is very small, likely BIOS boot. Skipping mount.
-TRACE: /etc/functions(1086): mount_possible_boot_device
-TRACE: /etc/functions(1046): is_gpt_bios_grub
-TRACE: /dev/sda2 is partition 2 of sda
-TRACE: /etc/functions(1023): find_lvm_vg_name
 TRACE: /etc/functions(1113): mount_possible_boot_device
-DEBUG: Try mounting /dev/sda2 as /boot
+DEBUG: Try mounting /dev/sda1 as /boot
 TRACE: /bin/gui-init(310): clean_boot_check
 TRACE: /bin/gui-init(338): check_gpg_key
 TRACE: /bin/gui-init(178): update_totp
@@ -128,7 +122,7 @@
 PCR-04: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 PCR-05: 15 E5 9E D1 7F 19 E8 B6 2F 35 7C 00 72 51 31 11 1A 10 06 35
 PCR-06: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
-PCR-07: 88 88 3D 8A B6 2C 64 2E F4 D5 05 27 96 06 95 BD 94 F0 BB E6
+PCR-07: 4C 53 34 AA D3 24 24 A1 96 9D 4D 5F 11 B2 02 70 2F 86 E0 03
 TRACE: /bin/gui-init(150): generate_totp_hotp
 TRACE: /bin/seal-totp(10): main
 DEBUG: Sealing TOTP with actual state of PCR0-3
@@ -152,15 +146,14 @@
 TRACE: /bin/tpmr(197): replay_pcr
 TRACE: /bin/tpmr(108): extend_pcr_state
 TRACE: /bin/tpmr(35): hex2bin
-TRACE: /bin/tpmr(35): hex2bin
-DEBUG: Replayed cbmem -L clean boot state of PCR=4 ALG=sha1 : 01a7743dbaff104014b36d254f5c3615cd9b7c51
+DEBUG: Replayed cbmem -L clean boot state of PCR=4 ALG=sha1 : 0000000000000000000000000000000000000000
 DEBUG: Sealing TOTP neglecting PCR5 involvement (Dynamically loaded kernel modules are not firmware integrity attestation related)
 DEBUG: Sealing TOTP without PCR6 involvement (LUKS header consistency is not firmware integrity attestation related)
 DEBUG: Sealing TOTP with actual state of PCR7 (User injected stuff in cbfs)
 TRACE: /bin/tpmr(71): tpm1_pcrread
 TRACE: /bin/tpmr(35): hex2bin
 DEBUG: tpm pcrread -ix 7
-LOG: tpm stdout: 88883d8ab62c642ef4d50527960695bd94f0bbe6
+LOG: tpm stdout: 4c5334aad32424a1969d4d5f11b202702f86e003
 TRACE: /bin/tpmr(505): tpm1_seal
 DEBUG: tpm1_seal arguments: file=/tmp/secret/totp.key index=4d47 pcrl=0,1,2,3,4,7 pcrf=/tmp/secret/pcrf.bin sealed_size=312 pass=<empty> tpm_password=<empty>
 DEBUG: Running at_exit handlers
@@ -171,45 +164,41 @@
 TRACE: /bin/tpmr(626): tpm1_unseal
 DEBUG: Running at_exit handlers
 TRACE: /bin/tpmr(394): cleanup_shred
-DEBUG: lsusb output: Bus 003 Device 001: ID 1d6b:0002
-Bus 002 Device 002: ID 8087:0024
-Bus 001 Device 001: ID 1d6b:0002
-Bus 001 Device 004: ID 147e:2020
-Bus 003 Device 002: ID 1d6b:0104
-Bus 001 Device 002: ID 8087:0024
-Bus 004 Device 001: ID 1d6b:0003
-Bus 001 Device 006: ID 04f2:b2ea
-Bus 002 Device 001: ID 1d6b:0002
-Bus 001 Device 005: ID 0a5c:21e6
-Bus 001 Device 003: ID 20a0:4108
- !!! ERROR: Unable to unseal HOTP secret !!!
+TRACE: /bin/seal-hotpkey(12): mount_boot
+TRACE: /etc/functions(364): enable_usb
+TRACE: /sbin/insmod(9): main
+DEBUG: /lib/modules/ehci-hcd.ko: already loaded, skipping
+TRACE: /sbin/insmod(9): main
+DEBUG: /lib/modules/ehci-pci.ko: already loaded, skipping
+TRACE: /sbin/insmod(9): main
+DEBUG: /lib/modules/xhci-hcd.ko: already loaded, skipping
+TRACE: /sbin/insmod(9): main
+DEBUG: /lib/modules/xhci-pci.ko: already loaded, skipping
+TRACE: /sbin/insmod(9): main
+DEBUG: /lib/modules/usbhid.ko: already loaded, skipping
+DEBUG: Signature key was created at Wed Apr 16 18:39:38 UTC 2025
+DEBUG: HOTP related PIN retry counter is 3
+ *** WARNING: Default GPG Admin PIN detected.  Please change this as soon as possible with Options > OEM Factory Reset / Re-Ownership ***
 TRACE: /bin/gui-init(178): update_totp
 TRACE: /bin/unseal-totp(8): main
 TRACE: /bin/tpmr(626): tpm1_unseal
 DEBUG: Running at_exit handlers
 TRACE: /bin/tpmr(394): cleanup_shred
- !!! ERROR: Unable to unseal TOTP secret from TPM !!!
-TRACE: /bin/unseal-totp(8): main
+TRACE: /etc/functions(585): reseal_tpm_disk_decryption_key
+DEBUG: LUKS TPM Disk Unlock Key is allowed in board configs. Continuing
+DEBUG: No TPM disk decryption key to reseal
+TRACE: /bin/gui-init(246): update_hotp
+TRACE: /bin/unseal-hotp(18): main
+TRACE: /bin/unseal-hotp(10): mount_boot_or_die
+DEBUG: Unsealing HOTP secret reuses TOTP sealed secret...
 TRACE: /bin/tpmr(626): tpm1_unseal
 DEBUG: Running at_exit handlers
 TRACE: /bin/tpmr(394): cleanup_shred
- !!! ERROR: Unable to unseal TOTP secret from TPM !!!
-TRACE: /bin/unseal-totp(8): main
-TRACE: /bin/tpmr(626): tpm1_unseal
-DEBUG: Running at_exit handlers
-TRACE: /bin/tpmr(394): cleanup_shred
- !!! ERROR: Unable to unseal TOTP secret from TPM !!!
-DEBUG: CONFIG_TPM: y
-DEBUG: CONFIG_TPM2_TOOLS:
-DEBUG: Show PCRs
-DEBUG: PCR-00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
-PCR-01: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
-PCR-02: 0D 4E 81 D5 32 0C E8 BE FB F1 09 6F B0 C3 77 A1 50 5A 81 3E
-PCR-03: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
-PCR-04: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
-PCR-05: 15 E5 9E D1 7F 19 E8 B6 2F 35 7C 00 72 51 31 11 1A 10 06 35
-PCR-06: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
-PCR-07: 88 88 3D 8A B6 2C 64 2E F4 D5 05 27 96 06 95 BD 94 F0 BB E6
+DEBUG: Incrementing HOTP counter under /boot/kexec_hotp_counter
+TRACE: /bin/gui-init(374): prompt_auto_default_boot
+TRACE: /bin/gui-init(665): main
+TRACE: /bin/gui-init(383): show_main_menu
+TRACE: /bin/gui-init(415): show_options_menu
 TRACE: /etc/functions(259): recovery
 DEBUG: Board x230-hotp-maximized_usb-kb - version CBET4000 Heads-v0.2.0-2731-gf572998
 TPM: Extending PCR[4] to prevent any further secret unsealing
@@ -249,12 +238,12 @@
 TRACE: /etc/functions(659): list_usb_storage
 DEBUG: Listing USB storage devices (including partitions)
 DEBUG: USB storage device of size greater then 0: /sys/block/sdb
-DEBUG: USB storage device without partition table: /dev/sdb
+DEBUG: USB storage device with partition table: /dev/sdb
 DEBUG: Cleaning /media directory
 TRACE: /etc/functions(659): list_usb_storage
 DEBUG: Listing USB storage devices (including partitions)
 DEBUG: USB storage device of size greater then 0: /sys/block/sdb
-DEBUG: USB storage device without partition table: /dev/sdb
-DEBUG: Checking if /dev/sdb is a LUKS device/partition
+DEBUG: USB storage device with partition table: /dev/sdb
+DEBUG: Checking if /dev/sdb2 is a LUKS device/partition
 DEBUG: Selected USB partition is not a LUKS device, continuing...
-DEBUG: Mounting /dev/sdb as read-write
+DEBUG: Mounting /dev/sdb2 as read-write

Analysis:
Well, for unknown reasons to me, your PCR4 is not zeroed (clean boot state).

Snippets of diff

-DEBUG: Replayed cbmem -L clean boot state of PCR=4 ALG=sha1 : 01a7743dbaff104014b36d254f5c3615cd9b7c51
+DEBUG: Replayed cbmem -L clean boot state of PCR=4 ALG=sha1 : 0000000000000000000000000000000000000000

So of course, trying to unseal from a expected clean state (without having been in recovery shell) fails and TPMTOTP secret sealed in nvram cannot be unsealed. But that doesn't explain why this value is not 0 though, as it should be.

So all PCRs related for firmware and dynamic modules loaded are the same (PCR 0-5):

PCR-00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
PCR-01: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
PCR-02: 0D 4E 81 D5 32 0C E8 BE FB F1 09 6F B0 C3 77 A1 50 5A 81 3E
PCR-03: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
PCR-04: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
PCR-05: 15 E5 9E D1 7F 19 E8 B6 2F 35 7C 00 72 51 31 11 1A 10 06 35

Only PCR:7 differs.
EDIT: the runtime config is different:

 Extracting CBFS file heads/initrd/etc/config.user into /etc/config.user
 TPM: Extending PCR[7] with filename /etc/config.user and then its content
 TPM: Extending PCR[7] with hash a0c02a9f82b98bc70224059ca53af19782ca0b28
-LOG: exec stdout: New value of PCR[7]: 42bef5f461984d203c7abcb8f9d0d5ec5c11747f
-TPM: Extending PCR[7] with hash 936e97235f2076c83dc362d8507604385ac766a9
-LOG: exec stdout: New value of PCR[7]: 88883d8ab62c642ef4d50527960695bd94f0bbe6
+LOG: exec stdout: New value of PCR[7]: 700dea51122158abb7a6b2881dbd7b31462cbe61
+TPM: Extending PCR[7] with hash bb52e739677822a892e4f3cb0417749a83684e13
+LOG: exec stdout: New value of PCR[7]: 4c5334aad32424a1969d4d5f11b202702f86e003

[tlaurion]

@marmarek

Only PCR:7 differs.
EDIT: the runtime config is different:

 Extracting CBFS file heads/initrd/etc/config.user into /etc/config.user
 TPM: Extending PCR[7] with filename /etc/config.user and then its content
 TPM: Extending PCR[7] with hash a0c02a9f82b98bc70224059ca53af19782ca0b28
-LOG: exec stdout: New value of PCR[7]: 42bef5f461984d203c7abcb8f9d0d5ec5c11747f
-TPM: Extending PCR[7] with hash 936e97235f2076c83dc362d8507604385ac766a9
-LOG: exec stdout: New value of PCR[7]: 88883d8ab62c642ef4d50527960695bd94f0bbe6
+LOG: exec stdout: New value of PCR[7]: 700dea51122158abb7a6b2881dbd7b31462cbe61
+TPM: Extending PCR[7] with hash bb52e739677822a892e4f3cb0417749a83684e13
+LOG: exec stdout: New value of PCR[7]: 4c5334aad32424a1969d4d5f11b202702f86e003

I initially missed it.
Can you please upload your /etc/config.user?

Still no clue why PCR4 sealed is not replayed with 0 on your side, which is expected non-recovery, non boot mode value to be unsealed on clean boot. :/

[marmarek]

my config.user:

export CONFIG_TPM2_TOOLS=""
export CONFIG_TPM="y"
export CONFIG_AUTO_BOOT_TIMEOUT=""
export CONFIG_DEBUG_OUTPUT="y"
export CONFIG_ENABLE_FUNCTION_TRACING_OUTPUT="y"
export CONFIG_QUIET_MODE="n"

[marmarek]

With debug enabled, I noticed cbmem SEGV on screen, could it be related?

[marmarek]

cbmem -L output (it crashes before finishing):

cbmem.txt

I think something seriously went wrong there...

[marmarek]

Should I dump flash content to compare against clean version? Or maybe the above cbmem.txt is enough that you could compare if it isn't corrupted flash content (few initial measurements seems to be saved correctly)?

[tlaurion]

Should I dump flash content to compare against clean version? Or maybe the above cbmem.txt is enough that you could compare if it isn't corrupted flash content (few initial measurements seems to be saved correctly)?

Since PCR2 is exactly the same, I take for granted the everything firmware related is the same, including the payload which is extracted and executed, creating our initramfs. I exclude SPI flash chip corruption.

@marmarek will test again. This would mean cbmem/coreboot had a regression when bumped to 24.12 and will probably need a coreboot bug report.