Ovmf memory debug logging4

[ajyoung-oracle]

Description

OVMF Memory Debug Logging Support

Overview:

This PR adds support to log OVMF debug messages to a memory buffer. The memory buffer can be extracted from a VM's QEMU process (or a core file) to debug issues.

Background:

OVMF currently offers 2 methods to collect debug messages:

• debugcon - i.e. via the debug IO port and logged by QEMU to a host disk file
• the serial port/console (-D DEBUG_ON_SERIAL_PORT).

Both of these methods are impractical under normal VM operation. They increase VM boot time, clutter the serial console, and use host disk resources. In contrast, logging the debug messages to memory is fast and takes no disk resources allowing it to be enabled by default for customer environments. This allows for "first-time" issue analysis, negating the need to reproduce issues with debug messages enabled, which makes for a much better customer experience should an issue occur.

Code Overview:

The code introduces a new PEIM:

• OvmfPkg/MemDebugLogPei: When memory becomes available, this PEIM allocates the main memory debug log buffer and installs a HOB containing the address of the memory log buffer.

The code also introduces 1 new library:

• OvmfPkg/Library/MemDebugLogLib: This library implements the key MemDebugLogWrite() function (called by DebugLib). There is a different implementation of MemDebugLogWrite() for the different module types (i.e. SEC, PEIM, DXE, Runtime). See more detail below. This library also contains core functions to manage access to the memory debug log circular buffer (these "common" functions are used by both this library and the new MemDebugLogPei PEIM)

A bit of complication arises from the the fact that main memory is not available until the PEI phase and several debug messages are logged before then. To remedy this, the code makes use of a "early" memory buffer (taken from the top of PeiMemFvBase) which is used to log the initial (pre-memory) messages. Once memory becomes available, the PEIM allocates the main memory debug log buffer, copies the messages from the early buffer and installs a HOB containing the main memory buffer address.

Thus, the SEC version of MemDebugLogLib:MemDebugLogWrite() writes debug messages to the "early" debug log buffer. The PeiCore and PEIM version of MemDebugLogLib:MemDebugLogWrite() also writes initial messages to the early buffer until the MemDebugLogPei HOB is installed. Once the HOB is installed it switches to write to the main memory buffer (obtaining the main memory buffer address from the HOB). The DXE and runtime versions of the MemDebugLogLib:MemDebugLogWrite() write to the main memory debug log buffer (by also using the HOB). The runtime version of MemDebugLogLib:MemDebugLogWrite() (used by DXE_RUNTIME_DRIVER type drivers) will allow memory debug log writes until the OS makes the SetVirtualAddressMap() BS call - at which point memory debug logging ends.

The OvmfPkg/Library/PlatformDebugLibIoPort library was modified to also write debug messages to memory. This allows both writes to debugcon and memory simultaneously.

Putting it all together, when debugcon is configured in the build (the default), the OVMF debug message call sequence is:

• OVMF code makes DEBUG() call - which ends up calling DebugPrintMarker() from OvmfPkg/Library/PlatformDebugLibIoPort/DebugLib.c
• DebugPrintMarker() was modified to also call MemDebugLogWrite()
• MemDebugLogWrite() (from OvmfPkg/Library/MemDebugLogLib) obtains the address of either the "early" memory log buffer or the main memory log buffer - depending on the type of module (SEC, PEI, DXE, etc) - and calls MemDebugLogWriteBuffer()
• MemDebugLogWriteBuffer() handles the details of writing the debug message to the circular memory log buffer.

The feature (i.e. the new MemDebugLogLib library and PEIM) are disabled by default and are enabled via the new "DEBUG_TO_MEM" build flag (which can be enabled on the build command line - similar to DEBUG_ON_SERIAL_PORT).

Notes:

The debug log memory buffer size can be configured via FwCfg (-fw_cfg name=opt/ovmf/MemDebugLogPages,string=n - where 'n' denotes the number of pages) and is circular - i.e. only the most recent debug messages are retained if the memory buffer overflows. This is appropriate as typically only the most recent debug messages are relevant when debugging an issue. The code currently defaults to a 32 page (128K) sized memory debug log buffer (the default is configured via a PCD). The maximum size that can be specified via FwCfg is 2MB. The feature can be disabled by specifying 0 via FwCfg.

It is straight-forward to implement a host-side tool/utility to search the VM's QEMU memory regions to locate the OVMF memory debug log buffer (located on a page boundary), decipher the buffer header and display the circular buffer contents (debug messages). We (Oracle) already have such a utility which can extract the OVMF memory debug log from a live QEMU process or a QEMU core file.

This feature doesn't work with Confidential Computing VMs as the guest memory is encrypted and thus not readable from the host. A future enhancement could possibly be made to mark the OVMF memory debug log buffer as unencrypted (?) TBD

This PR only covers OVMF (x86_64) but was written to be easily extended to AAVMF (Arm) in a future PR. TBD

• [x ] Breaking change?
  • Breaking change - Does this PR cause a break in build or boot behavior?
  • Examples: Does it add a new library class or move a module to a different repo.
• Impacts security?
  • Security - Does this PR have a direct security impact?
  • Examples: Crypto algorithm change or buffer overflow fix.
• Includes tests?
  • Tests - Does this PR include any explicit test code?
  • Examples: Unit tests or integration tests.

How This Was Tested

OVMF builds for both OvmfPkgIa32X64.dsc and OvmfPkgX64.dsc (both with and without -D DEBUG_TO_MEM) were built/tested and the OVMF Memory Debug Log extracted from the VM's QEMU process (by a custom utility) and assessed for correctness.

Integration Instructions

N/A

[tianocore-assign-reviewers]

⚠ WARNING: Cannot add some reviewers: A user specified as a reviewer for this PR is not a collaborator of the repository. Please add them as a collaborator to the repository so they can be requested in the future.

Non-collaborators requested:

• @CorvinK

Attn Admins:

• @jljusten
• @ardbiesheuvel
• @lgao4
• @mdkinney
• @makubacki
• @jkmathews
• @miki-intel-work

---

Admin Instructions:

• Add the non-collaborators as collaborators to the appropriate team(s) listed in teams
• If they are no longer needed as reviewers, remove them from Maintainers.txt

[kraxel]

FYI: you can simply force-push to your branch, there is no need to open a new PR for each revision of the patch set.

[ardbiesheuvel]

Please don't create yet another PR for this series.

[ajyoung-oracle]

Please don't create yet another PR for this series.

Sorry, just saw this. #11031 will be the last PR from me on this series.

[ardbiesheuvel]

No, please use this PR instead.

[ardbiesheuvel]

Unfortunately, I don't believe this is possible. Last week, when attempting to rebase my commits, I hosed up my branch and could not figure out how to fix it.

I'm sure you'll figure it out.

[ajyoung-oracle]

It appears that PRs cannot be reopened after the branch has been force-pushed or recreated (without some extreme git hackery)...

[ardbiesheuvel]

Just do

git push -f [email protected]:ajyoung-oracle/edk2 59f80a8973071e6832fa6a2f62bcf18a36d68a48:OVMF_Memory_Debug_Logging4

[ajyoung-oracle]

Just do

git push -f [email protected]:ajyoung-oracle/edk2 59f80a8973071e6832fa6a2f62bcf18a36d68a48:OVMF_Memory_Debug_Logging4

Doesn't work:

$ git push -f origin 59f80a8973071e6832fa6a2f62bcf18a36d68a48:OVMF_Memory_Debug_Logging4
Username for 'https://github.com': ajyoung-oracle
Password for 'https://[email protected]': 
fatal: bad object 59f80a8973071e6832fa6a2f62bcf18a36d68a48
fatal: the remote end hung up unexpectedly
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly
error: failed to push some refs to 'https://github.com/ajyoung-oracle/edk2.git'

Commit 59f80a8 got vaporized when I deleted/recreated the branch last week

[ardbiesheuvel]

Just do

git push -f [email protected]:ajyoung-oracle/edk2 59f80a8973071e6832fa6a2f62bcf18a36d68a48:OVMF_Memory_Debug_Logging4

Doesn't work:

$ git push -f origin 59f80a8973071e6832fa6a2f62bcf18a36d68a48:OVMF_Memory_Debug_Logging4
Username for 'https://github.com': ajyoung-oracle
Password for 'https://[email protected]': 
fatal: bad object 59f80a8973071e6832fa6a2f62bcf18a36d68a48
fatal: the remote end hung up unexpectedly
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly
error: failed to push some refs to 'https://github.com/ajyoung-oracle/edk2.git'

Commit 59f80a8 got vaporized when I deleted/recreated the branch last week

That is not the reason. You are using the wrong protocol. Did you try using the URL I suggested rather than origin? You will need SSH access configured, though.

Misread the log, apologies. Please fetch the commit from tianocore/edk2 first.

[ardbiesheuvel]

You can fetch 59f80a8973071e6832fa6a2f62bcf18a36d68a48 from tianocore/edk2 btw

[ajyoung-oracle]

You can fetch 59f80a8973071e6832fa6a2f62bcf18a36d68a48 from tianocore/edk2 btw

I think this is good to go now. Thanks for the assistance.

[kraxel]

Adding MemDebugLogBufPages to TotalPages in GetPeiMemoryCap() is missing still.

Adding a MemDebugLogEnabled() -> BOOLEAN function to the libary API is a good idea I think. With that it is possible to skip the debug string formating in case if is not needed (i.e. both debugcon and memory logging are turned off).

[ajyoung-oracle]

Adding MemDebugLogBufPages to TotalPages in GetPeiMemoryCap() is missing still.

It's part of fb584b5

Let me know if that is not sufficient.

Adding a MemDebugLogEnabled() -> BOOLEAN function to the libary API is a good idea I think. With that it is possible to skip the debug string formating in case if is not needed (i.e. both debugcon and memory logging are turned off).

Good idea. I added the MemDebugLogEnabled() and checks in PlatformDebugLibIoPort/DebugLib.c to skip the debug string formatting.