Optimize FEL USB performance#234
Conversation
jameshilliard
force-pushed
the
toc0-boot
branch
3 times, most recently
from
2f16236 to
c7aa218
Compare
jameshilliard
changed the title
jameshilliard
force-pushed
the
toc0-boot
branch
10 times, most recently
from
f712eac to
df2c0b6
Compare
jameshilliard
changed the title
jameshilliard
force-pushed
the
toc0-boot
branch
4 times, most recently
from
12a1b8d to
710b55a
Compare
| uint32_t rvbar_reg_alt;/* alternative MMIO address of RVBARADDR0_L register */ | ||
| uint32_t ver_reg; /* MMIO address of "Version Register" */ | ||
| uint32_t usb_musb_base;/* base address of the USB OTG controller */ | ||
| uint32_t fel_endpoint_table_addr; /* BROM FEL endpoint table */ |
There was a problem hiding this comment.
Is it possible to read this address from the MUSB controller at runtime, instead of storing it per SoC?
There was a problem hiding this comment.
I don't think this can be read from the MUSB controller itself.
fel_endpoint_table_addr is not a MUSB register address. It points to the
Boot-ROM's in-SRAM FEL endpoint state table. The MUSB registers expose the
hardware controller and endpoint state, but they do not provide a pointer back
to the BROM's software endpoint descriptors.
The table layout seen in the supported BROMs is:
endpoint_table + 0x10: first FEL bulk endpoint entry
endpoint_table + 0x20: second FEL bulk endpoint entry
entry + 0x00: pointer to endpoint object, or zero
entry + 0x08: cached maxpacket value in the table entry
object + 0x04: cached maxpacket value in the endpoint object
The high-speed thunk patches those cached maxpacket fields to 512 before the
USB reconnect. This keeps the BROM FEL command loop's endpoint bookkeeping in
sync with the high-speed bulk packet size. Without that, the controller can be
switched to high-speed while the BROM's software state still describes 64-byte
full-speed bulk packets.
jameshilliard
force-pushed
the
toc0-boot
branch
2 times, most recently
from
0cf057c to
498c80f
Compare
Separate the runtime SMC-workaround probe from the code that executes the workaround so later SoCs can choose a different implementation without mixing that change into the existing direct-SMC path. Describe the workaround method in SoC data and make every existing secure-FEL user select the direct-SMC method explicitly. This preserves the current behaviour because direct SMC remains the only implementation in this patch. Also make the old-format thunk header rule pattern-based while it still only builds the existing SPL thunk header. Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
On H616 with the secure boot fuse set, FEL starts in non-secure state. The older direct SMC workaround is not sufficient there because SMC enters monitor mode instead of directly leaving the BROM FEL command loop in secure SVC state. Add a secure-SVC SMC thunk for this case and keep the existing global startup workaround model. The thunk preserves the BROM SRAM workspace using the same swap-table convention as the SPL thunk, installs a temporary monitor-mode SMC handler by patching only the SMC vector word, then issues SMC. The temporary handler restores the original vector word, clears SCR.NS, clears MVBAR, restores the secure GIC view expected by the BROM, copies the saved SVC SP/LR into the secure bank, switches to secure SVC, and returns to the FEL command loop. After the transition, the normal runtime probe sees secure state and suppresses repeat application in that sunxi-fel process, so normal SID reads and SPL execution use the existing code paths. H616 selects the secure-SVC SMC thunk path and gates the workaround on a non-zero secure boot status word at SID base + 0xa0. The zero-word runtime probe still has to match before the thunk is applied, so non-secure H616 boards and already-transitioned secure-FEL sessions do not enter the secure path just because one of those checks matches. Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Secure boot sunxi images use TOC0 containers instead of a bare eGON SPL header. Teach the spl/uboot command path to recognize a TOC0 image, validate its header and checksum, extract the firmware item, and wrap it in a synthetic eGON SPL header at the SoC SPL load address. After the wrapped SPL returns to FEL, pass the payload appended after the TOC0 container to the existing U-Boot image loader so FIT-based u-boot-sunxi-with-spl.bin images can be loaded by the normal uboot command path. Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
|
My output with the latest push made 23 minutes ago: |
jameshilliard
force-pushed
the
toc0-boot
branch
2 times, most recently
from
2cfe059 to
e6b3d27
Compare
|
Latest test on Orange Pi Zero 3 ran in near instantaneous time: |
|
Thanks for this PR! On the T113s3, the The upload time went from 52 to just 2 seconds. (sunxi-fel is loading the main uboot binary twice because uboot creates a FIT with uboot as |
I don't think this is true, can you run this command with sunxi-fel from master without any changes/patches?: state=$(sunxi-fel readl 0x00047d08); table=$(sunxi-fel readl $((state + 0x24))); printf 'state=0x%08x table=0x%08x bulk_ep_addr=0x%08x fel_endpoint_table_addr=0x%08x\n' "$state" "$table" "$((table + 0x10))" "$table" |
Sure I cannot reproduce the dynamic table pointer but I still have some logs from my experimentation that show state/table with a different address |
|
@CodeInPolish I forced pushed some changes adding T113s3 support, let me know if that works on your board. |
|
If that doesn't work try running this command(from master immediately after a hard power cycle): |
|
@CodeInPolish can you also dump the BROM using master from a cold boot? I think something like this: |
Teach sunxi-fel to switch supported MUSB FEL sessions to high-speed USB. The USB speed switch disconnects the device, so run the BROM endpoint maxpacket update and MUSB POWER write from a small device-side thunk. Do not rely on ordinary FEL commands after the disconnect starts. Treat the expected disconnect during that EXEC path as success. Add SID-based device selection and reuse it for re-enumeration tracking. This lets command sequences reopen the same FEL device after the speed switch, avoids accidentally continuing on another attached FEL device, and tolerates transient libusb open and version-probe failures while the device is reappearing. Keep --no-high-speed as an escape hatch for the old full-speed path. Apply the secure-state workaround after the optional speed switch, so it runs once on the final FEL handle. This still runs before commands on --no-high-speed, already-high-speed, and unsupported high-speed paths. Before requesting an RMR warm reset, clear the USB soft-connect bit when the USB controller is known and mark the handle disconnected so libusb cleanup does not wait on a device that intentionally left FEL. Provide USB controller base and BROM FEL state-pointer addresses for SoCs whose BROM dumps show the compatible MUSB FEL endpoint layout. The high-speed thunk follows the BROM state pointer to find the live endpoint table before patching the two bulk endpoint maxpacket values. Enable this on A33, A64, H3, H5, A63, H6, H616, V853, R528/T113-S3/T528, V5, A523 and A133. Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Hmm, yeah, I think you're right, I pushed a new revision that dynamically calculates the location in the high speed switch thunk. I updated all SoC's to use this new approach as well based on static BROM analysis. |
Dump source is [1]. Link: linux-sunxi/sunxi-tools#234 (comment) [1]
This is strange, in the current code I've kept it at your tested value but would be nice to know at what size it starts failing. If you can also paste the output of this script run from master on a cold boot. |
|
I retested based on commit I also tested with Do we have any idea why the state pointer seems to be increasing?
There you go: And thanks again for this PR! It really is a huge performance gain |
I pushed some more optimizations just now, could you test that the latest commit works as well?
The current evidence points to this not being extra BootROM state allocation. In the T113/R528 BROM, That explains the observed movement: the global pointer is stable, but the stack-local state object can move if the BROM enters FEL with a slightly different stack pointer or call path. The endpoint table moves with it. In all reported values so far, the relative layout stays consistent: state - endpoint_table = 0xa4 Examples: 0x00044f44 - 0x00044ea0 = 0xa4 So this looks like stack placement variance, not the BootROM growing some persistent storage area over time. The right thing for sunxi-fel is to keep the fixed pointer-slot address in the SoC table and resolve the live endpoint table through it at runtime, instead of hard-coding one observed endpoint table address. |
Still works on my side with commit
That makes sense! Thanks |
The newer BROM FEL path copies received USB data byte-by-byte from the MUSB FIFO. Even after switching the controller to high-speed mode, this leaves large post-SPL DRAM writes far slower than the USB link can support. After SPL has initialized DRAM, install a parameterized RX DMA thunk for writes into the known DRAM window. The thunk builds the MMU remap it needs, hooks the BROM RX FIFO copy helper, and replaces it with a MUSB/USBC endpoint-DMA receive path. It routes the active RX endpoint DRQ through VEND0 before starting the internal DMA channel, so the FEL wire protocol stays unchanged. Put the common thunk code in fel_lib.c and install it from the shared aw_fel_write_buffer() path. This makes normal writes and FIT image loading use the same post-SPL DRAM write preparation path. Put the SoC-specific hook, translation-table, shadow-page, endpoint-state and GIC parameters in the SoC table. Enable the path for SoCs whose BROM dumps contain the same FIFO-copy helper ABI and compatible MUSB register layout: A33, A64, H3, H5, A63, H6, H616, V853, R528/T113-S3/T528, V5, A523 and A133. Coalesce full high-speed packets into bounded RX DMA requests. Keep each host write as a single AW_FEL_1_WRITE request, and have the thunk copy the final short packet with PIO after stopping DMA. This keeps arbitrary post-SPL DRAM write sizes working without changing the FEL wire protocol. Wait for DMA completion with a temporary IRQ vector and WFI instead of busy-waiting for the entire transfer. The thunk temporarily drops the BROM USB IRQ active state, accepts either the DMA completion interrupt or SOF as the wake source, and restores the BROM interrupt state before returning. The SoC table supplies either GICv2 CPU-interface state or GICv3 distributor state as needed. Use the largest 512-byte-aligned DMA request size below the observed 18-bit byte-count wrap point. This avoids the 0x40000 wrap while reducing per-request setup overhead. The R528/T113-S3/T528 reports confirmed that this cap is stable there too. Add the checked-in DMA thunk header to the sunxi-fel prerequisites and thunk documentation. This makes the binary rebuild when the generated header changes without making the sunxi-fel target regenerate thunk headers. Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Use a small queue of asynchronous libusb bulk OUT transfers for the normal write path. This keeps the host controller fed during fast high-speed RX-DMA writes without relying on large individual transfers, while still keeping the disconnect-prone re-enumeration paths on synchronous transfers. Use 16 KiB transfers with a queue depth of four. USB tracing on H616 shows this keeps the queue full during the main FIT write and reaches the same throughput plateau as deeper queues, while avoiding excessive in-flight allocation. Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
|
@CodeInPolish I pushed some more refactoring, let me know if that still works on your board. |
4 participants
Implementation details: README.FEL-SPEED.md
With optimizations:
Without optimizations: