Fix qos-test and updated CI pipeline#50
Conversation
hungmtruong
force-pushed
the
jt-fix-qostest
branch
from
c43fa81 to
ef7ba5e
Compare
ho28
left a comment
ho28
left a comment
There was a problem hiding this comment.
I know you might still have some changes on the way, but left a few comments here for you
| */ | ||
| if (!qtest_enabled()) { | ||
| rc = fdt_pack(fdt); | ||
| /* Should only fail if we've built a corrupted tree */ |
There was a problem hiding this comment.
Thanks for the detailed commit message - but what exactly does "it triggers a BQL deadlock
in the virtio-scsi iothread dataplane stop path" mean, and how is it related to the size of the FDT?
I know you probably don't have the answer right now. If you want to look into and find out, then can you update the commit message and this comment with that info? Otherwise, just drop the specifics and mention that this causes qtest issues for reasons that are unclear to us.
There was a problem hiding this comment.
Used GDB to look into this, and this is what I have found:
The deadlock is between two threads during qos-test:
- main thread: holds BQL, calls virtio_scsi_dataplane_stop(), waits for the iothread
- iothread: completing a SCSI DMA unmap to address 0, needs BQL
Regarding the size of the FDT, I am not sure.
There was a problem hiding this comment.
ok - I ask because the fix appears to be to not pack the DTB for the qtest. I'm fine with that fix. But its not clear to me how the size of the DTB affects synchronization around the Big Qemu Lock (BQL). i'm pushing on it because conceptually the DTB and the virtio scsi data path have nothing to do with each other. if theres a race condition somewhere, it would be nice to know the exact steps that race and how we end up in the deadlock vs the normal functioning path. But like i said before, it's not necessary to find this answer for us to ship this fix.
There was a problem hiding this comment.
I have fixed the commit message.
hungmtruong
force-pushed
the
jt-fix-qostest
branch
2 times, most recently
from
596970b to
1bbb884
Compare
hungmtruong
force-pushed
the
jt-fix-qostest
branch
4 times, most recently
from
2adc767 to
953d575
Compare
fdt_pack() causes the aarch64 qos-test (iothread-attach-node) to hang indefinitely. The exact mechanism of how fdt_pack triggers the hang is unclear, but skipping it in qtest mode resolves the issue. Signed-off-by: Jason Truong <jason.truong@windriver.com>
Signed-off-by: Jason Truong <jason.truong@windriver.com>
Add --disable-vhost-net --disable-vhost-user to configure since vhost tests require kernel support unavailable in CI containers. Signed-off-by: Jason Truong <jason.truong@windriver.com>
hungmtruong
force-pushed
the
jt-fix-qostest
branch
from
953d575 to
0575c7d
Compare
2 participants
Summary
Fix the aarch64 qos-test iothread hang caused by a BQL deadlock, and update CI to work reliably in containerized environments.
Root Cause
The qos-test (iothread-attach-node) was hanging in CI because of a deadlock: the main thread holds BQL and waits for the iothread to stop, but the iothread needs BQL to finish a DMA write. Both wait forever, and then timeout.
Fix
Guard
fdt_pack()with!qtest_enabled()so it only runs during real kernel boot where DTB size optimization matters, not during qtest where it triggers the deadlock. Real VxWorks/Linux boot behavior is unchanged.Changes
fdt_pack()in qtest mode--disable-vhost-net --disable-vhost-user(vhost tests need kernel support unavailable in CI)
Testing