Latest validation layer in SDK 1.4.335.0 gives apparently incorrect error about format of swapchain image that is not even presented, no error given by older SDK 1.3.275.0

[elmar-k]

Environment:

• OS: Ubuntu 22.04
• GPU and driver version: Geforce RTX 4080, 580.95.05
• SDK or header version if building from repo: 1.4.335.0
• Options enabled (synchronization, best practices, etc.): VK_LAYER_KHRONOS_validation

Describe the Issue
I have a swapchain with three images: 0x280000000028, 0x290000000029, 0x2a000000002a
The render loop works fine, but at one point my app runs out of Vulkan memory in the middle of the frame, which is swapchain image 0x2a000000002a in this example. It therefore submits the half-finished command buffer and calls vkDeviceWaitIdle, wanting to allocate new memory after the GPU finished. While it hangs waiting for the device to become idle, the validation layer in a separate thread suddenly raises the error that image 0x2a000000002a is in the wrong format for presentation, even though I triple checked that my app never tried to present this image (since it's half finished now, and it was presented fine three frames ago). This error did not happen in my previous SDK 1.3.275.0

Here is the traceback of my app waiting in vkDeviceWaitIdle:

Thread 1 (Thread 0x7138038deb80 (LWP 24104) "yasara"):

#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38

#1 0x00007137844d9706 in std::__atomic_futex_unsigned_base::_M_futex_wait_until_steady(unsigned int*, unsigned int, bool, std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> >) () from /lib/x86_64-linux-gnu/libstdc++.so.6

#2 0x00007137716f4bad in std::__atomic_futex_unsigned<2147483648u>::_M_load_and_test_until_steady (__ns=..., __s=..., __has_timeout=true, __mo=std::memory_order_acquire, __equal=true, __operand=1, __assumed=, this=) at /usr/include/c++/11/bits/atomic_futex.h:140

#3 std::__atomic_futex_unsigned<2147483648u>::_M_load_and_test_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __atime=..., __mo=std:--Type for more, q to quit, c to continue without paging--

:memory_order_acquire, __equal=true, __operand=1, __assumed=, this=) at /usr/include/c++/11/bits/atomic_futex.h:190

#4 std::__atomic_futex_unsigned<2147483648u>::_M_load_when_equal_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __mo=std::memory_order_acquire, __val=1, this=) at /usr/include/c++/11/bits/atomic_futex.h:270

#5 std::__future_base::_State_baseV2::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__abs=..., this=0x3a6a3680) at /usr/include/c++/11/future:389

#6 std::__basic_future::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__abs=..., this=) at /usr/include/c++/11/future:710

#7 vvl::Queue::Wait (this=0x3805a8c0, loc=..., until_seq=2215, until_seq@entry=18446744073709551615) at /vulkan-sdk/1.4.335.0/source/Vulkan-ValidationLayers/layers/state_tracker/queue_state.cpp:133

#8 0x000071377173bc62 in vvl::DeviceState::PostCallRecordDeviceWaitIdle (this=0x3737a600, device=, record_obj=...) at /usr/include/c++/11/bits/shared_ptr_base.h:1295

#9 0x00007137713ad39a in vulkan_layer_chassis::DeviceWaitIdle (device=0x3748c960) at /vulkan-sdk/1.4.335.0/source/Vulkan-ValidationLayers/layers/vulkan/generated/chassis.cpp:620

#10 0x000000000051ebcc in vk_finish (combuffered=1) at /home/twinset/yasara/inout.c:6523

#11 0x000000000051f137 in vk_allocarea (allocadd=allocadd@entry=0x7ffccd37cd98, offsetadd=offsetadd@entry=0x713785890528, type=type@entry=0, size=4194304, imagealignment=0) at /home/twinset/yasara/inout.c:4784

#12 0x000000000051f997 in vk_alloc (memobj=memobj@entry=0x713785890518, type=type@entry=0, required=required@entry=0x7ffccd37cdf0, imaged=imaged@entry=0) at /home/twinset/yasara/inout.c:4928

Here is the traceback of the separate validation layer thread that complains that image 0x2a000000002a has not been transitioned to VK_IMAGE_LAYOUT_PRESENT_SRC_KHR, even though I didn't try to present this image:

#59 0x0000000000511129 in vk_printdebug (flags=, objtype=, srcobject=, location=, code=0, layerprefix=0x7137720--Type for more, q to quit, c to continue without paging--

892ae "Validation", message=0x7137041892e0 "vkQueuePresentKHR(): pPresentInfo->pSwapchains[0] images passed to present must be in layout VK_IMAGE_LAYOUT_PRESENT_SRC_KHR or VK_IMAGE_LAYOUT_SHARED_PRESENT_KHR but VkImage 0x2a000000002a is in VK_I"..., userdata=0x0) at /home/twinset/yasara/inout.c:3511

#60 0x0000713771be2cd8 in DebugReport::LogMessage (this=, msg_flags=8, vuid_text=..., objects=..., loc=..., main_message="images passed to present must be in layout VK_IMAGE_LAYOUT_PRESENT_SRC_KHR or VK_IMAGE_LAYOUT_SHARED_PRESENT_KHR but VkImage 0x2a000000002a is in VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL.") at /vulkan-sdk/1.4.335.0/source/Vulkan-ValidationLayers/layers/error_message/logging.cpp:252

#61 0x0000713771be38db in DebugReport::LogMessageVaList (this=0x3523c790, msg_flags=msg_flags@entry=8, vuid_text="VUID-VkPresentInfoKHR-pImageIndices-01430", objects=..., loc=..., format=0x713771f22a30 "images passed to present must be in layout VK_IMAGE_LAYOUT_PRESENT_SRC_KHR or VK_IMAGE_LAYOUT_SHARED_PRESENT_KHR but %s is in %s.", argptr=0x713719dffa60) at /vulkan-sdk/1.4.335.0/source/Vulkan-ValidationLayers/layers/error_message/logging.cpp:711

#62 0x0000713770fc2a41 in Logger::LogError (this=this@entry=0x3736e988, vuid_text=..., objlist=..., loc=..., format=format@entry=0x713771f22a30 "images passed to present must be in layout VK_IMAGE_LAYOUT_PRESENT_SRC_KHR or VK_IMAGE_LAYOUT_SHARED_PRESENT_KHR but %s is in %s.") at /vulkan-sdk/1.4.335.0/source/Vulkan-ValidationLayers/layers/./error_message/logging.h:289

#63 0x00007137711eb77e in QueueSubmissionValidator::Validate (this=0x3805ac60, submission=...) at /usr/include/c++/11/bits/char_traits.h:399

#64 0x00007137711e0466 in core::QueueSubState::Retire (this=0x3805ac50, submission=...) at /vulkan-sdk/1.4.335.0/source/Vulkan-ValidationLayers/layers/core_checks/cc_state_tracker.cpp:1180

#65 0x00007137716f4905 in vvl::Queue::Retire (submission=..., this=0x3805a8c0) at /vulkan-sdk/1.4.335.0/source/Vulkan-ValidationLayers/layers/state_tracker/queue_state.cpp:307

#66 vvl::Queue::Retire (this=0x3805a8c0, submission=...) at /vulkan-sdk/1.4.335.0/source/Vulkan-ValidationLayers/layers/state_tracker/queue_state.cpp:297

#67 0x00007137716f5be4 in vvl::Queue::ThreadFunc (this=0x3805a8c0) at /vulkan-sdk/1.4.335.0/source/Vulkan-ValidationLayers/layers/state_tracker/queue_state.cpp:328

#68 0x00007137844dc253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

#69 0x0000713803294ac3 in start_thread (arg=) at ./nptl/pthread_create.c:442

This is the complete validation error:

Vulkan driver error: [Validation] Code 0: vkQueuePresentKHR(): pPresentInfo->pSwapchains[0] images passed to present must be in layout VK_IMAGE_LAYOUT_PRESENT_SRC_KHR or VK_IMAGE_LAYOUT_SHARED_PRESENT_KHR but VkImage 0x2a000000002a is in VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL.

The Vulkan spec states: Each element of pImageIndices must be the index of a presentable image acquired from the swapchain specified by the corresponding element of the pSwapchains array, and the presented image subresource must be in the VK_IMAGE_LAYOUT_PRESENT_SRC_KHR or VK_IMAGE_LAYOUT_SHARED_PRESENT_KHR layout at the time the operation is executed on a VkDevice (https://vulkan.lunarg.com/doc/view/1.4.335.0/linux/antora/spec/latest/chapters/VK_KHR_surface/wsi.html#VUID-VkPresentInfoKHR-pImageIndices-01430)

This error did not happen in my previous SDK 1.3.275.0. I can provide a binary for reproducing the problem..

[elmar-k]

Please note that my post contains two tracebacks, the switch is between traceback lines #12 and #59, the bug report tool removed by separation lines ;-)

[artem-lunarg]

@elmar-k could you clarify the meaning of:

in the middle of the frame, which is swapchain image 0x2a000000002a in this example

It is not clear whether it is about recording command buffer or submit time. If you can provide pseudo code that might help alot (relevant commands in command buffers, submit command, queue present commands).

We also fixed a lot since 1.3.275. It is possible that's a valid behavior of the layer, but it first I need to understand what's going on.

[elmarkrieger]

Here is the pseudo code:

• Call vkAcquireNextImage to get the next free swapchain image, 0x2a000000002a in this example
• Record command buffer for first half of the frame
• Submit command buffer
• Call vkDeviceWaitIdle
• The validation layer complains from a different thread that the format of swapchain image 0x2a000000002a is not right for presentation, even though I didn't call vkQueuePresent.

I saw that you recently added fine-grained locking. It seems that you additionally tried to optimize performance by moving validation to a different thread (as my tracebacks above show), I beg you to undo this or add an option to undo this. The power of the validation layer is that in case of an error, I can simply look at the traceback and follow the call sequence MyApp -> Vulkan function call -> Validation layer -> Error detected -> MyApp error handler to see where things go wrong. If a separate validation thread complains out of nowhere, I am stuck, as in this example.

Many thanks for your help, example app can be provided, source code is too complex.

[artem-lunarg]

@elmarkrieger could you explain the meaning of “first half of the frame” (VVL works with the spec which even does not define a frame concept). If that’s a real issue i need to write a test and need to have a good understanding

[artem-lunarg]

I saw that you recently added fine-grained locking. It seems that you additionally tried to optimize performance by moving validation to a different thread (as my tracebacks above show), I beg you to undo this or add an option to undo this.

It was done few years ago. The additional threads are not only about performance but to solve some correctness issues, so this does not simply go away. I personally also don’t like threading system and considering possibility to rework it in the future.

[elmar-k]

This just means that the command buffer does not contain the commands for an entire frame, but only for part of the frame. This shouldn't matter at all for validation. The real problem is that the validation layer raises an error in a separate thread, and there is no way to know where the error came from, since I cannot get a call stack that bridges threads.

We can of course guess that my call to vkQueueSubmit triggered QueueSubmissionValidator::Validate , but then the question is why it complains about the swapchain image format, since I didn't call vkQueuePresent.

[elmar-k]

I saw that you recently added fine-grained locking. It seems that you additionally tried to optimize performance by moving validation to a different thread (as my tracebacks above show), I beg you to undo this or add an option to undo this.

It was done few years ago. The additional threads are not only about performance but to solve some correctness issues, so this does not simply go away. I personally also don’t like threading system and considering possibility to rework it in the future.

I see. Interestingly, I've been using the LunarG validation layers since 2018, and now with 1.4.335.0 it happened for the first time that I didn't get a direct traceback from a validation error. (I am aware that tracebacks are impossible for complex things like GPU assisted validation, but certainly not for swapchain image format checks ;-)

[artem-lunarg]

The real problem is that the validation layer raises an error in a separate thread, and there is no way to know where the error came from, since I cannot get a call stack that bridges threads.

It is not fundamentally different than it was before. It is submit time validation. The same as before it can’t stop execution at specific point in the command buffer because some validation can happen only when we analyze submitted batch. Previously it was done during QueueSubmit but that does not handle a general case (command buffers can be processed in different order than submit order due to timeline semaphores). with current architecture this validation is done on background queue threads that resolve batch order. I agree it is better to do this as part of QueueSubmit, but this not an easy change (again I am aware and have plans to move in the direction to make it happen).

That was explanation of how it works and the reasons, but please note the error message itself contains the same information as before. I will try to reproduce this scenario during the week and maybe will ask additional questions.

[artem-lunarg]

I see. Interestingly, I've been using the LunarG validation layers since 2018, and now with 1.4.335.0

We did that change around half a year (maybe year) ago. Mostly to make image layout validation to work correctly with timeline semaphores. One option to consider is to have two codepaths of submit time validation for binary and timeline semaphores, but this has to be robust and simple enough (that code is already quite complex). i will do a quick investigation.

[artem-lunarg]

The real problem is that the validation layer raises an error in a separate thread, and there is no way to know where the error came from, since I cannot get a call stack that bridges threads.

I also still dont fully understand if the above behavior is the reason to open this issue? Because if yes, then it is not a validation false positive but just a way some types of submit time validations report the error . Please confirm if you think that the error message itself is s false positive?

[artem-lunarg]

This just means that the command buffer does not contain the commands for an entire frame, but only for part of the frame.

@elmarkrieger There is one issue with the above clarification in the context that we want to reproduce this scenario with a test. Vulkan does not have a concept of the "frame" - it is application concept and there are many ways how the apps define and synchronize the frames. When we get error report we write an internal VVL test that reproduces the issue. In this case it is not clear how to write a test and to differentiate between "full frame" and "partial frame". For example, I can read it as a) command buffer recording was started but not finished, b) command buffer was recorded but was not submitted, c) it was submitted but we did not call QueuePresent after submit, etc. That's why I asked about pseudo code - no need to provide all details, but it has to be specific enough which commands were called, and if something is incomplete/partial then it has to be expressed in the sequence of vulkan commands, so we can reproduce this in the test (one example of pseudo code from the recent issue we fixed #11493 (comment)).

If there is possibility to attach source code/executable we can work with it too (we also have an option to send artifacts privately to LunarG).

[elmar-k]

Sure, I think that the error message is a false positive. Look at the code that creates the error in cc_submit.cpp:

// Check that image being presented has correct layout
// NOTE: Do separate check that swapchain and its images are not destroyed at this point.
//       For example, you can destroy swapchain after it was used as the old swapchain.
if (submission.swapchain && !submission.swapchain_image->Destroyed()) {
    std::vector<VkImageLayout> layouts;
    if (submission.swapchain_image && FindLayouts(*submission.swapchain_image, layouts)) {
        for (auto layout : layouts) {
            if (layout != VK_IMAGE_LAYOUT_PRESENT_SRC_KHR && layout != VK_IMAGE_LAYOUT_SHARED_PRESENT_KHR) {
                core_checks.LogError(
                    "VUID-VkPresentInfoKHR-pImageIndices-01430", submission.swapchain_image->Handle(), submission.loc.Get(),
                    "images passed to present must be in layout VK_IMAGE_LAYOUT_PRESENT_SRC_KHR or "
                    "VK_IMAGE_LAYOUT_SHARED_PRESENT_KHR but %s is in %s.",
                    core_checks.FormatHandle(submission.swapchain_image->Handle()).c_str(), string_VkImageLayout(layout));

Are you 100% sure that this code is only reached via a call to vkQueuePresent and not via a call to vkQueueSubmit? If you are sure, then it means that your code is validating the vkQueuePresent call from three frames ago (two swapchain images have been presented in the mean time). Is it correct that vkQueuePresent validation lags three frames behind? If yes, then verify that you are indeed validating the layout of image 0x2a000000002a as it was three frames ago, when it was last submitted for presentation. When the validation layer raises the error, image 0x2a000000002a is indeed not in presentation layout, so it may be that the validation layer does not correctly look into the past. Ask yourself what changed in this part of the code since SDK 1.3.275.0, which doesn't raise an error at all. The check if the image presented is in presentation format is absolutely standard, this was certainly present from the beginning.