Reproducible INITIAL_READ single-bit errors on RTX 3090 at recurring address suffix, while CUDA controls pass

[Olari-A]

Reproducible INITIAL_READ single-bit errors on RTX 3090 at recurring address suffix, while CUDA controls pass

Summary

memtest_vulkan v0.5.0 repeatedly reports INITIAL_READ errors with exactly 20 GiB allocated on an NVIDIA GeForce RTX 3090. The failures reproduce at stock clocks, with memory locked one supported step lower, and with both the lower memory clock and a reduced graphics clock. Across all runs, the reported address ranges have different prefixes but the same B8D8..B8DB suffix, and the bit-level table consistently reports one single-bit index and one toggle.

In contrast, a comprehensive 20 GiB CUDA memory pass (tests 0–8 and 10) completed with zero errors, and a 600-second Tensor-Core gpu-burn run using about 20.2 GiB completed with zero calculation errors. No NVIDIA Xid was recorded in any of these runs.

Environment

• GPU: NVIDIA GeForce RTX 3090, PCI device ID 10de:2204, Ampere
• VBIOS: 94.02.26.88.3C
• NVIDIA driver: 595.84 (open kernel module)
• Driver-reported CUDA compatibility: 13.2
• Vulkan loader/instance: 1.4.341
• Selected device Vulkan API: 1.4.329
• OS: Ubuntu 26.04 LTS (Resolute Raccoon), x86_64
• Kernel: 7.0.0-29-generic
• Vulkan ICD: NVIDIA ICD explicitly selected

v0.5.0 provenance

• Official Desktop Linux x86_64 release archive SHA-256: 0e058c28b9d0d6eb04f5edeebce73e8106f88c4ea307d46e98c715da61117d52
• Extracted executable SHA-256: eb9d518170e43700e97abc5df4f14d3bfc31c79366ea0c311fd5581192664760
• A fresh extraction was verified byte-for-byte identical to the executable used for the final run.

Vulkan reproductions

All runs selected NVIDIA GeForce RTX 3090, DevId=0x2204, allocated exactly 21474836480 bytes (20.0GB as printed by the tool), used the NVIDIA ICD, and had no Xid.

Run	Loaded clock configuration	Runtime / stop	Error observations
Stock reproduction	Memory 9751 MHz; graphics observed 1650 MHz	2 s; SIGINT after error	Iteration 6, 0x18B72B8D8..=0x18B72B8DB, value 0x1179E5AF
Reduced memory clock	Memory locked and verified at 9501 MHz; graphics 1620–1680 MHz	5 s; SIGINT after errors	Iterations 63 and 86, 0x18942B8D8..=0x18942B8DB, values 0x40EF713A, 0x3B10712B
Reduced memory and graphics clocks	Memory 9501 MHz and graphics 1200 MHz, both verified under load	4 s; SIGINT after error	Iteration 36, 0x18711B8D8..=0x18711B8DB, value 0xB430049B
Corrected final stock run	Memory 9751 MHz; graphics 1485–1665 MHz	37.147 s; SIGINT when buffered error output became visible	13 records through iteration 775, all at 0x1ABC9B8D8..=0x1ABC9B8DB

The repeated low address suffix is therefore B8D8 (range ending B8DB) despite the differing high prefixes. Lowering the memory clock by 250 MHz and separately limiting graphics to 1200 MHz did not eliminate the report. Every record is mode INITIAL_READ, reports 0x1 error out of 0x34000000 (0.00000011%), has FFFFFFFF-like count:0, and shows one SinglIdx entry plus TogglCnt of one.

The final run exposed an output-buffering complication: multiple complete records accumulated before they became visible to the external supervisor. SIGINT was sent immediately when the buffered batch appeared.

Complete final-run error table

All rows use address range 0x1ABC9B8D8..=0x1ABC9B8DB and mode INITIAL_READ.

Iteration	Tool values range	1sInValu bucket
13	0x500F09FA..=0x500F09FA	0xE
79	0x207E2C0E..=0x207E2C0E	0xD
97	0x40B4B80A..=0x40B4B80A	0xB
103	0xC5B2141A..=0xC5B2141A	0xD
110	0xB09D2E0B..=0xB09D2E0B	0xF
148	0x07450C3B..=0x07450C3B	0xD
241	0x66D0120B..=0x66D0120B	0xC
271	0x13E385AA..=0x13E385AA	0xF
360	0x232E7C2A..=0x232E7C2A	0xF
504	0x7475012A..=0x7475012A	0xD
549	0x519E12AA..=0x519E12AA	0xE
767	0x32E5833A..=0x32E5833A	0xF
775	0x3658B6AB..=0x3658B6AB	0x11

The tool does not label separate expected and actual values in these records; it emits only values range, so I have preserved that field without reinterpreting it.

CUDA comparison

• cuda_memtest clean pinned source commit: e94e1ee54e0689c9f154a8a08202554452552ec6
• CUDA 13.0, compiled for sm_86
• Exactly 20 GiB allocated
• Tests 0–8 and 10 enabled; Test9 disabled
• One pass, externally measured 460.025 seconds
• Exit code 0, zero reported errors, zero Xid
• Peak temperature 66 C; peak VRAM use 21645 MiB

gpu-burn also completed a 600-second Tensor-Core run at 90% available memory (20714 MB used by the test) with exit code 0, GPU 0: OK, zero calculation errors, and zero Xid. Peak temperature was 79 C.

Question

Do these results look like a real hardware error, or could the repeated B8D8 suffix be caused by the Vulkan shader or NVIDIA driver? The same card passes the complete CUDA memory test and gpu-burn, so what would you test next to separate those possibilities? I can run a modified build if that would help.

[galkinvv]

This looks like a hardware error. memtest_vulkan is not known to produce false-positives (except obvious cases when it crashes immediately due to some shader implementation errors).

Reporting errors on same adderss ending is a sign of a hardware error on a single address within a specific VRAM IC.

Most prpobably its only single bit. To analyze this better - make a similar table with buckets for TogglCnt and SingleIdx buckets based on original log. TogglCnt is excpected to have all values in bucket 0x1, SingleIdx - in a single bucket. And please attach the full log.

For most GPU applications such 1-bit-from-20GB rare hardware error can be just ignored. Say "Games crashing once a year". For AIs inference - that would be "0.0001% more chance that it would be wrong". Tasks likes physics simulation maybe the kind of tasks when this is not acceptable/

The values-related table is not so useful (it becomes useful with another kind of hardware errors).

[galkinvv]

@Olari-A
Moved this to card-specific discussions, since its card problem not software problem.

Most other memtests use more predictable address access patterns, memtest_vulan is more intensive in this sense, so it found more rare errors

[Olari-A]

Thanks — that explains why the other tests missed it.

I had Codex go back through the original capture and aggregate every complete error record from the corrected 20 GiB stock-clock run. There are 13 in total:

• SinglIdx[0x3] = 13; every other SinglIdx bucket is 0.
• TogglCnt[0x1] = 13; every other TogglCnt bucket is 0.

So all 13 reports are single-bit errors, and all point to bit index 3. The complete original capture is included below. The normalized copy was byte-for-byte identical.

Does that narrow it down to a particular VRAM chip, or only confirm that the same data bit is failing?

Complete original memtest_vulkan log

Verbose feature enabled (or 'verbose' found in name). Vulkan instance 1.4.341
Available: 
VK_LAYER_NV_optimus, VK_LAYER_NV_present, VK_LAYER_MESA_anti_lag, VK_LAYER_MESA_device_select, VK_LAYER_MESA_overlay, VK_LAYER_MESA_screenshot, VK_LAYER_INTEL_nullhw
Extensions: VK_KHR_device_group_creation, VK_KHR_display, VK_KHR_external_fence_capabilities, VK_KHR_external_memory_capabilities, VK_KHR_external_semaphore_capabilities, VK_KHR_get_display_properties2, VK_KHR_get_physical_device_properties2, VK_KHR_get_surface_capabilities2, VK_KHR_surface, VK_KHR_surface_maintenance1, VK_KHR_surface_protected_capabilities, VK_KHR_wayland_surface, VK_KHR_xcb_surface, VK_KHR_xlib_surface, VK_EXT_acquire_drm_display, VK_EXT_acquire_xlib_display, VK_EXT_debug_report, VK_EXT_debug_utils, VK_EXT_direct_mode_display, VK_EXT_display_surface_counter, VK_EXT_surface_maintenance1, VK_EXT_swapchain_colorspace, VK_NV_display_stereo, VK_KHR_portability_enumeration, VK_LUNARG_direct_driver_loading, VK_EXT_layer_settings

[Vulkan Loader] ERROR:          loader_validate_layers: Layer 0 does not exist in the list of available layers
Not using validation layers due to ERROR_LAYER_NOT_PRESENT while getting erupt::generated::InstanceLoader in context instance with validation
Loading memory info for selected device index 0...
 1 MemoryType { property_flags: DEVICE_LOCAL, heap_index: 0 } 
 2 MemoryType { property_flags: DEVICE_LOCAL, heap_index: 0 } 
 3 MemoryType { property_flags: HOST_VISIBLE | HOST_COHERENT, heap_index: 1 } 
 4 MemoryType { property_flags: HOST_VISIBLE | HOST_COHERENT | HOST_CACHED, heap_index: 1 } 
 5 MemoryType { property_flags: DEVICE_LOCAL | HOST_VISIBLE | HOST_COHERENT, heap_index: 2 } 
CoherentIO memory          type 5 inside heap MemoryHeap { size: 257949696, flags: DEVICE_LOCAL }
Trying  20.000GB buffer...
Test memory size  20.0GB   type  1: MemoryType { property_flags: DEVICE_LOCAL, heap_index: 0 } MemoryHeap { size: 25769803776, flags: DEVICE_LOCAL }
Standard 5-minute test of 1: Bus=0x2D:00 DevId=0x2204 API 1.4.329  v595(0x94D50000)  24GB NVIDIA GeForce RTX 3090
      1 iteration. Passed  0.0459 seconds  written:   16.2GB 776.8GB/sec        checked:   19.5GB 782.1GB/sec
Error found. Mode INITIAL_READ, total errors 0x1 out of 0x34000000 (0.00000011%)
Errors address range: 0x1ABC9B8D8..=0x1ABC9B8DB  iteration:13
values range: 0x500F09FA..=0x500F09FA   FFFFFFFF-like count:0    bit-level stats table:
         0x0 0x1  0x2 0x3| 0x4 0x5  0x6 0x7| 0x8 0x9  0xA 0xB| 0xC 0xD  0xE 0xF
SinglIdx                1|                 |                 |                 
TogglCnt       1         |                 |                 |                 
1sInValu                 |                 |                 |            1    

     23 iteration. Passed  1.0022 seconds  written:  357.5GB 773.8GB/sec        checked:  429.0GB 794.1GB/sec
Error found. Mode INITIAL_READ, total errors 0x1 out of 0x34000000 (0.00000011%)
Errors address range: 0x1ABC9B8D8..=0x1ABC9B8DB  iteration:79
values range: 0x207E2C0E..=0x207E2C0E   FFFFFFFF-like count:0    bit-level stats table:
         0x0 0x1  0x2 0x3| 0x4 0x5  0x6 0x7| 0x8 0x9  0xA 0xB| 0xC 0xD  0xE 0xF
SinglIdx                1|                 |                 |                 
TogglCnt       1         |                 |                 |                 
1sInValu                 |                 |                 |       1         

Error found. Mode INITIAL_READ, total errors 0x1 out of 0x34000000 (0.00000011%)
Errors address range: 0x1ABC9B8D8..=0x1ABC9B8DB  iteration:97
values range: 0x40B4B80A..=0x40B4B80A   FFFFFFFF-like count:0    bit-level stats table:
         0x0 0x1  0x2 0x3| 0x4 0x5  0x6 0x7| 0x8 0x9  0xA 0xB| 0xC 0xD  0xE 0xF
SinglIdx                1|                 |                 |                 
TogglCnt       1         |                 |                 |                 
1sInValu                 |                 |                1|                 

Error found. Mode INITIAL_READ, total errors 0x1 out of 0x34000000 (0.00000011%)
Errors address range: 0x1ABC9B8D8..=0x1ABC9B8DB  iteration:103
values range: 0xC5B2141A..=0xC5B2141A   FFFFFFFF-like count:0    bit-level stats table:
         0x0 0x1  0x2 0x3| 0x4 0x5  0x6 0x7| 0x8 0x9  0xA 0xB| 0xC 0xD  0xE 0xF
SinglIdx                1|                 |                 |                 
TogglCnt       1         |                 |                 |                 
1sInValu                 |                 |                 |       1         

Error found. Mode INITIAL_READ, total errors 0x1 out of 0x34000000 (0.00000011%)
Errors address range: 0x1ABC9B8D8..=0x1ABC9B8DB  iteration:110
values range: 0xB09D2E0B..=0xB09D2E0B   FFFFFFFF-like count:0    bit-level stats table:
         0x0 0x1  0x2 0x3| 0x4 0x5  0x6 0x7| 0x8 0x9  0xA 0xB| 0xC 0xD  0xE 0xF
SinglIdx                1|                 |                 |                 
TogglCnt       1         |                 |                 |                 
1sInValu                 |                 |                 |                1

    133 iteration. Passed  5.0146 seconds  written: 1787.5GB 772.8GB/sec        checked: 2145.0GB 794.0GB/sec
Error found. Mode INITIAL_READ, total errors 0x1 out of 0x34000000 (0.00000011%)
Errors address range: 0x1ABC9B8D8..=0x1ABC9B8DB  iteration:148
values range: 0x07450C3B..=0x07450C3B   FFFFFFFF-like count:0    bit-level stats table:
         0x0 0x1  0x2 0x3| 0x4 0x5  0x6 0x7| 0x8 0x9  0xA 0xB| 0xC 0xD  0xE 0xF
SinglIdx                1|                 |                 |                 
TogglCnt       1         |                 |                 |                 
1sInValu                 |                 |                 |       1         

Error found. Mode INITIAL_READ, total errors 0x1 out of 0x34000000 (0.00000011%)
Errors address range: 0x1ABC9B8D8..=0x1ABC9B8DB  iteration:241
values range: 0x66D0120B..=0x66D0120B   FFFFFFFF-like count:0    bit-level stats table:
         0x0 0x1  0x2 0x3| 0x4 0x5  0x6 0x7| 0x8 0x9  0xA 0xB| 0xC 0xD  0xE 0xF
SinglIdx                1|                 |                 |                 
TogglCnt       1         |                 |                 |                 
1sInValu                 |                 |                 |   1             

Error found. Mode INITIAL_READ, total errors 0x1 out of 0x34000000 (0.00000011%)
Errors address range: 0x1ABC9B8D8..=0x1ABC9B8DB  iteration:271
values range: 0x13E385AA..=0x13E385AA   FFFFFFFF-like count:0    bit-level stats table:
         0x0 0x1  0x2 0x3| 0x4 0x5  0x6 0x7| 0x8 0x9  0xA 0xB| 0xC 0xD  0xE 0xF
SinglIdx                1|                 |                 |                 
TogglCnt       1         |                 |                 |                 
1sInValu                 |                 |                 |                1

Error found. Mode INITIAL_READ, total errors 0x1 out of 0x34000000 (0.00000011%)
Errors address range: 0x1ABC9B8D8..=0x1ABC9B8DB  iteration:360
values range: 0x232E7C2A..=0x232E7C2A   FFFFFFFF-like count:0    bit-level stats table:
         0x0 0x1  0x2 0x3| 0x4 0x5  0x6 0x7| 0x8 0x9  0xA 0xB| 0xC 0xD  0xE 0xF
SinglIdx                1|                 |                 |                 
TogglCnt       1         |                 |                 |                 
1sInValu                 |                 |                 |                1

Error found. Mode INITIAL_READ, total errors 0x1 out of 0x34000000 (0.00000011%)
Errors address range: 0x1ABC9B8D8..=0x1ABC9B8DB  iteration:504
values range: 0x7475012A..=0x7475012A   FFFFFFFF-like count:0    bit-level stats table:
         0x0 0x1  0x2 0x3| 0x4 0x5  0x6 0x7| 0x8 0x9  0xA 0xB| 0xC 0xD  0xE 0xF
SinglIdx                1|                 |                 |                 
TogglCnt       1         |                 |                 |                 
1sInValu                 |                 |                 |       1         

Error found. Mode INITIAL_READ, total errors 0x1 out of 0x34000000 (0.00000011%)
Errors address range: 0x1ABC9B8D8..=0x1ABC9B8DB  iteration:549
values range: 0x519E12AA..=0x519E12AA   FFFFFFFF-like count:0    bit-level stats table:
         0x0 0x1  0x2 0x3| 0x4 0x5  0x6 0x7| 0x8 0x9  0xA 0xB| 0xC 0xD  0xE 0xF
SinglIdx                1|                 |                 |                 
TogglCnt       1         |                 |                 |                 
1sInValu                 |                 |                 |            1    

Error found. Mode INITIAL_READ, total errors 0x1 out of 0x34000000 (0.00000011%)
Errors address range: 0x1ABC9B8D8..=0x1ABC9B8DB  iteration:767
values range: 0x32E5833A..=0x32E5833A   FFFFFFFF-like count:0    bit-level stats table:
         0x0 0x1  0x2 0x3| 0x4 0x5  0x6 0x7| 0x8 0x9  0xA 0xB| 0xC 0xD  0xE 0xF
SinglIdx                1|                 |                 |                 
TogglCnt       1         |                 |                 |                 
1sInValu                 |                 |                 |                1

Error found. Mode INITIAL_READ, total errors 0x1 out of 0x34000000 (0.00000011%)
Errors address range: 0x1ABC9B8D8..=0x1ABC9B8DB  iteration:775
values range: 0x3658B6AB..=0x3658B6AB   FFFFFFFF-like count:0    bit-level stats table:
         0x0 0x1  0x2 0x3| 0x4 0x5  0x6 0x7| 0x8 0x9  0xA 0xB| 0xC 0xD  0xE 0xF
SinglIdx                1|                 |                 |                 
TogglCnt       1         |                 |                 |                 
1sInValu                 |                 |                 |                 
   0x1?        1         |                 |                 |                 

[galkinvv]

Same lower address bits + same Single bit idx confirms thats specific bit in a signble VRAM IC is failing.

Does that narrow it down to a particular VRAM chip

No. Relating address data to specific chip is technically too coplex for NVIDIA GPUs, so that can be treated as "nearly impossible".

And from memtest_vulkan output it is really impossible to get physical VRAM address (in theory it would require adding a lot of runtime inspection I thought about but failed to invent how to implement)

[Olari-A]

Thank you. Much appreciated.