Kohya please help us to debug the reason why Windows and WSL is almost 2x slower than Linux machine same libraries - Qwen Image training

[FurkanGozukara]

I am using Torch 2.8 and CUDA 12.9 basically every library i use is exactly same on my Windows and Linux machine

However, on Linux with Adafactor Qwen Image training is almost 2x faster

Other optimizers seems like not affected like this

I mean for example best Windows speed is around 9.5 second / it meanwhile Linux is 5.9 second / it

Exactly same settings

I even installed Ubuntu 22 WSL and same here screenshot

It simply can't use GPU watt for some reason

I test Linux on RunPod and their machine is slower than mine

[FurkanGozukara]

ok i did some testing and the issue is block swapping @kohya-ss

without block swapping the training time getting closer to RTX 6000 PRO speeds

so the issue is block swapping slower on Windows I think

steps:   0%|                                                        | 5/5600 [01:32<28:49:26, 18.55s/it, avr_loss=0.11]
[qwen-image-block] BLOCK SWAP TIMING (#380):
  Total: 156.9ms | Modules: 18
  Build jobs: 0.1ms
  Sync before: 6.4ms
  Create buffers: 0.0ms
  GPU→Staging: 48.8ms
  CPU→Staging: 27.3ms ⚠️
  Event sync: 0.1ms
  Staging→GPU: 47.4ms
  Staging→CPU: 26.7ms ⚠️
  Final sync: 0.0ms

[qwen-image-block] BLOCK SWAP TIMING (#390):
  Total: 163.8ms | Modules: 18
  Build jobs: 0.1ms
  Sync before: 6.0ms
  Create buffers: 0.0ms
  GPU→Staging: 53.5ms
  CPU→Staging: 27.0ms ⚠️
  Event sync: 0.1ms
  Staging→GPU: 49.9ms
  Staging→CPU: 26.9ms ⚠️
  Final sync: 0.1ms

[qwen-image-block] BLOCK SWAP TIMING (#400):
  Total: 167.0ms | Modules: 18
  Build jobs: 0.1ms
  Sync before: 5.9ms
  Create buffers: 0.0ms
  GPU→Staging: 55.5ms
  CPU→Staging: 29.4ms ⚠️
  Event sync: 0.1ms
  Staging→GPU: 48.5ms
  Staging→CPU: 27.2ms ⚠️
  Final sync: 0.1ms

[qwen-image-block] BLOCK SWAP TIMING (#410):
  Total: 206.9ms | Modules: 18
  Build jobs: 0.1ms
  Sync before: 55.5ms
  Create buffers: 0.0ms
  GPU→Staging: 49.4ms
  CPU→Staging: 27.2ms ⚠️
  Event sync: 0.0ms
  Staging→GPU: 47.2ms
  Staging→CPU: 27.2ms ⚠️
  Final sync: 0.0ms

[qwen-image-block] BLOCK SWAP TIMING (#420):
  Total: 217.7ms | Modules: 18
  Build jobs: 0.2ms
  Sync before: 57.3ms
  Create buffers: 0.0ms
  GPU→Staging: 49.9ms
  CPU→Staging: 31.9ms ⚠️
  Event sync: 0.1ms
  Staging→GPU: 48.6ms
  Staging→CPU: 29.2ms ⚠️
  Final sync: 0.1ms

[qwen-image-block] BLOCK SWAP TIMING (#430):
  Total: 212.6ms | Modules: 18
  Build jobs: 0.2ms
  Sync before: 57.8ms
  Create buffers: 0.0ms
  GPU→Staging: 49.5ms
  CPU→Staging: 28.8ms ⚠️
  Event sync: 0.1ms
  Staging→GPU: 48.1ms
  Staging→CPU: 27.7ms ⚠️
  Final sync: 0.0ms

[qwen-image-block] BLOCK SWAP TIMING (#440):
  Total: 214.4ms | Modules: 18
  Build jobs: 0.2ms
  Sync before: 57.4ms
  Create buffers: 0.0ms
  GPU→Staging: 49.7ms
  CPU→Staging: 30.0ms ⚠️
  Event sync: 0.1ms
  Staging→GPU: 48.4ms
  Staging→CPU: 28.2ms ⚠️
  Final sync: 0.0ms
steps:   0%|                                                       | 6/5600 [01:50<28:37:31, 18.42s/it, avr_loss=0.108]

[FurkanGozukara]

@kohya-ss i started a linux machine on RunPod

exactly same config file

here the reason of difference

the difference is literally insane can we fix?

On Linux Speed Below

steps:   1%|▊                                                                                                           | 40/5600 [03:34<8:16:24,  5.36s/it, avr_loss=0.122]
[qwen-image-block] 🐧 LINUX BLOCK SWAP #2970 (pinned):
  Total: 57.6ms | Modules: 18
  Build jobs: 0.3ms
  Create buffers: 0.0ms
  GPU→Staging: 1.1ms
  CPU→Staging: 29.5ms
  Event sync: 0.1ms
  Staging→GPU: 0.3ms
  Staging→CPU: 26.0ms
  Final sync: 0.0ms

[qwen-image-block] 🐧 LINUX BLOCK SWAP #2980 (pinned):
  Total: 57.5ms | Modules: 18
  Build jobs: 0.3ms
  Create buffers: 0.0ms
  GPU→Staging: 1.2ms
  CPU→Staging: 29.3ms
  Event sync: 0.1ms
  Staging→GPU: 0.3ms
  Staging→CPU: 26.0ms
  Final sync: 0.0ms

[qwen-image-block] 🐧 LINUX BLOCK SWAP #2990 (pinned):
  Total: 58.5ms | Modules: 18
  Build jobs: 0.2ms
  Create buffers: 0.0ms
  GPU→Staging: 1.6ms
  CPU→Staging: 29.9ms
  Event sync: 0.1ms
  Staging→GPU: 0.3ms
  Staging→CPU: 26.0ms
  Final sync: 0.0ms

[qwen-image-block] 🐧 LINUX BLOCK SWAP #3000 (pinned):
  Total: 58.0ms | Modules: 18
  Build jobs: 0.3ms
  Create buffers: 0.0ms
  GPU→Staging: 0.6ms
  CPU→Staging: 27.3ms
  Event sync: 0.1ms
  Staging→GPU: 0.4ms
  Staging→CPU: 28.9ms
  Final sync: 0.0ms

[qwen-image-block] 🐧 LINUX BLOCK SWAP #3010 (pinned):
  Total: 59.0ms | Modules: 18
  Build jobs: 0.3ms
  Create buffers: 0.0ms
  GPU→Staging: 0.5ms
  CPU→Staging: 29.0ms
  Event sync: 0.1ms
  Staging→GPU: 0.3ms
  Staging→CPU: 28.5ms
  Final sync: 0.0ms

[qwen-image-block] 🐧 LINUX BLOCK SWAP #3020 (pinned):
  Total: 57.5ms | Modules: 18
  Build jobs: 0.3ms
  Create buffers: 0.0ms
  GPU→Staging: 0.6ms
  CPU→Staging: 27.0ms
  Event sync: 0.1ms
  Staging→GPU: 0.3ms
  Staging→CPU: 28.8ms
  Final sync: 0.0ms

[qwen-image-block] 🐧 LINUX BLOCK SWAP #3030 (pinned):
  Total: 55.5ms | Modules: 18
  Build jobs: 0.3ms
  Create buffers: 0.0ms
  GPU→Staging: 0.4ms
  CPU→Staging: 27.8ms
  Event sync: 0.1ms
  Staging→GPU: 0.3ms
  Staging→CPU: 26.1ms
  Final sync: 0.0ms
steps:   1%|▊                                                                                                           | 41/5600 [03:39<8:15:24,  5.35s/it, avr_loss=0.124]

On Windows Speed Below

steps:   0%|                                                       | 7/5600 [01:24<18:42:21, 12.04s/it, avr_loss=0.118]
[qwen-image-block] 🪟 WINDOWS BLOCK SWAP #500 (pinned):
  Total: 102.3ms | Modules: 18
  Build jobs: 0.1ms
  Pre sync: 0.0ms
  Create buffers: 0.0ms
  GPU→Staging: 8.8ms
  CPU→Staging: 31.4ms
  Event sync: 0.1ms
  Staging→GPU: 0.7ms
  Staging→CPU: 49.6ms
  Final sync: 11.2ms

[qwen-image-block] 🪟 WINDOWS BLOCK SWAP #510 (pinned):
  Total: 101.5ms | Modules: 18
  Build jobs: 0.1ms
  Pre sync: 0.0ms
  Create buffers: 0.0ms
  GPU→Staging: 0.9ms
  CPU→Staging: 35.3ms
  Event sync: 0.1ms
  Staging→GPU: 0.4ms
  Staging→CPU: 57.2ms
  Final sync: 7.1ms

[qwen-image-block] 🪟 WINDOWS BLOCK SWAP #520 (pinned):
  Total: 95.6ms | Modules: 18
  Build jobs: 0.2ms
  Pre sync: 0.0ms
  Create buffers: 0.0ms
  GPU→Staging: 1.4ms
  CPU→Staging: 40.1ms
  Event sync: 0.1ms
  Staging→GPU: 0.6ms
  Staging→CPU: 31.7ms
  Final sync: 21.2ms

[qwen-image-block] 🪟 WINDOWS BLOCK SWAP #530 (pinned):
  Total: 125.8ms | Modules: 18
  Build jobs: 0.2ms
  Pre sync: 6.6ms
  Create buffers: 0.0ms
  GPU→Staging: 1.0ms
  CPU→Staging: 32.5ms
  Event sync: 13.1ms
  Staging→GPU: 0.3ms
  Staging→CPU: 28.4ms
  Final sync: 43.6ms

[qwen-image-block] 🪟 WINDOWS BLOCK SWAP #540 (pinned):
  Total: 137.4ms | Modules: 18
  Build jobs: 0.2ms
  Pre sync: 0.0ms
  Create buffers: 0.0ms
  GPU→Staging: 1.4ms
  CPU→Staging: 37.6ms
  Event sync: 24.1ms
  Staging→GPU: 0.6ms
  Staging→CPU: 29.7ms
  Final sync: 43.4ms

[qwen-image-block] 🪟 WINDOWS BLOCK SWAP #550 (pinned):
  Total: 136.7ms | Modules: 18
  Build jobs: 0.2ms
  Pre sync: 0.1ms
  Create buffers: 0.0ms
  GPU→Staging: 1.3ms
  CPU→Staging: 36.0ms
  Event sync: 27.1ms
  Staging→GPU: 0.2ms
  Staging→CPU: 27.7ms
  Final sync: 43.8ms

[qwen-image-block] 🪟 WINDOWS BLOCK SWAP #560 (pinned):
  Total: 144.6ms | Modules: 18
  Build jobs: 0.2ms
  Pre sync: 6.5ms
  Create buffers: 0.0ms
  GPU→Staging: 1.0ms
  CPU→Staging: 32.3ms
  Event sync: 31.3ms
  Staging→GPU: 0.3ms
  Staging→CPU: 28.9ms
  Final sync: 43.7ms
steps:   0%|                                                       | 8/5600 [01:34<18:23:33, 11.84s/it, avr_loss=0.111]

[AiInfluenceGr]

This is a known , issue. My GPU is only using MAX 320WATT on 100% power (4090 24gb). Please fix this! Thanks

[kohya-ss]

To some extent, this is an intended result. Qwen-Image is lightweight compared to video generation models, so as the number of block swaps increases, the bottleneck becomes transfer speed rather than computation speed.

Looking at your results, it seems that the transfer time between the CPU and staging memory is shorter on Linux. This may be because the bandwidth of Linux's main RAM is higher than on Windows.
Could you please tell us the memory specifications of each PC (DDR4 or 5, number of channels)?

It may also be possible to increase the amount of calculation per step to shift the bottleneck to computation speed. Could you share the command line options and your dataset settings?

[FurkanGozukara]

To some extent, this is an intended result. Qwen-Image is lightweight compared to video generation models, so as the number of block swaps increases, the bottleneck becomes transfer speed rather than computation speed.

Looking at your results, it seems that the transfer time between the CPU and staging memory is shorter on Linux. This may be because the bandwidth of Linux's main RAM is higher than on Windows. Could you please tell us the memory specifications of each PC (DDR4 or 5, number of channels)?

It may also be possible to increase the amount of calculation per step to shift the bottleneck to computation speed. Could you share the command line options and your dataset settings?

my cpu z here

https://valid.x86.fr/204pgu

dataset

[[datasets]]
image_directory = "C:/Users/Furkan/Pictures/28_imgs_1328/1_ohwx"
num_repeats = 1
cache_directory = "C:/Users/Furkan/Pictures/28_imgs_1328/1_ohwx/cache_dir"

[general]
resolution = [ 1328, 1328]
caption_extension = ".txt"
batch_size = 1
enable_bucket = false
bucket_no_upscale = false

command line

F:\SECourses_Musubi_Trainer_v16\SECourses_Musubi_Trainer\venv\Scripts\accelerate.EXE launch --dynamo_backend no --gpu_ids 0 --mixed_precision bf16 --num_processes 1 --num_machines 1 --num_cpu_threads_per_process 2 F:/SECourses_Musubi_Trainer_v16/SECourses_Musubi_Trainer/musubi-tuner/src/musubi_tuner/qwen_image_train_network.py --config_file F:\SECourses_Musubi_Trainer_v16\test_trainings\Tier1_21700_MB_VRAM_Need_More_RAM_20251026-024300.toml

I am testing with 34 block swap right now

[kohya-ss]

Thank you for sharing this information. If Linux PC has, for example, 4 channels RAM, Linux is likely to be faster.

Also, I would appreciate it if you could share the contents of your config.toml. I'm wondering if gradient checkpointing is disabled.

[FurkanGozukara]

Thank you for sharing this information. If Linux PC has, for example, 4 channels RAM, Linux is likely to be faster.

Also, I would appreciate it if you could share the contents of your config.toml. I'm wondering if gradient checkpointing is disabled.

oh it is here

it is enabled

i used same config on linux too

also on linux machine i couldnt get RAM details since it is on runpod - i bet it is worse than my RAM

Tier1_21700_MB_VRAM_Need_More_RAM_20251026-025820.toml.txt

when Gradient checkpointing is disabled even with 58 block swap it uses 10 GB shared vram on RTX 5090 - not fitting into VRAM

I tested like this

the file is here : https://github.com/FurkanGozukara/musubi-tuner/blob/main/src/musubi_tuner/modules/custom_offloading_utils.py

    def swap_weight_devices_cuda(self, device: torch.device, layer_to_cpu: nn.Module, layer_to_cuda: nn.Module):
        assert layer_to_cpu.__class__ == layer_to_cuda.__class__

        overall_start = time.perf_counter()
        timings = {}

        weight_swap_jobs = []

        # This is not working for all cases (e.g. SD3), so we need to find the corresponding modules. kept here for reference:
        # for module_to_cpu, module_to_cuda in zip(layer_to_cpu.modules(), layer_to_cuda.modules()):
        #     print(module_to_cpu.__class__, module_to_cuda.__class__)
        #     if hasattr(module_to_cpu, "weight") and module_to_cpu.weight is not None:
        #         weight_swap_jobs.append((module_to_cpu, module_to_cuda, module_to_cpu.weight.data, module_to_cuda.weight.data))

        t0 = time.perf_counter()
        modules_to_cpu = {k: v for k, v in layer_to_cpu.named_modules()}
        for module_to_cuda_name, module_to_cuda in layer_to_cuda.named_modules():
            if hasattr(module_to_cuda, "weight") and module_to_cuda.weight is not None:
                module_to_cpu = modules_to_cpu.get(module_to_cuda_name, None)
                if module_to_cpu is not None and module_to_cpu.weight.shape == module_to_cuda.weight.shape:
                    weight_swap_jobs.append((module_to_cpu, module_to_cuda, module_to_cpu.weight.data, module_to_cuda.weight.data))
                else:
                    if module_to_cuda.weight.data.device.type != device.type:
                        module_to_cuda.weight.data = module_to_cuda.weight.data.to(device)
        timings['build_jobs'] = time.perf_counter() - t0

        t0 = time.perf_counter()
        torch.cuda.current_stream().synchronize()  # this prevents the illegal loss value
        timings['pre_sync'] = time.perf_counter() - t0

        stream = torch.cuda.Stream()
        with torch.cuda.stream(stream):
            if self.staging_buffer_a is None:
                t0 = time.perf_counter()
                # Create staging buffer as pinned memory (as shared GPU ram). We specify device for correct pinning on multi-GPU systems
                self.staging_buffer_a = [
                    torch.empty_like(cuda_data_view, device="cpu").pin_memory(device=device)
                    for _, _, cuda_data_view, _ in weight_swap_jobs
                ]
                self.staging_buffer_b = [
                    torch.empty_like(cuda_data_view, device="cpu").pin_memory(device=device)
                    for _, _, cuda_data_view, _ in weight_swap_jobs
                ]
                timings['create_buffers'] = time.perf_counter() - t0

            events = [torch.cuda.Event() for _ in weight_swap_jobs]  # Waiting events for staging buffer A to CPU non-blocking copy

            # Copy weights to staging buffers and record events
            t0 = time.perf_counter()
            gpu_to_staging_time = 0
            cpu_to_staging_time = 0
            for event, sbuf_a, sbuf_b, (module_to_cpu, module_to_cuda, cuda_data_view, cpu_data_view) in zip(
                events, self.staging_buffer_a, self.staging_buffer_b, weight_swap_jobs
            ):
                # CUDA to staging buffer A, non-blocking copy
                t1 = time.perf_counter()
                sbuf_a.copy_(cuda_data_view.data, non_blocking=True)
                event.record(stream)
                gpu_to_staging_time += time.perf_counter() - t1

                # CPU to staging buffer B, CPU to pinned CPU, synchronous copy. Can overlap with CUDA to staging buffer A
                # Making this multithreaded does not help, and 'non_blocking=True' does not help either.
                t1 = time.perf_counter()
                sbuf_b.copy_(module_to_cuda.weight.data)  # BOTTLENECK
                cpu_to_staging_time += time.perf_counter() - t1
            timings['gpu_to_staging'] = gpu_to_staging_time
            timings['cpu_to_staging'] = cpu_to_staging_time

        with torch.cuda.stream(stream):
            event_sync_time = 0
            staging_to_gpu_time = 0
            staging_to_cpu_time = 0
            for event, sbuf_a, sbuf_b, (module_to_cpu, module_to_cuda, cuda_data_view, cpu_data_view) in zip(
                events, self.staging_buffer_a, self.staging_buffer_b, weight_swap_jobs
            ):
                # Wait for staging buffer A to be ready, and CUDA data view can be reused
                t1 = time.perf_counter()
                event.synchronize()
                event_sync_time += time.perf_counter() - t1

                # Staging buffer B to CUDA, non-blocking copy.
                t1 = time.perf_counter()
                cuda_data_view.copy_(sbuf_b, non_blocking=True)
                staging_to_gpu_time += time.perf_counter() - t1

                # Staging buffer A to CPU, synchronous copy. Can overlap with staging buffer B to CUDA
                t1 = time.perf_counter()
                cpu_data_view.copy_(sbuf_a)  # BOTTLENECK
                staging_to_cpu_time += time.perf_counter() - t1

                # Update references
                module_to_cuda.weight.data = cuda_data_view
                module_to_cpu.weight.data = cpu_data_view
            timings['event_sync'] = event_sync_time
            timings['staging_to_gpu'] = staging_to_gpu_time
            timings['staging_to_cpu'] = staging_to_cpu_time

        t0 = time.perf_counter()
        stream.synchronize()  # Synchronize staging buffer B to CUDA
        torch.cuda.current_stream().synchronize()  # This prevents the illegal loss value
        timings['final_sync'] = time.perf_counter() - t0

        total_time = time.perf_counter() - overall_start
        
        # Print detailed timing statistics
        if not hasattr(self, '_swap_count'):
            self._swap_count = 0
        
        self._swap_count += 1
        if self._swap_count % 10 == 0:  # Print every 10th swap
            print(f"\n[{self.block_type}] 🪟 WINDOWS BLOCK SWAP #{self._swap_count} (pinned):")
            print(f"  Total: {total_time*1000:.1f}ms | Modules: {len(weight_swap_jobs)}")
            print(f"  Build jobs: {timings.get('build_jobs', 0)*1000:.1f}ms")
            print(f"  Pre sync: {timings.get('pre_sync', 0)*1000:.1f}ms")
            print(f"  Create buffers: {timings.get('create_buffers', 0)*1000:.1f}ms")
            print(f"  GPU→Staging: {timings.get('gpu_to_staging', 0)*1000:.1f}ms")
            print(f"  CPU→Staging: {timings.get('cpu_to_staging', 0)*1000:.1f}ms")
            print(f"  Event sync: {timings.get('event_sync', 0)*1000:.1f}ms")
            print(f"  Staging→GPU: {timings.get('staging_to_gpu', 0)*1000:.1f}ms")
            print(f"  Staging→CPU: {timings.get('staging_to_cpu', 0)*1000:.1f}ms")
            print(f"  Final sync: {timings.get('final_sync', 0)*1000:.1f}ms")

[kohya-ss]

When I tried this on a PC with dual-channel DDR3200 memory, I got the following results.

[qwen-image-block] 🪟 WINDOWS BLOCK SWAP #620 (pinned):
  Total: 149.9ms | Modules: 18
  Build jobs: 0.1ms
  Pre sync: 0.0ms
  Create buffers: 0.0ms
  GPU→Staging: 0.7ms
  CPU→Staging: 63.5ms
  Event sync: 0.5ms
  Staging→GPU: 0.2ms
  Staging→CPU: 68.0ms
  Final sync: 16.5ms

This problem is likely due to main RAM bandwidth.

I think the solution would be to add a setting that does not use the staging buffer for environments with ample main RAM.
(However, this will take some time.)

For your current training, could you run fp8 based training by specifying --fp8_scaled --fp8_base? This can reduce the number of block swaps, and the bottleneck should become calculation speed.