panic on copy

[DanielHabenicht]

Which version of the AzCopy was used?

Note: The version is visible when running AzCopy without any argument

Latest: 10.19.0

Which platform are you using? (ex: Windows, Mac, Linux)

Linux

What command did you run? / What problem was encountered?

/tmp/azcopy copy "https://masterthesisdata.dfs.core.windows.net/benchmark-cache/datagen/synthetic-days{READACTED}" /home/azureuser/.ldim_benchmark_cache/datagen --recursive
INFO: Scanning...
INFO: Any empty folders will be processed, because source and destination both support folders

Job ef754a42-6397-0c40-7f6a-26a3056e4ca3 has started
Log file is located at: /root/.azcopy/ef754a42-6397-0c40-7f6a-26a3056e4ca3.log

panic: Folder /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-352/flows shouldn't finish tracking until it's been recorded
Current Contents:
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-245/demands V: 8426
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-292/flows V: 9677
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-152/levels V: 5183
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-345/flows V: 9132
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-327/pressures V: 7117
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-261 V: 9136
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-354/flows V: 8994
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-362/pressures V: 4519
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-347/levels V: 4942
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-188 V: 9669
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-165/levels V: 3464
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-71/levels V: 5090
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-247/demands V: 1085
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-357/flows V: 9260
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-143/levels V: 7151
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-19/levels V: 9523
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-129/levels V: 971
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-281 V: 5064
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-365 V: 7233
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-195/demands V: 3091
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-254/flows V: 9553
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-237/levels V: 8813
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-318/pressures V: 8203
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-198/levels V: 3131
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-199/demands V: 7194
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-171/levels V: 5353
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-301/demands V: 9493
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-300/flows V: 5865
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-184/pressures V: 3993
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-328/demands V: 6132
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-267/flows V: 5232
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-256/demands V: 5270
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-122/levels V: 5978
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-299/levels V: 7792
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-113/levels V: 1006
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-198/demands V: 5251
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-194 V: 9155
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-286/levels V: 2784
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-275/pressures V: 6701
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-320/demands V: 6138
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-287/pressures V: 5571
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-342/flows V: 9335
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-354 V: 8452
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-123/levels V: 8925
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-345/demands V: 5425
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-308/levels V: 5577
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-214/demands V: 5666
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-350/flows V: 6243
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-246 V: 7174
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-305 V: 9862
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-89/levels V: 9571
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-98/levels V: 961
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-322 V: 2228
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-331 V: 9192
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-257/levels V: 3826
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-3/levels V: 9405
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-325 V: 9240
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-332/demands V: 9579
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-211/levels V: 2620
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-320/flows V: 7887
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-289/flows V: 8259
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-28/levels V: 3686
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-250 V: 7296
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-278/demands V: 7486
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-132/levels V: 6398
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-365/flows V: 6812
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-282 V: 6737
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-202 V: 6394
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-328/pressures V: 9911
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-73/levels V: 7791
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-315/levels V: 8952
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-44/levels V: 9483
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-199/flows V: 7811
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-203 V: 4565
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-293 V: 8974
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-8/levels V: 8584
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-155/levels V: 2992
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-262/levels V: 8934
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-287/levels V: 9077
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-251/pressures V: 2553
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-296/pressures V: 9831
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-1/levels V: 5230
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-104/levels V: 6785
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-361/pressures V: 816
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-264 V: 4091
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-337/demands V: 3690
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-362/demands V: 3766
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-210/flows V: 6242
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-318 V: 1715
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-341/demands V: 9073
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-234/demands V: 9850
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-200/pressures V: 1793
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-302/flows V: 8150
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-326/pressures V: 918
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-353 V: 6293
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-307/flows V: 8585
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-258 V: 5235
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-136/levels V: 9989
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-144/levels V: 660
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-290/pressures V: 2791
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-330/levels V: 9026
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-117/levels V: 1345
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-214/flows V: 8603
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-188/flows V: 5498
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-185/flows V: 5891
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-197/levels V: 4527
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-240/levels V: 7288
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-13/levels V: 3974
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-234/levels V: 6111
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-270/pressures V: 1459
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-192/levels V: 6029
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-309/demands V: 7553
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-356/flows V: 8167
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-38/levels V: 8329
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-227/levels V: 9848
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-249/demands V: 3911
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-199/pressures V: 5467
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-243/flows V: 3376
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-297/demands V: 3820
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-196 V: 2116
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-12/levels V: 3146
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-45/levels V: 4284
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-340/levels V: 8251
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-37/levels V: 9114
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-183 V: 3957
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-340/flows V: 4084
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-258/pressures V: 8959
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-263/levels V: 3420
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-306/flows V: 9795
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-229/demands V: 8969
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-278 V: 6270
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-292/pressures V: 3463
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-351/demands V: 9917
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-218/pressures V: 7555
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-255/pressures V: 9903
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-291 V: 3483
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-32/levels V: 4791
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-186/levels V: 1679
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-331/pressures V: 8008
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-360/levels V: 5268
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-193 V: 9623
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-263/demands V: 8781
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-357/levels V: 9877
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-266/pressures V: 6236
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-364 V: 8836
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-111/levels V: 4460
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-244/pressures V: 5364
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-303/pressures V: 7070
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-285/pressures V: 8143
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-287 V: 1196
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-134/levels V: 6051
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-321/levels V: 9159
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-357/demands V: 8451
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-339/pressures V: 8311
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-196/levels V: 5716
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-205/pressures V: 7369
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-198/pressures V: 9473
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-245 V: 6686
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-220/levels V: 2107
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-241/pressures V: 3622
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-259/flows V: 8407
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-231/demands V: 4174
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-248/flows V: 7631
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-212/pressures V: 7163
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-272/flows V: 8052
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-356 V: 8107
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-253/flows V: 1600
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-217 V: 3461
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-297 V: 5163
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-312 V: 6670
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-184/demands V: 9736
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-197/pressures V: 8665
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-322/flows V: 9621
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-348/demands V: 9791
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-334/pressures V: 2705
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-277/flows V: 7753
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-161/levels V: 4147
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-248/demands V: 5271
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-333/flows V: 8565
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-229/levels V: 1508
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-253/demands V: 4034
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-200/flows V: 5324
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-355/flows V: 8761
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-338/demands V: 2891
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-278/flows V: 5556
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-264/demands V: 7947
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-150/levels V: 4354
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-212/demands V: 9561
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-302/pressures V: 7840
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-118/levels V: 3699
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-282/flows V: 7372
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-30/levels V: 9418
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-235/pressures V: 8995
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-357 V: 9267
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-222/demands V: 9556
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-239/pressures V: 8209
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-10/levels V: 9351
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-340/demands V: 9993
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-359/demands V: 6643
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-329/levels V: 810
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-305/levels V: 5011
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-267/pressures V: 9380
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-319/demands V: 9491
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-284/levels V: 7015
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-324/demands V: 7759
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-187/demands V: 2701
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-242/flows V: 7919
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-222/flows V: 8905
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-222/levels V: 5583
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-216/pressures V: 8894
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-282/levels V: 7527
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-238 V: 6723
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-252/levels V: 4738
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-350/demands V: 7713
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-80/levels V: 5678
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-61/levels V: 7429
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-280 V: 9663
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-307 V: 5895
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-292/demands V: 6883
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-346 V: 7698
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-228/flows V: 9345
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-97/levels V: 6856
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-296/demands V: 2030
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-351 V: 8540
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-247/pressures V: 9583
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-189/levels V: 1067
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-311/demands V: 8412
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-347/flows V: 4614
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-317/levels V: 5922
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-306/pressures V: 6831
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-242/levels V: 939
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-229 V: 7531
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-359 V: 6405
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-25/levels V: 7131
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-255/demands V: 7109
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-204/pressures V: 3046
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-281/flows V: 8848
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-217/levels V: 2071
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-225/levels V: 4539
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-36/levels V: 4563
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-224/levels V: 6062
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-96/levels V: 3923
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-319/levels V: 4809
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-257/flows V: 9138
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-250/levels V: 7021
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-359/pressures V: 9294
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-40/levels V: 9944
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-181/demands V: 6799
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-221 V: 8088
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-101/levels V: 5273
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-281/pressures V: 4619
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-206/demands V: 1831
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-361 V: 9981
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-274/levels V: 611
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-291/demands V: 8858
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-359/flows V: 8737
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-241/flows V: 6914
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-329 V: 5204
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-365/levels V: 8960
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-214/levels V: 9816
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-237/flows V: 5172
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-286/demands V: 7226
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-141/levels V: 8608
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-189/pressures V: 2775
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-311 V: 6553
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-75/levels V: 5340
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-204/flows V: 7950
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-33/levels V: 8810
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-84/levels V: 1358
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-348/pressures V: 7478
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-253/pressures V: 6330
K: /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-346/demands V: 9580
K: /home/azureuse

goroutine 2938 [running]:
github.com/Azure/azure-storage-azcopy/v10/ste.(*jpptFolderTracker).StopTracking(0xc001c46520, {0xc004b169c0, 0x55})
        /home/vsts/work/1/s/ste/folderCreationTracker.go:172 +0x2e5
github.com/Azure/azure-storage-azcopy/v10/ste.remoteToLocal_folder({0x11af820, 0xc002219dc0}, {0x800000?, 0xc00252c7e0?}, {0x10e?, 0x0?}, 0x105c590)
        /home/vsts/work/1/s/ste/xfer-remoteToLocal-folder.go:73 +0x32a
github.com/Azure/azure-storage-azcopy/v10/ste.remoteToLocal({0x11af820, 0xc002219dc0}, {0x1198a40, 0xc00270f840}, {0x7fb498348998, 0xc0004343c0}, 0x2e?)
        /home/vsts/work/1/s/ste/xfer-remoteToLocal-file.go:43 +0x1ab
github.com/Azure/azure-storage-azcopy/v10/ste.computeJobXfer.func4({0x11af820?, 0xc002219dc0?}, {0x1198a40?, 0xc00270f840?}, {0x7fb498348998?, 0xc0004343c0?})
        /home/vsts/work/1/s/ste/xfer.go:70 +0x43
github.com/Azure/azure-storage-azcopy/v10/ste.(*jobPartMgr).StartJobXfer(0x1817f00?, {0x11af820?, 0xc002219dc0?})
        /home/vsts/work/1/s/ste/mgr-JobPartMgr.go:776 +0x50
github.com/Azure/azure-storage-azcopy/v10/ste.(*jobPartTransferMgr).StartJobXfer(0xc002219dc0?)
        /home/vsts/work/1/s/ste/mgr-JobPartTransferMgr.go:246 +0x32
github.com/Azure/azure-storage-azcopy/v10/ste.(*jobMgr).transferProcessor.func1({0x11af820, 0xc002219dc0})
        /home/vsts/work/1/s/ste/mgr-JobMgr.go:1023 +0x199
github.com/Azure/azure-storage-azcopy/v10/ste.(*jobMgr).transferProcessor(0xc000875800, 0xc002e603ca?)
        /home/vsts/work/1/s/ste/mgr-JobMgr.go:1035 +0xbd
created by github.com/Azure/azure-storage-azcopy/v10/ste.NewJobMgr
        /home/vsts/work/1/s/ste/mgr-JobMgr.go:202 +0xc37

How can we reproduce the problem in the simplest way?

Create many directories with content and upload via az storage fs directory upload.
The main thing here is that for the directories empty blobs are created (representing empty folders), but also files in this path.

Have you found a mitigation/solution?

Yes, if during the upload empty blobs (representing an empty folder) where created but the real files in this folder also exist the 0 Byte blobs have to be deleted:

Set-AzContext <YourContextId>
$Context = New-AzStorageContext -StorageAccountName masterthesisdata
Get-AzStorageBlob -Context $Context -Container "benchmark-cache" -Blob "datagen/synthetic-days*" | Where-Object {$_.Length -eq 0} | Remove-AzStorageBlob

Other

I tried using a previous version (bundled with azure-cli: 10.13.0). But this version seems to fail to download many files.
(Empty File Creation error open /home/azureuser/.ldim_benchmark_cache/datagen/synthetic-days/synthetic-days-238: is a directory) This is reproducible to at least version 1.18.0, which I also tried.

I uploaded all the files before via az storage fs directory upload
Also this seems to work sometimes, as I had successful copy jobs: https://github.com/ldimbenchmark/benchmark/actions/workflows/complexity-benchmark.yml

[DanielHabenicht]

Could it be that the it gets confused by the 0 Byte Blobs?

Yes this seems to be the issue, deleting them resulted in copying all files without any failed files.

Also I would really appreciate if minor version where available from the release page, instead of letting users guess what the download link is.

[gapra-msft]

@DanielHabenicht are you still experiencing this issue with latest AzCopy?

[DanielHabenicht]

I haven't encountered this usecase anymore (and thus also not the bug). But as I described how to reproduce it you can probably recreate the environment in which the bug surfaces.

Apart from that you can close it. I am happy to reopen it when I come across it again in half a year... 😜
Or did you already test it? In that case I can try it for myself to validate its working again.

[gapra-msft]

I'll make sure this gets looked into as soon as possible.

[gfnord]

I'm having the same issue with the latest version.

az storage fs directory download --account-name ets -d "./" --recursive --account-key *** -f adfmytestadf-hdinsightondemandlinkedservice-20160104154417

INFO: Scanning...
INFO: Any empty folders will be processed, because source and destination both support folders

Job d3b58a38-1649-9d4b-670e-df3fd4b79e96 has started
Log file is located at: C:\Users\gusta.azcopy\d3b58a38-1649-9d4b-670e-df3fd4b79e96.log

panic: Folder \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\HdiSamples\SensorSampleData\building shouldn't finish tracking until it's been recorded
Current Contents:
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\HdiSamples V: 194
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\mapred\history V: 216
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\mapred\history\done V: 176
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\apps V: 333
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\example\data\gutenberg V: 321
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\user\hdp\share\lib\hcatalog V: 186
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\HdiSamples\SensorSampleData\hvac V: 346
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\user\hdp\share\lib\hive V: 287
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\example\data V: 148
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\user\hdp\share\lib\sqoop V: 303
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\user\hdp\share\lib\pig V: 175
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\hive V: 202
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\user\hdp\share\lib\mapreduce-streaming V: 183
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\apps\tez V: 258
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\user\hdp\share\lib\distcp V: 295
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\HdiSamples\StorageAnalytics V: 152
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\HdiSamples\WebsiteLogSampleData\SampleLog V: 134
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\user V: 330
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\hive\warehouse V: 234
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\HdiSamples\MahoutMovieData V: 227
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\mapred\history\done_intermediate V: 5
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\HdiSamples\TwitterTrendsSampleData V: 188
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\HdiSamples\SensorSampleData V: 318
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\yarn\generic-history\ApplicationHistoryDataRoot V: 306
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\hive\warehouse\hivesampletable V: 272
K: \?\D:\Azure\ETS\adfmytestadf-hdinsightondemandlinkedservice-20160104154417\user\hdp\share\lib\oozie V: 201

goroutine 370 [running]:
github.com/Azure/azure-storage-azcopy/v10/ste.(*jpptFolderTracker).StopTracking(0xc0006c00f0, {0xc000526d20, 0x70})
D:/a/1/s/ste/folderCreationTracker.go:181 +0x2d4
github.com/Azure/azure-storage-azcopy/v10/ste.remoteToLocal_folder({0x12d4320, 0xc00032c230}, {0x423230?, 0xc00200c880?}, 0x1160740)
D:/a/1/s/ste/xfer-remoteToLocal-folder.go:80 +0x2ca
github.com/Azure/azure-storage-azcopy/v10/ste.remoteToLocal({0x12d4320, 0xc00032c230}, {0x22f46b647c0, 0xc00035f900}, 0xd4f595?)
D:/a/1/s/ste/xfer-remoteToLocal-file.go:42 +0xa9
github.com/Azure/azure-storage-azcopy/v10/ste.computeJobXfer.func4({0x12d4320?, 0xc00032c230?}, {0x22f46b647c0?, 0xc00035f900?})
D:/a/1/s/ste/xfer.go:68 +0x36
github.com/Azure/azure-storage-azcopy/v10/ste.(*jobPartMgr).StartJobXfer(0x194ea60?, {0x12d4320?, 0xc00032c230?})
D:/a/1/s/ste/mgr-JobPartMgr.go:472 +0x42
github.com/Azure/azure-storage-azcopy/v10/ste.(*jobPartTransferMgr).StartJobXfer(0xc00032c230?)
D:/a/1/s/ste/mgr-JobPartTransferMgr.go:246 +0x32
github.com/Azure/azure-storage-azcopy/v10/ste.(*jobMgr).transferProcessor.func1({0x12d4320, 0xc00032c230})
D:/a/1/s/ste/mgr-JobMgr.go:1093 +0x199
github.com/Azure/azure-storage-azcopy/v10/ste.(*jobMgr).transferProcessor(0xc000a80400, 0x8ee088296759e0c5?)
D:/a/1/s/ste/mgr-JobMgr.go:1105 +0xbd
created by github.com/Azure/azure-storage-azcopy/v10/ste.NewJobMgr
D:/a/1/s/ste/mgr-JobMgr.go:200 +0xc17
Failed to perform copy operation.