Dask may stop the instances on the middle of calculation without objective reason

[dbalabka]

Describe the issue:

I've noticed that Dask may stop instances w/o any visible reason. It happens constantly with some pipelines. In the scheduler logs, I see the following:

2025-03-14 10:43:12,444 - distributed.scheduler - WARNING - Removing worker 'tls://10.164.0.134:41247' caused the cluster to lose already computed task(s), which will be recomputed elsewhere: {('hash-join-transfer-2081eb1af961e350186493cae31882dc', 22), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 27), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 29), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 31), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 36), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 38), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 40), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 45), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 47), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 49), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 56), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 58), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 2), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 11), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 20), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 25), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 34), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 4), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 52), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 13), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 15), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 61), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 17), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 22), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 24), ('repartitiontomore-f3887ec7d5bdcee31b6b2136d4f242d5', 31), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 26), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 31), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 33), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 35), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 42), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 51), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 53), ('hash-join-transfer-aa706c26e89f675e44a8caea72172309', 31), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 6), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 11), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 20), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 29), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 38), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 1), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 47), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 3), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 8), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 10), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 12), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 17), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 19), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 21), ('shuffle-transfer-bf932d11fc8d214a57c67771c0dfa3fa', 53), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 28), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 30), ('shuffle-transfer-bf932d11fc8d214a57c67771c0dfa3fa', 62), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 37), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 39), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 55), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 1), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 6), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 15), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 86), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 24), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 33), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 42), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 3), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 5), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 7), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 12), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 14), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 16), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 23), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 25), ('shuffle-transfer-bf932d11fc8d214a57c67771c0dfa3fa', 57), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 32), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 34), ('hash-join-transfer-aa706c26e89f675e44a8caea72172309', 30), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 50), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 59), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 1), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 10), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 81), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 19), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 90), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 28), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 0), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 2), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 9), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 11), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 18), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 20), ('shuffle-transfer-bf932d11fc8d214a57c67771c0dfa3fa', 52), ('shuffle-transfer-bf932d11fc8d214a57c67771c0dfa3fa', 61), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 36), ('hash-join-transfer-aa706c26e89f675e44a8caea72172309', 34), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 45), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 54), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 63), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 67), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 5), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 76), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 14), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 85), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 23), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 4), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 6), ('shuffle-transfer-bf932d11fc8d214a57c67771c0dfa3fa', 56), ('read_parquet-astype-astype-070407fd3813cf872be5ea4d39c2356c', 12), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 31), ('hash-join-transfer-aa706c26e89f675e44a8caea72172309', 29), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 40), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 49), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 58), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 62), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 0), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 9), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 80), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 89), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 1), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 17), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 26), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 86), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 35), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 44), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 53), ('repartitiontomore-aabfacbe1360a63ee50521f9e63a093e', 50), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 48), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 57), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 66), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 4), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 75), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 84), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 89), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 59), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 12), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 21), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 81), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 30), ('repartitiontomore-dc05e54db0a81800df56ce7e62001688', 31), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 90), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 39), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 48), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 34), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 43), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 52), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 61), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 70), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 75), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 84), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 93), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 54), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 63), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 7), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 67), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 16), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 76), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 25), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 85), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 34), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 29), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 38), ('repartitiontomore-aabfacbe1360a63ee50521f9e63a093e', 49), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 47), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 56), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 65), ('shuffle-transfer-bf932d11fc8d214a57c67771c0dfa3fa', 9), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 79), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 40), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 88), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 49), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 58), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 60), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 2), ('read_parquet-astype-astype-946f79a5004e9a81cd4ea0aefec34d12', 76), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 62), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 11), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 20), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 80), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 89), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 24), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 33), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 42), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 51), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 56), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 65), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 26), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 74), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 35), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 83), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 44), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 46), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 92), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 48), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 53), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 55), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 57), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 62), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 6), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 66), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 15), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 75), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 84), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 10), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 19), ('p2pshuffle-8494bce365c5c181803f5bc6a0062578', 23), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 28), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 37), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 46), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 51), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 60), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 21), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 69), ('repartitiontomore-f3887ec7d5bdcee31b6b2136d4f242d5', 30), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 30), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 78), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 39), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 41), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 87), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 43), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 48), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 50), ('read_parquet-astype-astype-946f79a5004e9a81cd4ea0aefec34d12', 66), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 57), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 59), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 1), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 52), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 61), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 70), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 79), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 5), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 14), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 23), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 32), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 37), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 46), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 7), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 55), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 16), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 64), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 25), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 27), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 73), ('repartitiontomore-f3887ec7d5bdcee31b6b2136d4f242d5', 34), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 29), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 34), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 36), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 38), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 43), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 45), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 47), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 54), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 56), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 63), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 65), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 0), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 9), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 18), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 27), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 32), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 41), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 2), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 50), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 11), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 13), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 59), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 20), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 22), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 68), ('repartitiontomore-f3887ec7d5bdcee31b6b2136d4f242d5', 29), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 24), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 29), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 31), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 33), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 20), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 40), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 38), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 42), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 49), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 51), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 4), ('operation-6b01a2b7bcb7af92e8d53f89bad59e43', 51), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 18), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 27), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 36), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 45), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 6), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 8), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 54), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 10), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 15), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 17), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 19), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 24), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 26), ('repartitiontomore-f3887ec7d5bdcee31b6b2136d4f242d5', 33), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 28), ('shuffle-transfer-bf932d11fc8d214a57c67771c0dfa3fa', 60), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 35), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 37), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 44), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 46), ('hash-join-transfer-aa706c26e89f675e44a8caea72172309', 33), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 62), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 8), ('read_parquet-astype-astype-070407fd3813cf872be5ea4d39c2356c', 70), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 13), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 22), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 93), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 31), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 40), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 1), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 3), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 49), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 5), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 10), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 12), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 14), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 19), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 21), ('repartitiontomore-f3887ec7d5bdcee31b6b2136d4f242d5', 28), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 23), ('shuffle-transfer-bf932d11fc8d214a57c67771c0dfa3fa', 55), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 30), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 32), ('hash-join-transfer-aa706c26e89f675e44a8caea72172309', 28), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 57), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 8), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 79), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 17), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 88), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 26), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 35), ('read_parquet-astype-astype-946f79a5004e9a81cd4ea0aefec34d12', 14), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 5), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 7), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 0), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 9), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 16), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 18), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 25), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 27), ('shuffle-transfer-bf932d11fc8d214a57c67771c0dfa3fa', 59), ('hash-join-transfer-aa706c26e89f675e44a8caea72172309', 32), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 43), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 52), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 61), ('read_parquet-astype-1586f6cb2dc1c29018033a9238c38895', 76), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 3), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 74), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 12), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 21), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 92), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 0), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 2), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 4), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 11), ('shuffle-transfer-bf932d11fc8d214a57c67771c0dfa3fa', 54), ('shuffle-transfer-bf932d11fc8d214a57c67771c0dfa3fa', 63), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 29), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 38), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 47), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 56), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 60), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 69), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 7), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 78), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 16), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 6), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 8), ('shuffle-transfer-bf932d11fc8d214a57c67771c0dfa3fa', 58), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 24), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 33), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 93), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 42), ('read_parquet-astype-astype-070407fd3813cf872be5ea4d39c2356c', 41), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 51), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 60), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 55), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 64), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 2), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 82), ('shuffle-transfer-bf932d11fc8d214a57c67771c0dfa3fa', 8), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 10), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 19), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 28), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 88), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 37), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 46), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 41), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 50), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 59), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 68), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 77), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 82), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 91), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 52), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 61), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 14), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 74), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 23), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 83), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 32), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 92), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 41), ('read_parquet-astype-1586f6cb2dc1c29018033a9238c38895', 47), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 36), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 45), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 54), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 63), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 72), ('shuffle-transfer-bf932d11fc8d214a57c67771c0dfa3fa', 7), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 77), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 86), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 47), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 56), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 58), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 0), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 60), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 9), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 69), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 18), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 78), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 27), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 87), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 22), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 31), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 40), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 49), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 58), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 63), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 33), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 81), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 42), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 90), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 51), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 53), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 55), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 60), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 62), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 4), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 64), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 13), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 22), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 82), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 91), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 17), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 26), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 35), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 53), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 58), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 67), ('shuffle-transfer-bf932d11fc8d214a57c67771c0dfa3fa', 6), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 28), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 76), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 37), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 39), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 85), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 41), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 46), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 48), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 94), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 50), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 55), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 57), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 59), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 8), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 68), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 77), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 3), ('repartitiontomore-dc05e54db0a81800df56ce7e62001688', 27), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 12), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 21), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 30), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 39), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 44), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 53), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 14), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 62), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 23), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 71), ('repartitiontomore-f3887ec7d5bdcee31b6b2136d4f242d5', 32), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 32), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 34), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 80), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 36), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 41), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 43), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 45), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 50), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 52), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 54), ('repartitiontomore-ea4ad71f626daa3a8e193fecff945c5d', 61), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 63), ('hash-join-transfer-2081eb1af961e350186493cae31882dc', 72), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 7), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 16), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 25), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 30), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 39), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 48), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 9), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 57), ('repartitiontomore-1e2dce54ac1ec74823c2a3bf325beda4', 18), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 66)} (stimulus_id='handle-worker-cleanup-1741948992.442437')

2025-03-14 10:43:12,442 - distributed.scheduler - INFO - Remove worker addr: tls://10.164.0.134:41247 name: dask-4c0f7f11-worker-392fa16f (stimulus_id='handle-worker-cleanup-1741948992.442437')

2025-03-14 10:42:54,270 - distributed.scheduler - INFO - Starting worker compute stream, tls://10.164.0.134:41247

2025-03-14 10:42:54,270 - distributed.scheduler - INFO - Register worker addr: tls://10.164.0.134:41247 name: dask-4c0f7f11-worker-392fa16f

2025-03-14 10:42:53,250 - distributed.scheduler - WARNING - Removing worker 'tls://10.164.0.134:43155' caused the cluster to lose already computed task(s), which will be recomputed elsewhere: {('repartitiontomore-f3887ec7d5bdcee31b6b2136d4f242d5', 30), ('p2pshuffle-a267f785e86510ca8f4e6b0aba81d2ba', 38), ('read_parquet-astype-1586f6cb2dc1c29018033a9238c38895', 47), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 42), ('p2pshuffle-8494bce365c5c181803f5bc6a0062578', 43), ('repartitiontomore-7ad1987079876f8736b141289dcdbf7a', 1), ('shuffle-transfer-74cc40b768b2249cf4cd31d60d0be0dc', 12), ('shuffle-transfer-74cc40b768b2249cf4cd31d60d0be0dc', 70), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 41), ('read_parquet-astype-astype-070407fd3813cf872be5ea4d39c2356c', 70), ('read_parquet-astype-astype-070407fd3813cf872be5ea4d39c2356c', 12), ('repartitiontomore-f3887ec7d5bdcee31b6b2136d4f242d5', 29), ('repartitiontomore-f3887ec7d5bdcee31b6b2136d4f242d5', 32), ('p2pshuffle-8494bce365c5c181803f5bc6a0062578', 42), ('read_parquet-astype-1586f6cb2dc1c29018033a9238c38895', 76), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 13), ('repartitiontomore-f3887ec7d5bdcee31b6b2136d4f242d5', 28), ('read_parquet-astype-1586f6cb2dc1c29018033a9238c38895', 18), ('read_parquet-fused-assign-c0c0fd8da0fdfdb1ed93715dd8804ae8', 71), ('repartitiontomore-f3887ec7d5bdcee31b6b2136d4f242d5', 31), ('repartitiontomore-f3887ec7d5bdcee31b6b2136d4f242d5', 34), ('shuffle-transfer-74cc40b768b2249cf4cd31d60d0be0dc', 41), ('read_parquet-astype-astype-6abd097e429505a2450e94aef9dc53fa', 70), ('read_parquet-astype-astype-070407fd3813cf872be5ea4d39c2356c', 41), ('p2pshuffle-bf932d11fc8d214a57c67771c0dfa3fa', 45), ('repartitiontomore-f3887ec7d5bdcee31b6b2136d4f242d5', 33)} (stimulus_id='handle-worker-cleanup-1741948973.2494547')

2025-03-14 10:42:53,249 - distributed.scheduler - INFO - Remove worker addr: tls://10.164.0.134:43155 name: dask-4c0f7f11-worker-392fa16f (stimulus_id='handle-worker-cleanup-1741948973.2494547')

The GCP logs say, Instance terminated by guest OS shutdown.

Minimal Complete Verifiable Example:

TBD

Anything else we need to know?:

We utilise the idle-timeout setting with a 30m timeout. However, if it is a cause, we should see specific log records.

Environment:

We run a cluster with dask-cloudprivider on GCP, with a custom image based on the original Dask image. Also, we build custom GCP instance images with Packer for faster startup.

• Dask version: 2024.12.1
• Python version: 3.11
• Operating System: Linux
• Install method (conda, pip, source): Poetry

[jacobtomlinson]

Is there anything in the worker logs that indicates why it was shut down?

[dbalabka]

@jacobtomlinson , unfortunately, no, it is complete information from the Dask dashboard -> Info -> Logs. It is reproducible quite often. I should probably lower the log level to DEBUG to get more helpful information. Any suggestions are welcome 😅

[jacobtomlinson]

I think you'll want to check the worker node logs directly on the node, or via Google Cloud logging.

[guillaumeeb]

Hi @dbalabka, how are you starting the cluster, with which arguments? Are you using adaptive?

[dbalabka]

@guillaumeeb , thanks for the reply. We have the following options that I believe can affect nodes:

1. auto_shutdown=True
2. dask.config.set({"distributed.scheduler.idle-timeout": "30m"})

We don't use cluster.adapt(). I have to check what happened on the node directly, as Jacob suggested. It is not straightforward forward, and the scheduler node does not log any meaningful information.

Soon, I will get back to this pipeline that suffers from these node shutdowns and will try to investigate deeper.