Files are in limbo on the client after the death of one node

[bertalanimre]

Hello Guys,

I am at a loss here. I've built a 3 node GlusterFS and it worked like a charm. Even had times when one node disappeared for a while and had zero issues. Now, a new story occured. The 1st node have died (hardware issue) and the data most likely is gone from there. Checked the data on my 2nd and 3rd node, the data looks to be unaffected. - This is brilliant - I thought. Then started getting errors from my application using the brick. Turnes out, the client is unable to retrieve some of the files metadata. (See below).
gluster peer status command on the 2nd node sais

Number of Peers: 2

Hostname: pub-fileserver-1
Uuid: 55162904-ee8b-4f54-9361-350d99dff873
State: Peer in Cluster (Disconnected)

Hostname: 10.100.10.133
Uuid: 62e9927d-ab4a-4a4e-a12e-ad542e15eb32
State: Peer in Cluster (Connected)

which is expected.

On the client: ls -lah

ls: cannot access 'shared/storage/app/public/upload/40/40089c1e3d58ad9f15fca15247d1bc857a668648': No data available
ls: cannot access 'shared/storage/app/public/upload/40/40141e87b59e7c47263dfdc2a190f2dd91e2da6a': No data available
ls: cannot access 'shared/storage/app/public/upload/40/40155b080cd78ac3700da35b552ceadf7edc9db2': No data available
ls: cannot access 'shared/storage/app/public/upload/40/401bff286b9d1476c073aec0c80ceaa12e69a1f0': No data available
ls: cannot access 'shared/storage/app/public/upload/40/40385af82942c344e3582740fd890aa89a1d0760': No data available
ls: cannot access 'shared/storage/app/public/upload/40/403a96f25d032e473c47dfc38d54cefaddee6783': No data available
ls: cannot access 'shared/storage/app/public/upload/40/403eea39ede573b950d21b4b739b67656b3c4db5': No data available
ls: cannot access 'shared/storage/app/public/upload/40/40471d039b9c7f6df257782d0c8d0d4e7c275f2f': No data available
ls: cannot access 'shared/storage/app/public/upload/40/4048ea5392ce7ed8c917c583a34a76e42b4f65d9': No data available
ls: cannot access 'shared/storage/app/public/upload/40/404cba113ee89f40e01cf73d3970d27b257a45d1': No data available
ls: cannot access 'shared/storage/app/public/upload/40/40558e9f3d07157aec921f997601d7fc3aae7779': No data available
ls: cannot access 'shared/storage/app/public/upload/40/4055db7948bfc2c6093a7829656725e63d5e8170': No data available
ls: cannot access 'shared/storage/app/public/upload/40/406a632bebde7adc1f5848a1b1bd5db1afd34dc1': No data available
ls: cannot access 'shared/storage/app/public/upload/40/406e55155b139484cf781dd2c230a4d073ae5602': No data available
ls: cannot access 'shared/storage/app/public/upload/40/407213d0c4cbd113937610b63586458b87650d7e': No data available
ls: cannot access 'shared/storage/app/public/upload/40/40752024e22f37d76f170e9596279ffc5693a116': No data available
ls: cannot access 'shared/storage/app/public/upload/40/407970469cfac5df8ba14f44cfa5ac6f3a0dcc12': No data available
ls: cannot access 'shared/storage/app/public/upload/40/407be892abc6d49e0e741d5cda362f2d9ad8839b': No data available
ls: cannot access 'shared/storage/app/public/upload/40/407d9c04e09caaa0e95d2c807b2e01a3ca534792': No data available
ls: cannot access 'shared/storage/app/public/upload/40/408fca2f835d65b9c65551c0e0a270fc371ccf5f': No data available
ls: cannot access 'shared/storage/app/public/upload/40/40901a8dd4c4a5166398a36435653d005d840552': No data available
ls: cannot access 'shared/storage/app/public/upload/40/4094adb6604e6e9e5c98ad5a921a977496b19e94': No data available
ls: cannot access 'shared/storage/app/public/upload/40/4095ab6f6280f9aab8ded8e2adf14611173771bb': No data available
ls: cannot access 'shared/storage/app/public/upload/40/409680df63ea927e6aee7be8db1d4c6084678336': No data available
ls: cannot access 'shared/storage/app/public/upload/40/409d8e1e16868d988188a450a16f76b8b17c8d3d': No data available
ls: cannot access 'shared/storage/app/public/upload/40/40adbff831a513d9f603f4414be68189e61ce557': No data available
ls: cannot access 'shared/storage/app/public/upload/40/40b4e169656849d9a11170b1c4b67253831b0cd5': No data available
ls: cannot access 'shared/storage/app/public/upload/40/40b8c6fdd858524e45b581bfdcb6ad58646a1d6f': No data available
ls: cannot access 'shared/storage/app/public/upload/40/40c5211c44bc670a2360a6bc05e1d3e1b217ca8f': No data available
ls: cannot access 'shared/storage/app/public/upload/40/40d7b74163915093ed9adac94fc6e2ab47f49aa1': No data available
ls: cannot access 'shared/storage/app/public/upload/40/40df5f28ca832c99656ad79b706ab599a4917d30': No data available
ls: cannot access 'shared/storage/app/public/upload/40/40e333c86bd6818ee4516e1b4b771c5e57257700': No data available
ls: cannot access 'shared/storage/app/public/upload/40/40e995d04da14b91fc0055d4530a8e41e53999cd': No data available
ls: cannot access 'shared/storage/app/public/upload/40/40efdd2d1ad838c732e9ff6e20c836a6864f34d0': No data available
total 56K
drwxrwxrwx   43 web web 4.0K Apr  7  2025 .
drwxrwxr-x 2105 web web  52K May  5 11:00 ..
d?????????    ? ?   ?      ?            ? 40089c1e3d58ad9f15fca15247d1bc857a668648
d?????????    ? ?   ?      ?            ? 40141e87b59e7c47263dfdc2a190f2dd91e2da6a
d?????????    ? ?   ?      ?            ? 40155b080cd78ac3700da35b552ceadf7edc9db2
d?????????    ? ?   ?      ?            ? 401bff286b9d1476c073aec0c80ceaa12e69a1f0
d?????????    ? ?   ?      ?            ? 40385af82942c344e3582740fd890aa89a1d0760
d?????????    ? ?   ?      ?            ? 403a96f25d032e473c47dfc38d54cefaddee6783
d?????????    ? ?   ?      ?            ? 403eea39ede573b950d21b4b739b67656b3c4db5
d?????????    ? ?   ?      ?            ? 40471d039b9c7f6df257782d0c8d0d4e7c275f2f
d?????????    ? ?   ?      ?            ? 4048ea5392ce7ed8c917c583a34a76e42b4f65d9
d?????????    ? ?   ?      ?            ? 404cba113ee89f40e01cf73d3970d27b257a45d1
d?????????    ? ?   ?      ?            ? 40558e9f3d07157aec921f997601d7fc3aae7779
d?????????    ? ?   ?      ?            ? 4055db7948bfc2c6093a7829656725e63d5e8170
d?????????    ? ?   ?      ?            ? 406a632bebde7adc1f5848a1b1bd5db1afd34dc1
d?????????    ? ?   ?      ?            ? 406e55155b139484cf781dd2c230a4d073ae5602
d?????????    ? ?   ?      ?            ? 407213d0c4cbd113937610b63586458b87650d7e
d?????????    ? ?   ?      ?            ? 40752024e22f37d76f170e9596279ffc5693a116
d?????????    ? ?   ?      ?            ? 407970469cfac5df8ba14f44cfa5ac6f3a0dcc12
d?????????    ? ?   ?      ?            ? 407be892abc6d49e0e741d5cda362f2d9ad8839b
d?????????    ? ?   ?      ?            ? 407d9c04e09caaa0e95d2c807b2e01a3ca534792
drwxr-xr-x    2 web web   39 Apr  7  2025 4088BtVZi7brXLOVA68zOx7wNnary4by5d0cc3120d291
d?????????    ? ?   ?      ?            ? 408fca2f835d65b9c65551c0e0a270fc371ccf5f
d?????????    ? ?   ?      ?            ? 40901a8dd4c4a5166398a36435653d005d840552
d?????????    ? ?   ?      ?            ? 4094adb6604e6e9e5c98ad5a921a977496b19e94
d?????????    ? ?   ?      ?            ? 4095ab6f6280f9aab8ded8e2adf14611173771bb
d?????????    ? ?   ?      ?            ? 409680df63ea927e6aee7be8db1d4c6084678336
d?????????    ? ?   ?      ?            ? 409d8e1e16868d988188a450a16f76b8b17c8d3d
drwxr-xr-x    2 web web   42 Apr  7  2025 409eVSQV9vmKn5fUuH2kjZR33n9zc0PD63a41e738bc74
drwxr-xr-x    2 web web   37 Apr  7  2025 40DWIWBxeRu5OJkpVjQZBvMdLFfjUFsL62a06d168d55c
drwxr-xr-x    2 web web   81 Apr  7  2025 40UWxxoSxj63qqFfq9Bgot0U2eOHZz4y5f992fdd57d3b
drwxr-xr-x    2 web web   92 Apr  7  2025 40UrTbodTZuFNEcnARNcVXMmPMkTFrXy65c9d4acddcf1
d?????????    ? ?   ?      ?            ? 40adbff831a513d9f603f4414be68189e61ce557
d?????????    ? ?   ?      ?            ? 40b4e169656849d9a11170b1c4b67253831b0cd5
d?????????    ? ?   ?      ?            ? 40b8c6fdd858524e45b581bfdcb6ad58646a1d6f
d?????????    ? ?   ?      ?            ? 40c5211c44bc670a2360a6bc05e1d3e1b217ca8f
d?????????    ? ?   ?      ?            ? 40d7b74163915093ed9adac94fc6e2ab47f49aa1
d?????????    ? ?   ?      ?            ? 40df5f28ca832c99656ad79b706ab599a4917d30
d?????????    ? ?   ?      ?            ? 40e333c86bd6818ee4516e1b4b771c5e57257700
d?????????    ? ?   ?      ?            ? 40e995d04da14b91fc0055d4530a8e41e53999cd
d?????????    ? ?   ?      ?            ? 40efdd2d1ad838c732e9ff6e20c836a6864f34d0
drwxr-xr-x    2 web web   49 Apr  7  2025 40l8eWQqOAnhijLC7wOLcmPRgZtKsPPt618b9f28a80e4
drwxr-xr-x    2 web web   34 Apr  7  2025 40o7TlxwaMrmYpTblboRfcJAbaQLWuDi5c177da168adc

On the 2nd glusterfs node: ls -lah

total 108K
drwxrwxrwx   43 web web 4.0K Sep 17  2024 .
drwsrwxr-x 2235 web web  60K Aug 27 16:21 ..
drwxrwxrwx    4 web web   99 Apr  7  2020 40089c1e3d58ad9f15fca15247d1bc857a668648
drwxrwxrwx    3 web web   82 May  3  2013 40141e87b59e7c47263dfdc2a190f2dd91e2da6a
drwxr-xr-x    4 web web   99 Dec 29  2021 40155b080cd78ac3700da35b552ceadf7edc9db2
drwxrwxrwx    3 web web   82 May  6  2013 401bff286b9d1476c073aec0c80ceaa12e69a1f0
drwxrwxrwx    3 web web   82 Jun 25  2018 40385af82942c344e3582740fd890aa89a1d0760
drwxrwxrwx    2 web web   54 Nov 20  2020 403a96f25d032e473c47dfc38d54cefaddee6783
drwxrwxrwx    3 web web   82 May  3  2013 403eea39ede573b950d21b4b739b67656b3c4db5
drwxr-xr-x    4 web web   99 Apr 30  2021 40471d039b9c7f6df257782d0c8d0d4e7c275f2f
drwxrwxrwx    4 web web   99 Oct  8  2020 4048ea5392ce7ed8c917c583a34a76e42b4f65d9
drwxrwxrwx    2 web web   36 May  4  2013 404cba113ee89f40e01cf73d3970d27b257a45d1
drwxrwxrwx    3 web web   82 May  4  2013 40558e9f3d07157aec921f997601d7fc3aae7779
drwxrwxrwx    3 web web   82 Jul 20  2018 4055db7948bfc2c6093a7829656725e63d5e8170
drwxrwxrwx    4 web web   99 Jan  8  2020 406a632bebde7adc1f5848a1b1bd5db1afd34dc1
drwxrwxrwx    2 web web   36 May  5  2013 406e55155b139484cf781dd2c230a4d073ae5602
drwxr-xr-x    4 web web   99 Sep 17  2024 407213d0c4cbd113937610b63586458b87650d7e
drwxr-xr-x    4 web web   99 Mar  8  2023 40752024e22f37d76f170e9596279ffc5693a116
drwxrwxrwx    3 web web   78 May  5  2013 407970469cfac5df8ba14f44cfa5ac6f3a0dcc12
drwxrwxrwx    4 web web   99 Oct  1  2020 407be892abc6d49e0e741d5cda362f2d9ad8839b
drwxrwxrwx    4 web web   99 Dec 21  2020 407d9c04e09caaa0e95d2c807b2e01a3ca534792
drwxr-xr-x    2 web web   39 Apr  7  2025 4088BtVZi7brXLOVA68zOx7wNnary4by5d0cc3120d291
drwxrwxrwx    3 web web   82 May  4  2013 408fca2f835d65b9c65551c0e0a270fc371ccf5f
drwxrwxrwx    3 web web   85 May  3  2013 40901a8dd4c4a5166398a36435653d005d840552
drwxrwxrwx    3 web web   82 May  4  2013 4094adb6604e6e9e5c98ad5a921a977496b19e94
drwxrwxrwx    4 web web   99 Apr 17  2019 4095ab6f6280f9aab8ded8e2adf14611173771bb
drwxr-xr-x    4 web web   99 Oct 20  2022 409680df63ea927e6aee7be8db1d4c6084678336
drwxrwxrwx    3 web web   82 May  3  2013 409d8e1e16868d988188a450a16f76b8b17c8d3d
drwxr-xr-x    2 web web   42 Apr  7  2025 409eVSQV9vmKn5fUuH2kjZR33n9zc0PD63a41e738bc74
drwxr-xr-x    2 web web   37 Apr  7  2025 40DWIWBxeRu5OJkpVjQZBvMdLFfjUFsL62a06d168d55c
drwxr-xr-x    2 web web   81 Apr  7  2025 40UWxxoSxj63qqFfq9Bgot0U2eOHZz4y5f992fdd57d3b
drwxr-xr-x    2 web web   92 Apr  7  2025 40UrTbodTZuFNEcnARNcVXMmPMkTFrXy65c9d4acddcf1
drwxrwxrwx    2 web web   35 May  4  2013 40adbff831a513d9f603f4414be68189e61ce557
drwxrwxrwx    3 web web   82 May  3  2013 40b4e169656849d9a11170b1c4b67253831b0cd5
drwxrwxrwx    3 web web   82 May  5  2013 40b8c6fdd858524e45b581bfdcb6ad58646a1d6f
drwxrwxrwx    4 web web   99 Dec 13  2019 40c5211c44bc670a2360a6bc05e1d3e1b217ca8f
drwxr-xr-x    4 web web   99 Feb 19  2024 40d7b74163915093ed9adac94fc6e2ab47f49aa1
drwxrwxrwx    3 web web   96 Aug 31  2017 40df5f28ca832c99656ad79b706ab599a4917d30
drwxrwxrwx    3 web web   82 May  5  2013 40e333c86bd6818ee4516e1b4b771c5e57257700
drwxrwxrwx    4 web web   99 Aug  8  2020 40e995d04da14b91fc0055d4530a8e41e53999cd
drwxrwxrwx    3 web web   86 May  6  2013 40efdd2d1ad838c732e9ff6e20c836a6864f34d0
drwxr-xr-x    2 web web   49 Apr  7  2025 40l8eWQqOAnhijLC7wOLcmPRgZtKsPPt618b9f28a80e4
drwxr-xr-x    2 web web   34 Apr  7  2025 40o7TlxwaMrmYpTblboRfcJAbaQLWuDi5c177da168adc

I've tried to unmount the folder from the client and mount it back up but the results are the same. What am I supposed to do to restore the metadata for the files? I won't be able to restore the 1st node for a while, so I need this to work with just the 2 nodes.

I am at your mercy. Thank you in advance !

[bertalanimre]

Update:

I've removed the broken brick from the cluster, now it looks to be fixed. However, a confirmation from you would be much appreciated.

[rafikc30]

@bertalanimre could you please share the client logs?

[bertalanimre]

I am not going to paste the whole log as it is way too big and may contain sensitive data. But this is what it is mostly full with:

[2025-10-28 13:13:42.392757 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-gv0-client-1: remote operation failed. [{path=(null)}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2025-10-28 13:13:42.395182 +0000] W [fuse-bridge.c:1052:fuse_entry_cbk] 0-glusterfs-fuse: 2585133: LOOKUP() /app/public/upload/1f/1fedd9876c9d6380e1d9eeea0049e35d9dadbf66 => -1 (No data available)
[2025-10-28 13:13:42.394967 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-gv0-client-1: remote operation failed. [{path=(null)}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2025-10-28 13:13:42.396226 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-gv0-client-1: remote operation failed. [{path=/app/public/upload/1f/1ff44f94b005af5bb7fe6270743114d2687dcb96}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2025-10-28 13:13:42.397030 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-gv0-client-1: remote operation failed. [{path=(null)}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2025-10-28 13:13:42.399460 +0000] W [fuse-bridge.c:1052:fuse_entry_cbk] 0-glusterfs-fuse: 2585136: LOOKUP() /app/public/upload/1f/1ff44f94b005af5bb7fe6270743114d2687dcb96 => -1 (No data available)
[2025-10-28 13:13:42.399333 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-gv0-client-1: remote operation failed. [{path=(null)}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2025-10-28 13:13:42.400516 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-gv0-client-1: remote operation failed. [{path=/app/public/upload/1f/1ff81170027ebcd6c52b287b6253789abc1e08f0}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2025-10-28 13:13:42.401302 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-gv0-client-1: remote operation failed. [{path=(null)}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2025-10-28 13:13:42.403676 +0000] W [fuse-bridge.c:1052:fuse_entry_cbk] 0-glusterfs-fuse: 2585139: LOOKUP() /app/public/upload/1f/1ff81170027ebcd6c52b287b6253789abc1e08f0 => -1 (No data available)
[2025-10-28 13:13:42.403549 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-gv0-client-1: remote operation failed. [{path=(null)}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2025-10-28 13:13:42.404727 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-gv0-client-1: remote operation failed. [{path=/app/public/upload/1f/1ff839b763363ab7ac53400525c79298481efa6c}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2025-10-28 13:13:42.405501 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-gv0-client-1: remote operation failed. [{path=(null)}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2025-10-28 13:13:42.407610 +0000] W [fuse-bridge.c:1052:fuse_entry_cbk] 0-glusterfs-fuse: 2585142: LOOKUP() /app/public/upload/1f/1ff839b763363ab7ac53400525c79298481efa6c => -1 (No data available)
[2025-10-28 13:13:42.407443 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-gv0-client-1: remote operation failed. [{path=(null)}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2025-10-28 13:13:42.408657 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-gv0-client-1: remote operation failed. [{path=/app/public/upload/1f/1ff960ebe84e7edacaf4fdc0150ae67c93f73e52}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2025-10-28 13:13:42.409471 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-gv0-client-1: remote operation failed. [{path=(null)}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2025-10-28 13:13:42.411948 +0000] W [fuse-bridge.c:1052:fuse_entry_cbk] 0-glusterfs-fuse: 2585145: LOOKUP() /app/public/upload/1f/1ff960ebe84e7edacaf4fdc0150ae67c93f73e52 => -1 (No data available)
[2025-10-28 13:13:42.411865 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-gv0-client-1: remote operation failed. [{path=(null)}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2025-10-28 13:13:42.412952 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-gv0-client-1: remote operation failed. [{path=/app/public/upload/1f/1ffdb7010330e9202136dfba2f7977144f7da9dc}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2025-10-28 13:13:42.413725 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-gv0-client-1: remote operation failed. [{path=(null)}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]
[2025-10-28 13:13:42.416153 +0000] W [fuse-bridge.c:1052:fuse_entry_cbk] 0-glusterfs-fuse: 2585148: LOOKUP() /app/public/upload/1f/1ffdb7010330e9202136dfba2f7977144f7da9dc => -1 (No data available)
[2025-10-28 13:13:42.415980 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-gv0-client-1: remote operation failed. [{path=(null)}, {gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data available}]

After I've removed the missing brick, these logs disappeared. No faulty logs at all since then.

Is there a specific line you are looking for?

[rafikc30]

@bertalanimre Do you have the gluster v info before you performed the remove-brick? I expect the lookup shouldn't have gone to failed bricks, even if, then failed with ENOTCONN, but here the lookup failed with ENODATA, which will result in healing, and it might have failed. By any chance, were the bricks on the faulty node running or connected to the client process?

[bertalanimre]

Gluster v info was shared before the removal. So the setup is a tad complex, but the TL;DR is I have 3 hardware, each has VMs running one instance for DB server, GlusterFS, ZabbixProxy and 2 of them also runs a Webserver. One of the webservers (Node1) have died and fallen out of the architecture. The remaining 2 hardware are running fine.
I was also expecting GLusterFS to recognize, that one brick is dead and reroute all the traffic from the webserver of Node2 to the remaining bricks, but clearly it did not do that. Neither via HTTPS, nor via SSH (as seen above). That is why I am confused.
I've tested GlusterFS for situations like this, but if I have to be honest, I did not test it with this amount of files and folder (around 1.5 milion).