Data Transfer Fails Between Vaults with Different Chunk Sizes

[S4tvara]

Describe the bug
If two vaults have different chunk sizes, data transfer fails.

To Reproduce
Steps to reproduce the behavior:

1. Create two vaults with no encryption but different chunk sizes.
2. Attempt to sync data between the vaults.
3. Observe that chunks fail to transfer.

Expected behavior
The vault should:

• Retrieve all data from the source vault.
• Reassemble it into a complete file.
• Re-chunk the file according to the target vault’s chunk size.
• Successfully store the chunks.

Screenshots

Additional context
This issue occurs because the transfer process does not handle differing chunk sizes. Implementing a reassemble-and-rechunk step will ensure reliable data transfer between vaults with different chunk configurations.

[Udayan853]

I tried reproducing this issue but got slightly different behavior.
The chunks are being transferred successfully, and I’m also able to retrieve the content from both vaults — even when their respective chunk_size values don’t match.

Steps I followed:

• Created two unencrypted vaults: dune1 (chunk_size: 2MB) and dune2 (chunk_size: 4MB)
• Added a sample.pdf (~5.8MB) to each, one at a time and called sietch sync for both cases
• Verified that the chunks transferred and retrieval via get worked in both cases

When both vaults share the same filepath, the chunks are transferred but become orphaned; still, retrieval remains successful.

Could it be that this issue is caused by something other than differing chunk_sizes?

[S4tvara]

@Udayan853 ,
That's interesting lemme try it again. Btw can you check the chunk size on dune2

[Udayan853]

[S4tvara]

@Udayan853,
Surprisingly, I was able to replicate it again xD.

The configurations for my vaults were:

• vault1 → No encryption, chunk size: 120MB
• vault2 → No encryption, chunk size: 100MB

The file I used was around 4gb.
Try transferring chunks from a bigger chunk size to smaller.

[Udayan853]

This could be one of two issues:

• The transfer of a single chunk is taking more than 30 seconds (though this seems unlikely).
• Discover runs for exactly 60 seconds before being cancelled. However, it does not shut down gracefully, so the stream is closed abruptly. As a result, the json decoder cannot retrieve the remaining data, leading to the observed error.

I tried replicating this with a ~2 GB file, which took roughly 30 seconds to transfer. This makes me believe the second scenario is the more probable cause.

[Udayan853]

This is odd, now when I try to transfer the files, the chunks get orphaned, and the file is not transferred
Its the same configuration as above, I deleted dune1 and dune2 and reinitialized them

[Udayan853]

On making dune1 listen for connections

[S4tvara]

@Udayan853,
weird lemme deep dive into this problem.
can you share the vault configs here?

[Udayan853]

[S4tvara]

This is odd, now when I try to transfer the files, the chunks get orphaned, and the file is not transferred Its the same configuration as above, I deleted dune1 and dune2 and reinitialized them

@Udayan853 , can you try this again but after exchanging keys using discover

[Udayan853]

Same result

[S4tvara]

@Udayan853 did you try to deep into why this was happening?

[Udayan853]

Honestly, I was waiting for your input to confirm whether the issue we’re seeing is the same.
In the meantime, I did look into it a bit — my initial guess is that it might be related to a yaml parsing issue (though I could be wrong).
I’m away from my system right now, so I’ll need a bit more time to dig into the specifics.

[Udayan853]

I've looked into this issue further.
My current hypothesis is that the transfer fails if both the sender and receiver don’t have each other listed as trusted peers.

When we run the sync and discover commands, should the vault config files for both the sender (running sync) and the receiver (running discover) be updated with the trusted peers?
At the moment, only the vault config for the instance running discover seems to be getting updated.

[S4tvara]

Discovery is the process through which vaults are able to exchange fingerprints and public keys.
if the vaults haven't discovered each other then they should not even attempt to sync data.

[Udayan853]

Ah, makes sense — so all I had to do was run discovery on both vaults and exchange fingerprints.

I guess we have reached a conclusion with following 2 issues:

• Gracefully closing the stream when transferring chunks, and
• Rechunking with a new size when transferring to a different vault

[S4tvara]

@Udayan853 I’ll dig deeper into this later, but if you want to go ahead and fix both issues before that, feel free! Just create the relevant issues n I’ll set up the labels.