Reading specific pieces only

[Diniboy1123]

Hi,

First of all, let me express my respect for maintaining such a huge project, it's insanely cool. On the other hand, I am looking for something a little similar to #426.

All I want is to be able to load a torrent from a file and grab the first piece only. Nothing more, nothing less. I expect stuff to fit in memory, therefore storing this on disk sounds like a waste of storage.

Do you think I could call some internal directly to retrieve a piece? I assume there isn't much of the concurrency needed that it would make sense to implement an entire storage backend for this. However I am not sure if the library would be suitable for this use-case.

Thanks

[anacrolix]

Yes, this implementation is ideal for exactly this. You want to open a reader either to the Torrent, or the Piece, if it's exposed there. Either way you're seeking to the part of the torrent with the piece you want and just reading it until you finish the piece. The client will handle ensuring the data is available.

Per the other issue, a caching storage backed by anything is fine, or you can store the piece data yourself with your own implementation. That would have the advantage you can discard everything you never intend to use.

You can also request just the piece from the network (there's a piece priority in the API) and use a reader with no readahead to ensure the client doesn't read other pieces, or if you have your own storage implementation, you can prioritise the piece then just access the data in the storage directly when it's finished.

I would just test reading the piece directly using defaults first until you prove the concept before bothering with the more advanced techniques above.

[Diniboy1123]

Thanks for getting back to me so quick with valuable feedback!

I actually got some parts working before opening the issue, it was stupid from me not to include the code in the first place... Find it here.

It works, though there are several issues that I have with this implementation. First is the need to manually call t.CancelPieces(0, info.NumPieces()). Due to the very well done concurrency of this lib, it can happen that the client starts requesting chunks that I don't need. It would be nice to straight away reject all pieces and manually enable certain ones (in my case only the first one as I only care about metadata).

Second is the need for a disk. I read about in memory storage implementations, but so far I find the storage API a little too complicated. That's totally a PEBKAC issue though and I will try to figure out how to implement this properly. The only memory storage implementation I found was from Elementum, but that's really outdated now and won't even work on the current API.

To sum it up, I was hoping that I can escape from most of the background model of the main API of this lib which makes sense for a full blown client, but for my use case as a library, I would love to be able to control what exactly gets downloaded without all the concurrency overhead. I am sensing from your input that that's not quite possible and you suggest that I sit down and write my memory storage backend, right?

Edit: now that I re-interpreted your response, I feel like the concurrent piece receiving might be a feature of the bittorrent network and I cannot exactly control what arrives to me, the peers will send pieces to me as soon as I connect. And I can only update priorities. Am I getting that right? Because in that case I understand now why would I need a storage backend with discarding functionality.

[anacrolix]

I'll get a longer answer later, but short answer is: No, it shouldn't request data until you ask, so it's likely the API isn't being used quite right. I'll check your sample later.

[anacrolix]

Commented on your sample. The code as I read it won't be downloading anything it doesn't need, subsequent to the readahead not going into the next file. You should be good to go with that code or close to.