Ability to add raw blocks / raw leaves

[ProximaNova]

ipwb currently cannot add data as raw blocks. Benefits of raw blocks: they are faster and use up less storage space.

Whenever ipwb adds something to ipfs it uses this python api: https://github.com/ipfs-shipyard/py-ipfs-http-client (ipfshttpclient). Documentation:
https://ipfs.ssi.eecc.de/ipfs/QmSfSEXrkYzsLz4adKJzVe9zj8dT8V6X9rkXdXCenV35pQ/docs/http_client_ref.html

Looking at this section:
https://ipfs.ssi.eecc.de/ipfs/QmSfSEXrkYzsLz4adKJzVe9zj8dT8V6X9rkXdXCenV35pQ/docs/http_client_ref.html#ipfshttpclient.Client.add_bytes

it says nothing about add_bytes() having an option to add raw blocks. Therefore, perhaps use a different thing which can add raw blocks, or modify the source code of that Python API to be able to do that. ipfshttpclient/client/__init__.py defines the add_bytes method/function. As I understand, ipfshttpclient is a wrapper or implementation that communicates with Kubo IPFS's core API. For example, that __init__.py file says

	def add_bytes(self, data: bytes, **kwargs):
		"""Adds a set of bytes as a file to IPFS.
[...]
		"""
		body, headers = multipart.stream_bytes(data, chunk_size=self.chunk_size)
		return self._client.request('/add', decoder='json',
		                            data=body, headers=headers, **kwargs)

	@utils.return_field('Hash')
	@base.returns_single_item(dict)

which looks like an API call. Raw blocks normally look like this: https://gateway.pinata.cloud/ipfs/bafkreibr3eo4ryibk6yueswcgepfybw33nwngs4z4vmr4ck2ky43j7cn64 (starts with "bafk"). Kubo ipfs command to create it: "ipfs add --cid-version=1 setup.cfg".

[ProximaNova]

Tracking it down:

File py-ipfs-http-client/ipfshttpclient/client/__init__.py --> add_bytes() --> self._client.request('/add', decoder='json',... --> imported file base.py --> self._client = http.build_client_sync(... --> imported file http.py --> def build_client_sync(...> ClientSyncBase[ty.Any]: --> imported file http_common.py --> class ClientSyncBase(ty.Generic[S], metaclass=abc.ABCMeta): --> said to be "An HTTP client for interacting with the IPFS daemon".

Details:

ipfshttpclient/client/__init__.py has a thing to define "base", something like "api/v0". IIRC, it has a thing which defines address, that'll be http://10.0.0.50:5001 for this example. The "add" endpoint in the HTTP/RPC API:

• Documentation: https://ipfs.hypha.coop/ipns/docs.ipfs.tech/reference/kubo/rpc/#api-v0-add
• Bash: https://10.0.0.50:5001/api/v0/add?$quiet_$quieter_$silent_$progress_$trickle_$onlyhash_$wrapwithdirectory_$chunker_$rawleaves_$nocopy_$fscache_$cidversion_$hash_$inline_$inlinelimit_$pin_$tofiles_$preservemode_$preservemtime_$mode_$mtime_$mtimensecs_

Conclusion:

It might be easy to modify python codebase ipfshttpclient to add that functionality. Something like self._client.request('/add?raw-leaves=true&pin=false'...

[ProximaNova]

It might be easy to modify python codebase ipfshttpclient to add that functionality. Something like self._client.request('/add?raw-leaves=true&pin=false'...

Luckily, it was that easy. As of writing this, I made two edits to https://github.com/ProximaNova/py-ipfs-http-client which are seen at:

1. ipfs-shipyard/py-ipfs-http-client@c191872
2. ipfs-shipyard/py-ipfs-http-client@acac2b5...44b4345

Neither are pulled into the official ipfs-shipyard/py-ipfs-http-client repo. It may not make since for them to be accepted into that as add() can add data as raw blocks. add_bytes() couldn't do that until I made those edits, which makes sense for this project (oduwsdl/ipwb). Explaining both commits:

1. c191872706e1118d2cd76ea326a2a8d580899353: makes add_bytes() only add raw blocks - for testing purposes (the next commit is more production-focused)
2. acac2b5d5251774b6b5cdff1fee0b45abbff5913...44b4345c715bc52d114dbf942a5bf041353f30fb: add_bytes() acts as it always has, function add_raw_bytes() was added and only that adds raw bytes/blocks.

[machawk1]

@ProximaNova Thanks for reporting and spearheading the issue w/ the library.

What is the use case for ipwb to add raw blocks/leaves? I'm unfamiliar and will do some reading on them but did not want a response to fall through the cracks.

Also pinging @ibnesayeed for his 2¢, as he might be a bit more familiar.

[ibnesayeed]

Thanks for looking into this. Being able to store raw blocks of desired sizes has other potential benefits of sub-resource deduplication. This ability has been available in the JavaScript library for a while, but when we worked on this piece of code earlier, the Python library did not have such features, so we opted for the easy route that works for the proof-of-concept. Our next-generation decentralized web archiving system called IPARO (not implemented yet) has some discussions around storing slices of objects strategically for better storage optimization.

That said, if the upstream library accepts your changes, we will be happy to see a PR in this repo to test the implications.

[machawk1]

@ProximaNova In lieu of using a fork of py-ipfs-http-client until this change can be added to the master repo (if accepted), would it be beneficial / worthwhile to implement it some other way that enhances the functionality of add_bytes(), say, intercepting requests to store raw blocks to not be routed through the module?

Alternatively, would there be a large amount of overhead to create a separate module that can be imported into ipwb to add this ability?

I am still not clear on the use case. @ibnesayeed mentioned de-duplication but I'd like this spelled out a little more in the context of raw blocks/leaves cf. current approach by py-ipfs-http-client.

[ProximaNova]

@ibnesayeed

That said, if the upstream library accepts your changes, we will be happy to see a PR in this repo to test the implications.

@machawk1

until this change can be added to the master repo (if accepted)

I could make a pull request for my unmerged edit at ipfs-shipyard/py-ipfs-http-client@44b4345 however it seems that that idea was already proposed and rejected. Maybe? This seems to be a better-programmed version of my edit to add_bytes(): https://github.com/ipfs-shipyard/py-ipfs-http-client/pull/287/files

That's pull request number 287 - ipfs-shipyard/py-ipfs-http-client#287 - titled "add optional arguments to add_str()" with OP="[...] I've added the optional arguments I think are relevant. If this looks good, I'm happy to add the same processing for add_json and add_bytes." Don't know why it wasn't accepted.

[ProximaNova]

^Update - I made a pull request to the upstream API py-ipfs-http-client (which is like a Python wrapper for IPFS's HTTP RPC API): "Make add_bytes() work with raw blocks" ipfs-shipyard/py-ipfs-http-client#334

Repo ipfs-shipyard/py-ipfs-http-client is looking a bit like abandonware: last commit was 2 years ago, (excluding mine) the last pull request or issue open was 1 year ago.