Reduce library size

[ianopolous]

The latest version of this has almost doubled the size to 60mb. This is way too much for a glorified rest client. For comparison the Java S3 client is ~3mb. Is there any way to reduce this by stripping out unused dependencies? Or just including the S3 sdk, and not all of amazon?

[Stebalien]

Is that compiled size or source code size? We're using amazon's s3 sdk.

[ianopolous]

Compiled size. I've also opened an issue on amazons go sdk about importing only S3 client and dependencies. I'm also referring to Amazon's S3 client. I think the problem here is it's including clients for all of amazon's ~50 services. Not sure why Go isn't stripping out unused components though during compilation.

[ianopolous]

Amazon have replied and they are already working it. Tracking issue here:
aws/aws-sdk-go-v2#444

So I think it's mostly just a matter of waiting for that to be released and importing that sdk instead.

[ianopolous]

@Stebalien Is there a size where would you consider including it in ipfs proper, like 3mb say? Or is even that too much to make every user pay. If only we could store the ipfs binary in ipld proper and selectively download the merkle linked modules on demand. :-)

[Stebalien]

I'll have to think about it. Honestly, go-ipfs is already pretty massive so 3MiB might be acceptable.

We may just want to ship multiple IPFS versions, one with additional supported datastores. Plugins were supposed to solve this issue, but they just don't work reliably.

[ianopolous]

A better solution to this is to just implement our own S3 request signer. It is quite simple (<300 lines of Java). My Java implementation is here:
https://github.com/Peergos/Peergos/blob/master/src/peergos/server/storage/S3Request.java

Then this should be < 1 KiB not 60 MiB, and thus could be included in ipfs directly without requiring a plugin.

[ianopolous]

I've checked that I had implemented all the calls used by this library, and I'd missed bulk delete. So I've added that and the result is still ~ 500 lines of very simple code. Zero dependencies. This would make a good small dev grant target, @momack2.