Skip to content

[WIP] add experimental revive option for reproducible CIDs #6509

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
drbh wants to merge 2 commits into from
Closed

[WIP] add experimental revive option for reproducible CIDs #6509

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cf93cda
add experimental revive option
drbh Jul 13, 2019
4369ff7
update var names
drbh Jul 13, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 97 additions & 0 deletions core/commands/add.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ const (
hashOptionName = "hash"
inlineOptionName = "inline"
inlineLimitOptionName = "inline-limit"
reconstructOptionName = "reconstruct"
)

const adderOutChanSize = 8
Expand Down Expand Up @@ -129,6 +130,7 @@ You can now check what blocks have been created by:
cmds.StringOption(hashOptionName, "Hash function to use. Implies CIDv1 if not sha2-256. (experimental)").WithDefault("sha2-256"),
cmds.BoolOption(inlineOptionName, "Inline small blocks into CIDs. (experimental)"),
cmds.IntOption(inlineLimitOptionName, "Maximum block size to inline. (experimental)").WithDefault(32),
cmds.StringOption(reconstructOptionName, "Iterate through known params to reconstruct a CID from a file (experimental)"),
},
PreRun: func(req *cmds.Request, env cmds.Environment) error {
quiet, _ := req.Options[quietOptionName].(bool)
Expand Down Expand Up @@ -218,6 +220,101 @@ You can now check what blocks have been created by:

opts = append(opts, nil) // events option placeholder

revivalTargetCid, _ := req.Options[reconstructOptionName].(string)

var defaultOptions = []cmds.OptMap{
cmds.OptMap{
"inline-limit": 32,
"chunker": "size-262144",
"quiet": true,
"hash": "sha2-512",
"encoding": "json",
},
cmds.OptMap{
"inline-limit": 32,
"chunker": "size-262144",
"quiet": true,
"hash": "sha2-256",
"encoding": "json",
},
}

if revivalTargetCid != "" {

for _, defaultOpts := range defaultOptions {
special := req.Files
// fmt.Println("\n - - - - - ", index, "\n")
fmt.Println(defaultOpts)

inlineLimit, _ := defaultOpts[inlineLimitOptionName].(int)
chunker, _ := defaultOpts[chunkerOptionName].(string)
hashFunStr, _ := defaultOpts[hashOptionName].(string)
hashFunCode, ok := mh.Names[strings.ToLower(hashFunStr)]

if !ok {
return fmt.Errorf("unrecognized hash function: %s", strings.ToLower(hashFunStr))
}
myOpts := []options.UnixfsAddOption{
options.Unixfs.Hash(hashFunCode),
options.Unixfs.Inline(inline),
options.Unixfs.InlineLimit(inlineLimit),
options.Unixfs.Chunker(chunker),

options.Unixfs.Pin(dopin),
options.Unixfs.HashOnly(true),
options.Unixfs.FsCache(fscache),
options.Unixfs.Nocopy(nocopy),

options.Unixfs.Progress(progress),
options.Unixfs.Silent(silent),
}

if cidVerSet {
myOpts = append(myOpts, options.Unixfs.CidVersion(cidVer))
}

if rbset {
myOpts = append(myOpts, options.Unixfs.RawLeaves(rawblks))
Copy link
Copy Markdown
Member

@lidel lidel Jul 19, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caveat: RawLeaves are disabled by default when doing ipfs add --cid-version 0 (current default for CIDv0), but are enabled when user opts-in to CIDv1 (ipfs add --cid-version 0.

--reconstruct should see if CID is v0 or v1 and set proper default

}

if trickle {
myOpts = append(myOpts, options.Unixfs.Layout(options.TrickleLayout))
Copy link
Copy Markdown
Member

@lidel lidel Jul 19, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trickle DAG is created by ipfs files write by default, which complicates things: so we should always try enabling/disabling it as a fallback.

}

myOpts = append(myOpts, nil) // events option placeholder

fmt.Println(myOpts)

// var added int
addit := special.Entries()

for addit.Next() {
Copy link
Copy Markdown
Author

@drbh drbh Jul 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for addit.Next() doesn't initiate a loop on the second default settings. Seems like the entries are removed on each loop. Is there a way we can persist the file as we try different params?

Copy link
Copy Markdown
Member

@Stebalien Stebalien Sep 7, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically, you could buffer them all into memory. But that's really not something you want to do.

fmt.Println("In Loop")
events := make(chan interface{}, adderOutChanSize)
myOpts[len(myOpts)-1] = options.Unixfs.Events(events)

v, _ := api.Unixfs().Add(req.Context, addit.Node(), myOpts...)
h := enc.Encode(v.Cid())

// if err := res.Emit(&AddEvent{
// Name: "cat.jpg",
// Hash: h,
// Bytes: 0,
// Size: "378153",
// }); err != nil {
// return err
// }

fmt.Println("Hash: ", h)
if h == revivalTargetCid {
fmt.Println("SUCCESS!!")
return nil
}
}
}
return nil
}

var added int
addit := toadd.Entries()
for addit.Next() {
Expand Down