import/export raw buffer?

[dominictarr]

I'm working on a modular database project flumedb in which I am intending to use jsbloom to create a view to test whether a given item is within the database or not (etc)

I tested the import/export function you have provided which uses LZString, and unfortunately found that in certain cases it actually expands the size of the exported filter to be larger than the raw buffer! When the filter is either sparse or overfull it compresses well (because it's mostly 0s or mostly 1s) but when it's in the middle is when it compresses the worst.

I saw in the code https://github.com/cry/jsbloom/blob/master/bloom.js#L130 that you are just stringifying the buffer and then compressing it. This splits the buffer to comma separated integers, so that one byte becomes 2-4 bytes. While it should theoretically possible to compress this as well as another encoding, most compression algorithms are not that clever, so it helps if you give them an easier input.

Anyway, I want to be able to import/export the raw buffer, and want to ask about the best way to add this feature.

One option would be to pass in {encode, decode} functions which could then be used within the export/import functions. (this is a standard way for the level community https://github.com/level/levelup#custom-encodings ) the default encoding could then be your LZString, if you wanted to keep this backwards compatible.

A simple change, although less elegant, i think would be set bVector as the third argument.

It seems to me that it might also be important to serialize the other settings (items, probability) but maybe those can also be passed to {encode, decode}?

[cry]

Adjusting it to take encode and decode functions would be best in this situation I think, I'll create a new branch to implement this, unless you have progress you want to share?

I think settings can be passed as an extra argument if needed.

[cry]

@dominictarr I've committed 73df613 which introduces the exportImportInit function. From this, the user has the ability to set their own import/export functions to meet their needs, e.g. they may define their own compression algorithm and have full access to the internal variables such as HASH_ROUNDS or bVector to use as they see fit. If the user doesn't define their own function, the filter will automatically fallback on the old export/import functions. Does this approach fit your needs?

[dominictarr]

hmm, this doesn't look right to me. you just reassign the variables, but the returned object

jsbloom/bloom.js

Lines 157 to 158 in 73df613

	importData: importData,
	exportData: exportData

doesn't point to the variables, it points to the values the variables point at. what would work:

importData: function (raw, k) { return importData(raw, k) },
exportData: function () { return importData(bVector) }

in your code importData just uses a closure to see bVector the function that I pass in wont be able to see that unless you pass it as an argument, though.

[cry]

Hm, I see the issue with my code, the returned JSBloom object wont reference new methods. I'm not sure what your code is accomplishing though, wouldn't calling importData end in infinite recursion as its self referencing?

[dominictarr]

oh, sorry maybe I wasn't very clear, i was suggesting that importData: function (...) was a property on the returned object:
if you have something like:

//foo 1
function foo () { return 'foo' }
return {
  //foo 2
  foo: function () { return foo ()}
}

the scoped function //foo 1 is not the same reference as //foo 2. They keys of the object have the same name as the functions, but they are always obj.name which is different to name.

[cry]

I see, so the scoping is the issue. In this case, wouldn't it be best to allow the user to directly override the import/export functions on the object? i.e.

let jbloom = new JSBloom...

jbloom.exportData = function() {}

If my current understanding is incorrect, please let me know

[dominictarr]

That function will also have a different scope, and will be unable to see bVector.
Have a look at how douglass crockford makes private properties using closures
http://www.crockford.com/javascript/private.html

[cry]

I see what you're getting at, but as stated by Crockford:

Private and privileged members can only be made when an object is constructed. Public members can be added at any time.

What we're essentially doing is creating a user defined a privileged function. Looking at your code:

importData: function (raw, k) { return importData(raw, k) },

I just need some elaboration on how you would approach this problem.

[cry]

see 1d9d890, you can now use accessData to create your own import/export wrappers.

Example can be found in test for func. read/write is now possible for internal variables