bench exits with error status 1 (libjudy related?)

[SuperFluffy]

Trying to compile simhash-cpp, I am running into a couple of issues. The first one is that I am unable to compile test.cpp. The issue is described already in #2.

The other issue is related to bench.cpp. It compiles, but when I am trying to run the executable, I get the following error message:

Inserting 10000000 hashes...
Running 40000000 queries...
Queries complete with 0 errors
There are 9997291 items in the table
File 'simhash.h', line 137: Judy1FreeArray(), JU_ERRNO_* == 9, ID == 113
zsh: exit 1     ./bench 10000000

I have used the following packages for compilation:

judy 1.0.5
g++ 4.9.2
Catch build 53

Any idea what is happening?

EDIT: The JU_ERRNO above corresponds to (from July.h):

// JU_ERRNO_CORRUPT occurs when Judy detects an impossible value in a Judy data
// structure:
//
// Note:  The Judy data structure contains some redundant elements that support
// this type of checking.

        JU_ERRNO_CORRUPT        = 9

[dlecocq]

We've not encountered that, but we wouldn't be heartbroken to ditch libJudy. What OS are you running?

[dlecocq]

Er. Nevermind. We've been able to repro that locally.

[SuperFluffy]

Any news on this one?

[janpom]

Hi, I encountered the same problem in my project and found out it happens when freeing the Judy array after storing too many elements in it. Here's a simple test: http://dpaste.com/34QBQ41

It fails on 10000 elements in my environment -- judy-1.0.5 and
$ uname -a
Linux localhost 3.13.0-46-generic #75-Ubuntu SMP Tue Feb 10 15:24:04 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Have you guys made any progress at your end? Thanks.

[zhang-w]

On my Macbook the ./test reports failures as well. I am using all the latest things.
Catch 1.1, Build 1.
Judy-1.0.5
G++ for Mac, which is
Target: x86_64-apple-darwin14.1.0
Thread model: posix

Here are the outputs from terminal.

PVGM50844878A:src i064503$ make clean
rm -rf driver test *.o
PVGM50844878A:src i064503$ make
g++ -g -O3 -Wall -o simhash.o -c simhash.cpp
g++ -g -O3 -Wall -o bench bench.cpp simhash.o -lJudy
g++ -g -O3 -Wall -o test test.cpp -Ideps/Catch/single_include
PVGM50844878A:src i064503$ ./test

test is a Catch v1.1 b1 host application.
Run with -? for options

-------------------------------------------------------------------------------
cyclic
  multiple rotation
-------------------------------------------------------------------------------
test.cpp:14
...............................................................................

test.cpp:39: FAILED:
  REQUIRE( Simhash::Cyclic<uint64_t>::rotate(0xABCDEFFFABCDEFFF, 64) == 0xABCDEFFFABCDEFFF )
with expansion:
  18446744073709551615 == 0xabcdefffabcdefff

-------------------------------------------------------------------------------
tokenizers
  strspn
-------------------------------------------------------------------------------
test.cpp:85
...............................................................................

test.cpp:120: FAILED:
  REQUIRE( (next == __null) )
with expansion:
  false

===============================================================================
test cases:  3 |  1 passed | 2 failed
assertions: 30 | 28 passed | 2 failed

[SuperFluffy]

Due to @janpom's mention, I checked bench.cpp with different values of uint64_t limit. It turns out, that with a value of 4148 or higher, I get the libjudy error:

% ./bench      
Inserting 4148 hashes...
Running 16592 queries...
Queries complete with 0 errors
There are 4123 items in the table
File 'simhash.h', line 137: Judy1FreeArray(), JU_ERRNO_* == 9, ID == 113
zsh: exit 1     ./bench

At 4147, the queries complete with errors, but the program hangs without completing (needed to sigterm it):

% ./bench
Inserting 4147 hashes...
Running 16588 queries...
Queries complete with 3 errors
^C
zsh: interrupt  ./bench

Below that (up until 4100), it sometimes passes with 0 errors, sometimes doesn't with 3 errors. But everytime it just hangs.

At 4099 and below, everything seems to work nicely:

% ./bench      
Inserting 4099 hashes...
Running 16396 queries...
Queries complete with 0 errors
There are 4098 items in the table

Odd stuff. :)

[b4hand]

This appears to be fallout from libjudy and its brokenness on newer versions of GCC. seomoz/simhash-py#8 discusses a workaround.

[pombredanne]

Since with 922891f judy arrays are not used anymore, this should be closed IMHO

[dlecocq]

Agreed. I lack permissions on this repo to do that anymore. @lindseyreno - do you know who (if anyone) at Moz is looking after this one these days?