Intermittent test failures especially in test_search

[jayvdb]

Investigating the build failures on #57 , there are many intermittent errors, mostly in the test_search assertion added by #55.

The #57 build status were:

1. green
2. green
3. red - py2.7 - test_search
4. red - py3.5 - test_search
5. red - py3.5 - test_search and test_peers

I have done some Travis builds of master (78456e7), with results:

1. green
2. red - py35 - test_search
3. green
4. red - py27 - test_search

The test_peers failure has only occurred once so far.

$ nosetests --with-coverage tests
....F.F...
======================================================================
FAIL: Test finding peers.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/loklak/loklak_python_api/tests/test.py", line 77, in test_peers
    self.assertTrue('peers' in result)
AssertionError: False is not true
...

The logging is always correctly reporting:

requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): loklak.org
requests.packages.urllib3.connectionpool: DEBUG: "GET /api/search.json?count=18&query=doctor+who HTTP/1.1" 200 None

The test_search failures on Python 3.5 is always in the number of results:

$ nosetests --with-coverage tests
......F...
======================================================================
FAIL: test search result
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/jayvdb/loklak_python_api/tests/test.py", line 108, in test_search
    self.assertEqual(int(result['search_metadata']['count']), 18)
AssertionError: 17 != 18

https://travis-ci.org/loklak/loklak_python_api/jobs/159657573 is slightly different, with fewer returned results

$ nosetests --with-coverage tests
......F...
======================================================================
FAIL: Test search result.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/loklak/loklak_python_api/tests/test.py", line 110, in test_search
    self.assertEqual(int(result['search_metadata']['count']), 18)
AssertionError: 13 != 18

And this one only has 10 results:
https://travis-ci.org/loklak/loklak_python_api/jobs/159662252

======================================================================
FAIL: Test search result.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/loklak/loklak_python_api/tests/test.py", line 110, in test_search
    self.assertEqual(int(result['search_metadata']['count']), 18)
AssertionError: 10 != 18

On Python 2.7 it varies more, with higher results and other problems.

https://travis-ci.org/jayvdb/loklak_python_api/jobs/159741504 :

$ nosetests --with-coverage tests
......F...
======================================================================
FAIL: test search result
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/jayvdb/loklak_python_api/tests/test.py", line 105, in test_search
    self.assertTrue(len(result['statuses']) >= 1)
AssertionError: False is not true

https://travis-ci.org/loklak/loklak_python_api/jobs/159654156 :

$ nosetests --with-coverage tests
......F...
======================================================================
FAIL: Test search result.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/loklak/loklak_python_api/tests/test.py", line 109, in test_search
    int(result['search_metadata']['count']))
AssertionError: 24 != 14

[jayvdb]

On my second rebuild of #55 (2507e28) , it failed with

$ nosetests --with-coverage tests
......F...
======================================================================
FAIL: test search result
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/jayvdb/loklak_python_api/tests/test.py", line 108, in test_search
    self.assertEqual(int(result['search_metadata']['count']), 18)
AssertionError: 14 != 18

After another 10 rebuilds, it failed again:

$ nosetests --with-coverage tests
......F...
======================================================================
FAIL: test search result
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/jayvdb/loklak_python_api/tests/test.py", line 108, in test_search
    self.assertEqual(int(result['search_metadata']['count']), 18)
AssertionError: 10 != 18

Seems like something is wrong server side.

[jayvdb]

A simple way to reduce the number of failures ( & pretend it doesnt exist) is to assert the results are less than 18 hits.

[sudheesh001]

A simpler way to do this is by having the API responses be mocked rather than making an actual request. Would you want to pick this issue @jayvdb

[jayvdb]

Sorry, I would much rather be spending my time analyzing the bug and fixing it.

[jayvdb]

Continuing from #57 :

It looks like travis is failing because loklak's production api's on loklak.org return a 503.

@sudheesh001 , How did you arrive at the conclusion that it is returning a 503?

That conclusion seems inconsistent with the API returning 24 results instead of 18 at https://travis-ci.org/loklak/loklak_python_api/jobs/159654156 .

[sudheesh001]

I think this was with in the previous push when the server might be acting up. https://travis-ci.org/loklak/loklak_python_api/jobs/159662252

======================================================================
FAIL: Test finding peers.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/loklak/loklak_python_api/tests/test.py", line 77, in test_peers
    self.assertTrue('peers' in result)
AssertionError: False is not true
-------------------- >> begin captured logging << --------------------
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): loklak.org
requests.packages.urllib3.connectionpool: DEBUG: "GET /api/peers.json HTTP/1.1" 503 381
--------------------- >> end captured logging << ---------------------

And yes you're right it looks like it's returning 24, that seems like a server side problem since the server decides the result, the API only bring it to the developers, please open an issue on loklak/loklak_server for the same.

https://github.com/loklak/loklak_server