Docker image build error

[marcorosa]

docker build --tag python_libstorj . produces the following warnings:

running build_ext
building '_python_libstorj' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/lib
creating build/temp.linux-x86_64-2.7/lib/ext
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c lib/ext/python_libstorj_wrap.cxx -o build/temp.linux-x86_64-2.7/lib/ext/python_libstorj_wrap.o
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c lib/ext/python_libstorj.cpp -o build/temp.linux-x86_64-2.7/lib/ext/python_libstorj.o
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
lib/ext/python_libstorj.cpp: In function 'void get_info_cb(uv_work_t*, int)':
lib/ext/python_libstorj.cpp:31:55: warning: deprecated conversion from string constant to 'char*' [-Wwrite-strings]
     PyObject_CallFunction(_handle, "ss", error, result);
                                                       ^
lib/ext/python_libstorj.cpp: In function 'void delete_bucket_cb(uv_work_t*, int)':
lib/ext/python_libstorj.cpp:62:50: warning: deprecated conversion from string constant to 'char*' [-Wwrite-strings]
     PyObject_CallFunction(_handle, "s", error_str);
                                                  ^
lib/ext/python_libstorj.cpp: In instantiation of 'bool error_and_status_check(ReqType*, char**) [with ReqType = json_request_t]':
lib/ext/python_libstorj.cpp:27:59:   required from here
lib/ext/python_libstorj.cpp:9:11: warning: unused variable '_error' [-Wunused-variable]
     char *_error = NULL;
           ^
lib/ext/python_libstorj.cpp: In instantiation of 'bool error_and_status_check(ReqType*, char**) [with ReqType = create_bucket_request_t]':
lib/ext/python_libstorj.cpp:41:72:   required from here
lib/ext/python_libstorj.cpp:9:11: warning: unused variable '_error' [-Wunused-variable]
lib/ext/python_libstorj.cpp: In instantiation of 'bool error_and_status_check(ReqType*, char**) [with ReqType = get_buckets_request_t]':
lib/ext/python_libstorj.cpp:72:70:   required from here
lib/ext/python_libstorj.cpp:9:11: warning: unused variable '_error' [-Wunused-variable]
lib/ext/python_libstorj.cpp: In instantiation of 'bool error_and_status_check(ReqType*, char**) [with ReqType = list_files_request_t]':
lib/ext/python_libstorj.cpp:100:69:   required from here
lib/ext/python_libstorj.cpp:9:11: warning: unused variable '_error' [-Wunused-variable]

[marcorosa]

After running the container I tried to launch ./build.sh again, but it produced the same error.
Therefore, all the unit tests failed

[marcorosa]

This is the output of the unit tests. I guess they are not supposed to fail, so there may be something wrong in my options.yml (even if I only changed username and password)

root@104b0e20d867:/python_libstorj# python -m unittest discover
FEEEEEEEEEEEsE
======================================================================
ERROR: test_create_bucket_without_callback (tests.test_storj_env.TestCreateDestroy)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/python_libstorj/tests/test_storj_env.py", line 49, in test_create_bucket_without_callback
    self.assertEqual(bucket['name'], bucket_name)
TypeError: 'NoneType' object has no attribute '__getitem__'

======================================================================
ERROR: test_delete_bucket_with_callback (tests.test_storj_env.TestCreateDestroy)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/python_libstorj/tests/test_storj_env.py", line 70, in test_delete_bucket_with_callback
    bucket_name = self.get_bucket_id('python_libstorj-test')
  File "/python_libstorj/tests/test_storj_env.py", line 21, in get_bucket_id
    return next(bucket['id'] for bucket in buckets if bucket['name'] == name)
  File "/python_libstorj/tests/test_storj_env.py", line 21, in <genexpr>
    return next(bucket['id'] for bucket in buckets if bucket['name'] == name)
  File "/python_libstorj/lib/storj_env.py", line 67, in handle_
    raise Exception(error)
Exception: Couldn't resolve host name

======================================================================
ERROR: test_delete_bucket_without_callback (tests.test_storj_env.TestCreateDestroy)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/python_libstorj/tests/test_storj_env.py", line 65, in test_delete_bucket_without_callback
    bucket_name = self.get_bucket_id('python_libstorj-test2')
  File "/python_libstorj/tests/test_storj_env.py", line 21, in get_bucket_id
    return next(bucket['id'] for bucket in buckets if bucket['name'] == name)
  File "/python_libstorj/tests/test_storj_env.py", line 21, in <genexpr>
    return next(bucket['id'] for bucket in buckets if bucket['name'] == name)
  File "/python_libstorj/lib/storj_env.py", line 67, in handle_
    raise Exception(error)
Exception: Couldn't resolve host name

======================================================================
ERROR: test_get_info_with_callback (tests.test_storj_env.TestGetInfo)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/python_libstorj/tests/test_storj_env.py", line 37, in test_get_info_with_callback
    self.env.get_info(callback)
  File "/python_libstorj/lib/storj_env.py", line 51, in get_info
    return results[0]
IndexError: list index out of range

======================================================================
ERROR: test_get_info_without_callback (tests.test_storj_env.TestGetInfo)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/python_libstorj/tests/test_storj_env.py", line 26, in test_get_info_without_callback
    info = self.env.get_info()
  File "/python_libstorj/lib/storj_env.py", line 51, in get_info
    return results[0]
IndexError: list index out of range

======================================================================
ERROR: test_list_buckets_with_callback (tests.test_storj_env.TestListBuckets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/python_libstorj/tests/test_storj_env.py", line 110, in test_list_buckets_with_callback
    self.env.list_buckets(callback)
  File "/python_libstorj/lib/storj_env.py", line 71, in list_buckets
    return results[0]
IndexError: list index out of range

======================================================================
ERROR: test_list_buckets_with_callback (tests.test_storj_env.TestListBuckets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/python_libstorj/tests/test_storj_env.py", line 87, in tearDown
    self.env.delete_bucket(self.bucket['id'])
TypeError: 'NoneType' object has no attribute '__getitem__'

======================================================================
ERROR: test_list_buckets_without_callback (tests.test_storj_env.TestListBuckets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/python_libstorj/tests/test_storj_env.py", line 93, in test_list_buckets_without_callback
    buckets = self.env.list_buckets()
  File "/python_libstorj/lib/storj_env.py", line 71, in list_buckets
    return results[0]
IndexError: list index out of range

======================================================================
ERROR: test_list_buckets_without_callback (tests.test_storj_env.TestListBuckets)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/python_libstorj/tests/test_storj_env.py", line 87, in tearDown
    self.env.delete_bucket(self.bucket['id'])
TypeError: 'NoneType' object has no attribute '__getitem__'

======================================================================
ERROR: test_list_files_with_callback (tests.test_storj_env.TestListFiles)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/python_libstorj/tests/test_storj_env.py", line 124, in setUp
    upload_file_command = 'storj upload-file %s %s' % (self.bucket['id'], file_path)
TypeError: 'NoneType' object has no attribute '__getitem__'

======================================================================
ERROR: test_list_files_without_callback (tests.test_storj_env.TestListFiles)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/python_libstorj/tests/test_storj_env.py", line 124, in setUp
    upload_file_command = 'storj upload-file %s %s' % (self.bucket['id'], file_path)
TypeError: 'NoneType' object has no attribute '__getitem__'

======================================================================
ERROR: test_store_file_without_callback (tests.test_storj_env.TestStoreFile)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/python_libstorj/tests/test_storj_env.py", line 159, in tearDown
    self.bucket = self.env.delete_bucket(self.bucket['id'])
TypeError: 'NoneType' object has no attribute '__getitem__'

======================================================================
FAIL: test_create_bucket_with_callback (tests.test_storj_env.TestCreateDestroy)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/python_libstorj/tests/test_storj_env.py", line 61, in test_create_bucket_with_callback
    self.assertEqual(error, None)
AssertionError: "Couldn't resolve host name" != None

----------------------------------------------------------------------
Ran 11 tests in 1.432s

FAILED (failures=1, errors=12, skipped=1)

[bryanchriswhite]

@marcorosa this is actually great! The test suite is not passing for a couple of reasons, primarily because the test suite is still WIP. I took a detour in the middle of binding upload/download & reacting to a breaking change in libstorj to add tests for failure scenarios.

These exception tests should make things much easier to debug when people try to get the repo running.

Also, those warnings shouldn't prevent things from building and running. If you see errors in c compilation output, that might be another story.

So it seems like you're env is working. You should be able to run some python in the python_libstorj container (e.g. via volume --using docker-sompose.4) that imports the python_libstorj module:

# from project root:
import lib.ext.python_libstorj as pystorj

# from `./lib/` (if you're adding to the lib, for example):
from ext import python_libstorj as pystorj

One more thing

There were a couple tests that were surfacing "Couldn't resolve host name", which might be an indication that your bridge container (docker-compose.yml:10) is either:

• Not running and/or crashing
  You can check what your composition is doing with a docker-compose ps. Look for a container called pythonlibstorj_bridge_1; the container name is based on the service name, so if you changed the service to something other than bridge, the container name will change accordingly).

  If the container is running you can docker-compose exec bridge /bin/bash to get a shell on the bridge to poke around. There are a couple of things you could check:

  # check if the bridge server is responding to http requests
  curl localhost:6382
  
  # look at the logs for all pm2 services
  pm2 logs -f

  If it's not running, you can change the service's command to /bin/sleep infinity to ensure it doesn't crash on boot. You can then run the composition again and in another shell you can docker-compose exec bridge /bin/bash to debug; see storj/integration.

• Not being resolved via DNS properly
  If you're using a docker-compose.override.yml, ensure the links property for the service(s) you're using the python lib (and/or the storj ) from, include - bridge as seen in docker-compose.yml:

  Alternatively, if you need to use an "alias" (docker-compose links) other than bridge, ensure that:

  • the service in the links item matches the services item name; e.g.:

    # docker-compose.yml
    ...
    services:
      python_libstorj:
        ...
        links:
          - alt_bridge_service_name:alt_alias_and_dns_name
      alt_bridge: ...
    

  • and that the host in your options.yml matches the "alias" (alt_alias_and_dns_name in the example above); docker-compose does dns resolution for you within the virtual network for that composition.

[bryanchriswhite]

As of ded51f0, my test output looks like this in the docker compose env described by the readme:

root@435d7a6c58d6:/python_libstorj# python -m unittest discover
......
----------------------------------------------------------------------
Ran 6 tests in 0.203s

OK

I would be curious to know if you are able to reproduce this by checking out that commit. I'm still catching up on error handling in the test suite, but after that one thing I'm going to have to add is a step to the env build process to automate user creation and/or update the readme to document how to add users manually (probably using libstorj cli).

[bryanchriswhite]

^^ related: storj-archived/libstorj#435

[bryanchriswhite]

It's also noteworthy that currently, to use the libstorj cli in the docker-compose env, you'll need to set your STORJ_BRIDGE env var to http://bridge:6382 (assuming bridge service is still called bridge) so that the cli hits the local bridge.

[bryanchriswhite]

^^ as of 9752f88 this is handled by docker-compose.yml's environment section for the docker-compose env.

So to update an existing docker composition you would just do: docker-compose build python_libstorj, then run it like usual.

For using docker outside of docker-compose, there are some ARG and ENV lines that can be uncommented from the dockerfile.

[marcorosa]

Tests with repo at ded51f0, both with docker and docker-compose environments:

root@b72521d50595:/python_libstorj# python -m unittest discover
....EE
======================================================================
ERROR: test_get_info_with_callback_success (tests.test_storj_env.TestGetInfoSuccess)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/python_libstorj/tests/test_storj_env.py", line 72, in test_get_info_with_callback_success
    self.env.get_info(callback)
  File "/python_libstorj/lib/storj_env.py", line 66, in get_info
    return self._error_check(results)
  File "/python_libstorj/lib/storj_env.py", line 39, in _error_check
    raise result
Exception: Couldn't connect to server

======================================================================
ERROR: test_get_info_without_callback_success (tests.test_storj_env.TestGetInfoSuccess)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/python_libstorj/tests/test_storj_env.py", line 61, in test_get_info_without_callback_success
    info = self.env.get_info()
  File "/python_libstorj/lib/storj_env.py", line 66, in get_info
    return self._error_check(results)
  File "/python_libstorj/lib/storj_env.py", line 39, in _error_check
    raise result
Exception: Couldn't connect to server

----------------------------------------------------------------------
Ran 6 tests in 0.347s

FAILED (errors=2)

[marcorosa]

At the latest version of branch master, more tests fail (because more tests are implemented, actually), and all raising the same exception (the same as before, i.e., Exception: Couldn't connect to server)

[bryanchriswhite]

If you're just using docker, you can just uncomment Dockerfile:29-30 and then you don't have to worry about running a bridge locally.

If you do that though, you'll need to set your options.yml to point to production as well, weekend your production storj user credentials. You'll probably want to do a storj import-keys also, so you can use the libstorj cli.

[marcorosa]

My bad: with Dockerfile environment varriables everything works. I change the bridge protocol to https and the host to api.storj.io, but I forgot to change the port to 443.
Commit ded51f0 executes all the tests successfully, whereas the current master raises some errors:

root@81ee2e758491:/python_libstorj# python -m unittest discover
..............FF..EE
======================================================================
ERROR: test_list_files_with_callback_success (tests.test_storj_env.TestListFilesSuccess)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/python_libstorj/tests/test_storj_env.py", line 241, in setUp
    self.bucket = self.env.create_bucket('python_libstorj-test4')
  File "/python_libstorj/lib/storj_env.py", line 120, in create_bucket
    return self._error_check(results)
  File "/python_libstorj/lib/storj_env.py", line 53, in _error_check
    raise data
Exception: Unknown error

======================================================================
ERROR: test_list_files_without_callback_success (tests.test_storj_env.TestListFilesSuccess)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/python_libstorj/tests/test_storj_env.py", line 241, in setUp
    self.bucket = self.env.create_bucket('python_libstorj-test4')
  File "/python_libstorj/lib/storj_env.py", line 120, in create_bucket
    return self._error_check(results)
  File "/python_libstorj/lib/storj_env.py", line 53, in _error_check
    raise data
Exception: Unknown error

======================================================================
FAIL: test_list_buckets_with_callback (tests.test_storj_env.TestListBucketsSuccess)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/python_libstorj/tests/test_storj_env.py", line 211, in test_list_buckets_with_callback
    self.assertEqual(len(buckets), 1)
AssertionError: 4 != 1

======================================================================
FAIL: test_list_buckets_without_callback (tests.test_storj_env.TestListBucketsSuccess)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/python_libstorj/tests/test_storj_env.py", line 192, in test_list_buckets_without_callback
    self.assertEqual(len(buckets), 1)
AssertionError: 4 != 1

----------------------------------------------------------------------
Ran 20 tests in 16.488s

FAILED (failures=2, errors=2)

[bryanchriswhite]

Great! The failing tests have setUp behavior that tries to upload a file using the libstorj cli. So those would fail if you aren't able to manually storj cp ... or storj upload-file ....

Also, the tests failing with AssetionError: 4 != 1 are currently implemented by counting the number of buckets or files - so those will fall if you already have some buckets and/or files on the bridge you're talking to (production, in this case). Those are, unfortunately, the more important tests, but they're implementation will be improved soon as more bindings get done.