Number of notebooks with EG

[mihirkapadiap]

Using Jupyter Enterprise Gateway with Swarm resource manager.
Connected notebook server (using (elyra/nb2kg)) to EG.

Tried to open 150 python_on_docker kernel notebooks it works fine.
However on trying to open 200 notebook it starts throwing lots of error.
[E 22:35:37.263 NotebookApp] Exception writing message to websocket:
[E 22:37:18.007 NotebookApp] Exception writing message to websocket:
[E 22:37:18.009 NotebookApp] Exception writing message to websocket:

Has anyone tested/tried load test, testing how many notebooks/load EG can support?

Environment

• Enterprise Gateway Version 2.0.0.dev3
• Platform: Docker Swarm

[kevin-bates]

@mihirkapadiap - thanks for opening this issue. I believe sites have supported more notebooks, but I'm not sure at what point additional EG servers get configured.

Since the messages you present are logged on the client side, can you provide the log from EG?

And are there any details that follow each of the Exception writing message to websocket: messages?

[mihirkapadiap]

Hi Kevin,

Getting socket timeout (it worked fine till 168 notebooks or so thereafter it started throwing error.
Also it restarted Jupyter Enterprise Gateway.

I start EG as:

docker stack deploy -c docker-compose.yml enterprise-gateway

Notebook server:

docker run -t --rm
-e KG_URL='http://<EG_HOST>:8888'
-e KG_HTTP_USER=guest
-e KG_HTTP_PASS=guest-password
-p 9999:8888
-e VALIDATE_KG_CERT='no'
-e LOG_LEVEL=DEBUG
-e KG_REQUEST_TIMEOUT=60
-e KG_CONNECT_TIMEOUT=60
-v ${HOME}/notebooks/:/tmp/notebooks
-w /tmp/notebooks
elyra/nb2kg

Removed prefix (enterprise-gateway_enterprise-gateway.1.8r05e9z6m4zm@clv235sl-c342e3) from log lines!

|[D2019-09-0317:49:26.703EnterpriseGatewayApp]activityon9715ded9-f6da-4627-ab12-ec8647a4e70d:execute_input
|[D2019-09-0317:49:26.718EnterpriseGatewayApp]activityon8d53e04e-a0ed-4b90-85a0-22a43acb7dfc:status
|[D2019-09-0317:49:26.821EnterpriseGatewayApp]activityon0da31c85-7f68-487a-aa6a-5ab8cfa81879:execute_result
|[E19090317:50:31ioloop:909]Exceptionincallback<boundmethodKernelRestarter.pollof<jupyter_client.ioloop.restarter.IOLoopKernelRestarterobjectat0x7f449c22d2e8>>
|Traceback(mostrecentcalllast):
|File"/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py",line384,in_make_request
|six.raise_from(e,None)
|File"",line2,inraise_from
|File"/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py",line380,in_make_request
|httplib_response=conn.getresponse()
|File"/opt/conda/lib/python3.7/http/client.py",line1321,ingetresponse
|response.begin()
|File"/opt/conda/lib/python3.7/http/client.py",line296,inbegin
|version,status,reason=self._read_status()
|File"/opt/conda/lib/python3.7/http/client.py",line257,in_read_status
|line=str(self.fp.readline(_MAXLINE+1),"iso-8859-1")
|File"/opt/conda/lib/python3.7/socket.py",line589,inreadinto
|returnself._sock.recv_into(b)
|socket.timeout:timedout
|
|Duringhandlingoftheaboveexception,anotherexceptionoccurred:
|
|Traceback(mostrecentcalllast):
|File"/opt/conda/lib/python3.7/site-packages/requests/adapters.py",line449,insend
|timeout=timeout
|File"/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py",line638,inurlopen
|_stacktrace=sys.exc_info()[2])
|File"/opt/conda/lib/python3.7/site-packages/urllib3/util/retry.py",line368,inincrement
|raisesix.reraise(type(error),error,_stacktrace)
|File"/opt/conda/lib/python3.7/site-packages/urllib3/packages/six.py",line686,inreraise
|raisevalue
|File"/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py",line600,inurlopen
|chunked=chunked)
|File"/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py",line386,in_make_request
|self._raise_timeout(err=e,url=url,timeout_value=read_timeout)
|File"/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py",line306,in_raise_timeout
|raiseReadTimeoutError(self,url,"Readtimedout.(readtimeout=%s)"%timeout_value)
|urllib3.exceptions.ReadTimeoutError:UnixHTTPConnectionPool(host='localhost',port=None):Readtimedout.(readtimeout=60)
|
|Duringhandlingoftheaboveexception,anotherexceptionoccurred:
|
|Traceback(mostrecentcalllast):
|File"/opt/conda/lib/python3.7/site-packages/tornado/ioloop.py",line907,in_run
|returnself.callback()
|File"/opt/conda/lib/python3.7/site-packages/jupyter_client/restarter.py",line93,inpoll
|ifnotself.kernel_manager.is_alive():
|File"/opt/conda/lib/python3.7/site-packages/jupyter_client/manager.py",line453,inis_alive
|ifself.kernel.poll()isNone:
|File"/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/processproxies/container.py",line112,inpoll
|container_status=self.get_container_status(None)
|File"/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/processproxies/docker_swarm.py",line74,inget_container_status
|task=self._get_task()
|File"/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/processproxies/docker_swarm.py",line58,in_get_task
|tasks=service.tasks(filters={'desired-state':'running'})
|File"/opt/conda/lib/python3.7/site-packages/docker/models/services.py",line54,intasks
|returnself.client.api.tasks(filters=filters)
|File"/opt/conda/lib/python3.7/site-packages/docker/utils/decorators.py",line34,inwrapper
|returnf(self,*args,**kwargs)
|File"/opt/conda/lib/python3.7/site-packages/docker/api/service.py",line358,intasks
|returnself._result(self._get(url,params=params),True)
|File"/opt/conda/lib/python3.7/site-packages/docker/utils/decorators.py",line46,ininner
|returnf(self,*args,**kwargs)
|File"/opt/conda/lib/python3.7/site-packages/docker/api/client.py",line230,in_get
|returnself.get(url,**self._set_request_timeout(kwargs))
|File"/opt/conda/lib/python3.7/site-packages/requests/sessions.py",line546,inget
|returnself.request('GET',url,**kwargs)
|File"/opt/conda/lib/python3.7/site-packages/requests/sessions.py",line533,inrequest
|resp=self.send(prep,**send_kwargs)
|File"/opt/conda/lib/python3.7/site-packages/requests/sessions.py",line646,insend
|r=adapter.send(request,**kwargs)
|File"/opt/conda/lib/python3.7/site-packages/requests/adapters.py",line529,insend
|raiseReadTimeout(e,request=request)
|requests.exceptions.ReadTimeout:UnixHTTPConnectionPool(host='localhost',port=None):Readtimedout.(readtimeout=60)
|[E19090317:52:46ioloop:909]Exceptionincallback<boundmethodKernelRestarter.pollof<jupyter_client.ioloop.restarter.IOLoopKernelRestarterobjectat0x7f449c744710>>
|Traceback(mostrecentcalllast):
|File"/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py",line384,in_make_request
|six.raise_from(e,None)
|File"",line2,inraise_from
|File"/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py",line380,in_make_request
|httplib_response=conn.getresponse()
|File"/opt/conda/lib/python3.7/http/client.py",line1321,ingetresponse
|response.begin()
|File"/opt/conda/lib/python3.7/http/client.py",line296,inbegin
|version,status,reason=self._read_status()
|File"/opt/conda/lib/python3.7/http/client.py",line257,in_read_status
|line=str(self.fp.readline(_MAXLINE+1),"iso-8859-1")
|File"/opt/conda/lib/python3.7/socket.py",line589,inreadinto
|returnself._sock.recv_into(b)
|socket.timeout:timedout
|
|Duringhandlingoftheaboveexception,anotherexceptionoccurred:
|
|Traceback(mostrecentcalllast):
|File"/opt/conda/lib/python3.7/site-packages/requests/adapters.py",line449,insend
|timeout=timeout
|File"/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py",line638,inurlopen
|_stacktrace=sys.exc_info()[2])
|File"/opt/conda/lib/python3.7/site-packages/urllib3/util/retry.py",line368,inincrement
|raisesix.reraise(type(error),error,_stacktrace)
|File"/opt/conda/lib/python3.7/site-packages/urllib3/packages/six.py",line686,inreraise
|raisevalue
|File"/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py",line600,inurlopen
|chunked=chunked)
|File"/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py",line386,in_make_request
|self._raise_timeout(err=e,url=url,timeout_value=read_timeout)
|File"/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py",line306,in_raise_timeout
|raiseReadTimeoutError(self,url,"Readtimedout.(readtimeout=%s)"%timeout_value)
|urllib3.exceptions.ReadTimeoutError:UnixHTTPConnectionPool(host='localhost',port=None):Readtimedout.(readtimeout=60)
|
|Duringhandlingoftheaboveexception,anotherexceptionoccurred:
|
|Traceback(mostrecentcalllast):
|File"/opt/conda/lib/python3.7/site-packages/tornado/ioloop.py",line907,in_run
|returnself.callback()
|File"/opt/conda/lib/python3.7/site-packages/jupyter_client/restarter.py",line93,inpoll
|ifnotself.kernel_manager.is_alive():
|File"/opt/conda/lib/python3.7/site-packages/jupyter_client/manager.py",line453,inis_alive
|ifself.kernel.poll()isNone:
|File"/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/processproxies/container.py",line112,inpoll
|container_status=self.get_container_status(None)
|File"/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/processproxies/docker_swarm.py",line74,inget_container_status
|task=self._get_task()
|File"/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/processproxies/docker_swarm.py",line56,in_get_task
|service=self._get_service()
|File"/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/processproxies/docker_swarm.py",line40,in_get_service
|services=client.services.list(filters={'label':'kernel_id='+self.kernel_id})
|File"/opt/conda/lib/python3.7/site-packages/docker/models/services.py",line269,inlist
|forsinself.client.api.services(**kwargs)
|File"/opt/conda/lib/python3.7/site-packages/docker/utils/decorators.py",line34,inwrapper
|returnf(self,*args,**kwargs)
|File"/opt/conda/lib/python3.7/site-packages/docker/api/service.py",line284,inservices
|returnself._result(self._get(url,params=params),True)
|File"/opt/conda/lib/python3.7/site-packages/docker/utils/decorators.py",line46,ininner
|returnf(self,*args,**kwargs)
|File"/opt/conda/lib/python3.7/site-packages/docker/api/client.py",line230,in_get
|returnself.get(url,**self._set_request_timeout(kwargs))
|File"/opt/conda/lib/python3.7/site-packages/requests/sessions.py",line546,inget
|returnself.request('GET',url,**kwargs)
|File"/opt/conda/lib/python3.7/site-packages/requests/sessions.py",line533,inrequest
|resp=self.send(prep,**send_kwargs)
|File"/opt/conda/lib/python3.7/site-packages/requests/sessions.py",line646,insend
|r=adapter.send(request,**kwargs)
|File"/opt/conda/lib/python3.7/site-packages/requests/adapters.py",line529,insend
|raiseReadTimeout(e,request=request)
|requests.exceptions.ReadTimeout:UnixHTTPConnectionPool(host='localhost',port=None):Readtimedout.(readtimeout=60)
|[E19090317:53:50ioloop:909]Exceptionincallback<boundmethodKernelRestarter.pollof<jupyter_client.ioloop.restarter.IOLoopKernelRestarterobjectat0x7f449c034d30>>
|Traceback(mostrecentcalllast):
|File"/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py",line384,in_make_request
|six.raise_from(e,None)
|File"",line2,inraise_from
|File"/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py",line380,in_make_request
|httplib_response=conn.getresponse()
|File"/opt/conda/lib/python3.7/http/client.py",line1321,ingetresponse
|response.begin()
|File"/opt/conda/lib/python3.7/http/client.py",line296,inbegin
|version,status,reason=self._read_status()
|File"/opt/conda/lib/python3.7/http/client.py",line257,in_read_status
|line=str(self.fp.readline(_MAXLINE+1),"iso-8859-1")
|File"/opt/conda/lib/python3.7/socket.py",line589,inreadinto
|returnself._sock.recv_into(b)
|socket.timeout:timedout
|
|Duringhandlingoftheaboveexception,anotherexceptionoccurred:
|
|Traceback(mostrecentcalllast):
|File"/opt/conda/lib/python3.7/site-packages/requests/adapters.py",line449,insend
|timeout=timeout
|File"/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py",line638,inurlopen
|_stacktrace=sys.exc_info()[2])
|File"/opt/conda/lib/python3.7/site-packages/urllib3/util/retry.py",line368,inincrement
|raisesix.reraise(type(error),error,_stacktrace)
|File"/opt/conda/lib/python3.7/site-packages/urllib3/packages/six.py",line686,inreraise
|raisevalue
|File"/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py",line600,inurlopen
|chunked=chunked)
|File"/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py",line386,in_make_request
|self._raise_timeout(err=e,url=url,timeout_value=read_timeout)
|File"/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py",line306,in_raise_timeout
|raiseReadTimeoutError(self,url,"Readtimedout.(readtimeout=%s)"%timeout_value)
|urllib3.exceptions.ReadTimeoutError:UnixHTTPConnectionPool(host='localhost',port=None):Readtimedout.(readtimeout=60)
|
|Duringhandlingoftheaboveexception,anotherexceptionoccurred:
|
|Traceback(mostrecentcalllast):
|File"/opt/conda/lib/python3.7/site-packages/tornado/ioloop.py",line907,in_run
|returnself.callback()
|File"/opt/conda/lib/python3.7/site-packages/jupyter_client/restarter.py",line93,inpoll
|ifnotself.kernel_manager.is_alive():
|File"/opt/conda/lib/python3.7/site-packages/jupyter_client/manager.py",line453,inis_alive
|ifself.kernel.poll()isNone:
|File"/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/processproxies/container.py",line112,inpoll
|container_status=self.get_container_status(None)
|File"/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/processproxies/docker_swarm.py",line74,inget_container_status
|task=self._get_task()
|File"/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/processproxies/docker_swarm.py",line56,in_get_task
|service=self._get_service()
|File"/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/processproxies/docker_swarm.py",line40,in_get_service
|services=client.services.list(filters={'label':'kernel_id='+self.kernel_id})
|File"/opt/conda/lib/python3.7/site-packages/docker/models/services.py",line269,inlist
|forsinself.client.api.services(**kwargs)
|File"/opt/conda/lib/python3.7/site-packages/docker/utils/decorators.py",line34,inwrapper
|returnf(self,*args,**kwargs)
|File"/opt/conda/lib/python3.7/site-packages/docker/api/service.py",line284,inservices
|returnself._result(self._get(url,params=params),True)
|File"/opt/conda/lib/python3.7/site-packages/docker/utils/decorators.py",line46,ininner
|returnf(self,*args,**kwargs)
|File"/opt/conda/lib/python3.7/site-packages/docker/api/client.py",line230,in_get
|returnself.get(url,**self._set_request_timeout(kwargs))
|File"/opt/conda/lib/python3.7/site-packages/requests/sessions.py",line546,inget
|returnself.request('GET',url,**kwargs)
|File"/opt/conda/lib/python3.7/site-packages/requests/sessions.py",line533,inrequest
|resp=self.send(prep,**send_kwargs)
|File"/opt/conda/lib/python3.7/site-packages/requests/sessions.py",line646,insend
|r=adapter.send(request,**kwargs)
|File"/opt/conda/lib/python3.7/site-packages/requests/adapters.py",line529,insend
|raiseReadTimeout(e,request=request)
|requests.exceptions.ReadTimeout:UnixHTTPConnectionPool(host='localhost',port=None):Readtimedout.(readtimeout=60)

[kevin-bates]

@mihirkapadiap - thanks for the update. I took the liberty to strip out the prefix such that it doesn't also remove spaces...

[D 2019-09-03 17:49:26.718 EnterpriseGatewayApp] activity on 8d53e04e-a0ed-4b90-85a0-22a43acb7dfc: status
[D 2019-09-03 17:49:26.821 EnterpriseGatewayApp] activity on 0da31c85-7f68-487a-aa6a-5ab8cfa81879: execute_result
[E 190903 17:50:31 ioloop:909] Exception in callback <bound method KernelRestarter.poll of <jupyter_client.ioloop.restarter.IOLoopKernelRestarter object at 0x7f449c22d2e8>>
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "", line 2, in raise_from
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "/opt/conda/lib/python3.7/http/client.py", line 1321, in getresponse
response.begin()
File "/opt/conda/lib/python3.7/http/client.py", line 296, in begin
version, status, reason = self._read_status()
File "/opt/conda/lib/python3.7/http/client.py", line 257, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/opt/conda/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
enterprise-gateway_enterprise-gateway.1.8r05e9z6m4zm@clv235sl-c342e3 |
During handling of the above exception, another exception occurred:
enterprise-gateway_enterprise-gateway.1.8r05e9z6m4zm@clv235sl-c342e3 |
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/opt/conda/lib/python3.7/site-packages/urllib3/util/retry.py", line 368, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/opt/conda/lib/python3.7/site-packages/urllib3/packages/six.py", line 686, in reraise
raise value
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 306, in _raise_timeout
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
enterprise-gateway_enterprise-gateway.1.8r05e9z6m4zm@clv235sl-c342e3 |
During handling of the above exception, another exception occurred:
enterprise-gateway_enterprise-gateway.1.8r05e9z6m4zm@clv235sl-c342e3 |
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/tornado/ioloop.py", line 907, in _run
return self.callback()
File "/opt/conda/lib/python3.7/site-packages/jupyter_client/restarter.py", line 93, in poll
if not self.kernel_manager.is_alive():
File "/opt/conda/lib/python3.7/site-packages/jupyter_client/manager.py", line 453, in is_alive
if self.kernel.poll() is None:
File "/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/processproxies/container.py", line 112, in poll
container_status = self.get_container_status(None)
File "/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/processproxies/docker_swarm.py", line 74, in get_container_status
task = self._get_task()
File "/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/processproxies/docker_swarm.py", line 58, in _get_task
tasks = service.tasks(filters={'desired-state': 'running'})
File "/opt/conda/lib/python3.7/site-packages/docker/models/services.py", line 54, in tasks
return self.client.api.tasks(filters=filters)
File "/opt/conda/lib/python3.7/site-packages/docker/utils/decorators.py", line 34, in wrapper
return f(self, *args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/docker/api/service.py", line 358, in tasks
return self._result(self._get(url, params=params), True)
File "/opt/conda/lib/python3.7/site-packages/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/docker/api/client.py", line 230, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/opt/conda/lib/python3.7/site-packages/requests/sessions.py", line 546, in get
return self.request('GET', url, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/opt/conda/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/requests/adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
[E 190903 17:52:46 ioloop:909] Exception in callback <bound method KernelRestarter.poll of <jupyter_client.ioloop.restarter.IOLoopKernelRestarter object at 0x7f449c744710>>
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "", line 2, in raise_from
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "/opt/conda/lib/python3.7/http/client.py", line 1321, in getresponse
response.begin()
File "/opt/conda/lib/python3.7/http/client.py", line 296, in begin
version, status, reason = self._read_status()
File "/opt/conda/lib/python3.7/http/client.py", line 257, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/opt/conda/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
enterprise-gateway_enterprise-gateway.1.8r05e9z6m4zm@clv235sl-c342e3 |
During handling of the above exception, another exception occurred:
enterprise-gateway_enterprise-gateway.1.8r05e9z6m4zm@clv235sl-c342e3 |
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/opt/conda/lib/python3.7/site-packages/urllib3/util/retry.py", line 368, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/opt/conda/lib/python3.7/site-packages/urllib3/packages/six.py", line 686, in reraise
raise value
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 306, in _raise_timeout
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
enterprise-gateway_enterprise-gateway.1.8r05e9z6m4zm@clv235sl-c342e3 |
During handling of the above exception, another exception occurred:
enterprise-gateway_enterprise-gateway.1.8r05e9z6m4zm@clv235sl-c342e3 |
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/tornado/ioloop.py", line 907, in _run
return self.callback()
File "/opt/conda/lib/python3.7/site-packages/jupyter_client/restarter.py", line 93, in poll
if not self.kernel_manager.is_alive():
File "/opt/conda/lib/python3.7/site-packages/jupyter_client/manager.py", line 453, in is_alive
if self.kernel.poll() is None:
File "/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/processproxies/container.py", line 112, in poll
container_status = self.get_container_status(None)
File "/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/processproxies/docker_swarm.py", line 74, in get_container_status
task = self._get_task()
File "/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/processproxies/docker_swarm.py", line 56, in _get_task
service = self._get_service()
File "/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/processproxies/docker_swarm.py", line 40, in _get_service
services = client.services.list(filters={'label': 'kernel_id=' + self.kernel_id})
File "/opt/conda/lib/python3.7/site-packages/docker/models/services.py", line 269, in list
for s in self.client.api.services(**kwargs)
File "/opt/conda/lib/python3.7/site-packages/docker/utils/decorators.py", line 34, in wrapper
return f(self, *args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/docker/api/service.py", line 284, in services
return self._result(self._get(url, params=params), True)
File "/opt/conda/lib/python3.7/site-packages/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/docker/api/client.py", line 230, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/opt/conda/lib/python3.7/site-packages/requests/sessions.py", line 546, in get
return self.request('GET', url, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/opt/conda/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/requests/adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
[E 190903 17:53:50 ioloop:909] Exception in callback <bound method KernelRestarter.poll of <jupyter_client.ioloop.restarter.IOLoopKernelRestarter object at 0x7f449c034d30>>
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "", line 2, in raise_from
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "/opt/conda/lib/python3.7/http/client.py", line 1321, in getresponse
response.begin()
File "/opt/conda/lib/python3.7/http/client.py", line 296, in begin
version, status, reason = self._read_status()
File "/opt/conda/lib/python3.7/http/client.py", line 257, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/opt/conda/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
enterprise-gateway_enterprise-gateway.1.8r05e9z6m4zm@clv235sl-c342e3 |
During handling of the above exception, another exception occurred:
enterprise-gateway_enterprise-gateway.1.8r05e9z6m4zm@clv235sl-c342e3 |
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/opt/conda/lib/python3.7/site-packages/urllib3/util/retry.py", line 368, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/opt/conda/lib/python3.7/site-packages/urllib3/packages/six.py", line 686, in reraise
raise value
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 306, in _raise_timeout
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
enterprise-gateway_enterprise-gateway.1.8r05e9z6m4zm@clv235sl-c342e3 |
During handling of the above exception, another exception occurred:
enterprise-gateway_enterprise-gateway.1.8r05e9z6m4zm@clv235sl-c342e3 |
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/tornado/ioloop.py", line 907, in _run
return self.callback()
File "/opt/conda/lib/python3.7/site-packages/jupyter_client/restarter.py", line 93, in poll
if not self.kernel_manager.is_alive():
File "/opt/conda/lib/python3.7/site-packages/jupyter_client/manager.py", line 453, in is_alive
if self.kernel.poll() is None:
File "/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/processproxies/container.py", line 112, in poll
container_status = self.get_container_status(None)
File "/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/processproxies/docker_swarm.py", line 74, in get_container_status
task = self._get_task()
File "/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/processproxies/docker_swarm.py", line 56, in _get_task
service = self._get_service()
File "/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/processproxies/docker_swarm.py", line 40, in _get_service
services = client.services.list(filters={'label': 'kernel_id=' + self.kernel_id})
File "/opt/conda/lib/python3.7/site-packages/docker/models/services.py", line 269, in list
for s in self.client.api.services(**kwargs)
File "/opt/conda/lib/python3.7/site-packages/docker/utils/decorators.py", line 34, in wrapper
return f(self, *args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/docker/api/service.py", line 284, in services
return self._result(self._get(url, params=params), True)
File "/opt/conda/lib/python3.7/site-packages/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/docker/api/client.py", line 230, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/opt/conda/lib/python3.7/site-packages/requests/sessions.py", line 546, in get
return self.request('GET', url, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/opt/conda/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/requests/adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

What's curious to me is that Swarm appears to be having trouble performing this lookup...

File "/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/processproxies/docker_swarm.py", line 40, in _get_service
services = client.services.list(filters={'label': 'kernel_id=' + self.kernel_id})

and I'm wondering if Swarm itself is overloaded? The poll() method is used to determine if a given kernel is still alive and, if it thinks the kernel is not running, EG/Jupyter attempts to restart the kernel.

Are the Swarm or System logs showing any anomalies? (Not sure what additional logging Swarm might provide.)

[kevin-bates]

It might be interesting to issue docker service ls when this condition occurs.

[mihirkapadiap]

Hi Kevin,

Thanks for such quick response! You are amazing!

I checked /var/log/messages and I see some timeout happening!

I will recreate scenario and try out docker service ls and update what happens.

Sep 3 10:53:14 dockerd: time="2019-09-03T10:51:02.259364257-07:00" level=warning msg="memberlist: Push/Pull with 0922c68c1fc7 failed: dial tcp 10.155.176.130:7946: i/o timeout"
Sep 3 10:53:27 systemd-logind: Failed to start session scope session-22626.scope: Connection timed out
Sep 3 10:53:45 consul: 2019/09/03 10:53:08 [ERR] memberlist: failed to receive: read tcp 10.155.176.127:8301->10.155.40.34:37140: i/o timeout from=10.155.40.34:37140

I also have some questions about you mentioning "not sure at what point additional EG servers get configured."
I am trying to understand role of each notebook server, notebook kernel and EG.
Is every request routed via EG. I am going thru logs/code a bit let me get back to you with specific question once I have high level understanding.

[kevin-bates]

@mihirkapadiap - thanks for checking lower-level logs and perform docker service ls once back in this condition.

With the NB2KG server extension[*], the Notebook server that serves user's Notebook sessions has been configured to relay all kernel management and kernelspec operations to KG_URL. There, Enterprise Gateway (built on Jupyter Kernel Gateway), listens for those requests to retrieve kernelspecs and manage kernels. In addition, all traffic to/from the kernel itself, also gets sent via a websocket to the gateway server, where its split into/from the appropriate of the 5 ZeroMQ ports.

Enterprise Gateway is essentially Kernel Gateway, but with additional support for ProcessProxies - which facilitate lifecycle management of a given kernel relative to a given resource manager (in your case, Docker Swarm).

If you were to configure multiple EG servers and placed a load balancer application in front, then each kernel request could be distributed across the various EG servers. Keep in mind, however, that you'd need to configure the load-balancer to use sticky sessions, because we want the entire lifecycle of a given kernel to remain hosted by the same EG instance (at least until we support active/active HA).

I hope that helps.

[*]: Please note that in Notebook 6.0 the entire NB2KG extension and its configuration can be replaced with the command line option --gateway-url=<KG_URL>. Some of the other env names have been changed as well. Just thought I'd mention this.

[mihirkapadiap]

@kevin-bates

Ran again for 200 kernels!

I did see some error in /var/log/messages again.

However I could do docker service ls! I did it may be couple minutes after error received.
It seems for all notebook kernels image became 0/1 (lost all notebooks/kernel died).

docker service ls

ID NAME MODE REPLICAS IMAGE PORTS
cz9y8ekg2w7i enterprise-gateway_enterprise-gateway replicated 1/1 elyra/enterprise-gateway:dev
2867t16cjx3k guest-0cba85a3-1df3-4515-a74b-71268a78c2ca replicated 0/1 elyra/kernel-py:dev
se89667ot341 guest-0d31d807-cb08-42c5-8d71-b9c81c6b5876 replicated 0/1 elyra/kernel-py:dev
k2ybqapgpmg0 guest-0dffc888-c70b-4022-9cd5-52e7dcf4c031 replicated 0/1 elyra/kernel-py:dev

[mihirkapadiap]

@kevin-bates

How do I run notebook 6. Just download notebook version 6 and run! Even with version 6 it will still use EG to route request and do management (like before version 6).

I see example on https://jupyter-enterprise-gateway.readthedocs.io/en/latest/getting-started.html
As of now I am using it!
docker run -t --rm
-e KG_URL='http://:8888'
-e KG_HTTP_USER=guest
-e KG_HTTP_PASS=guest-password
-p 8888:8888
-e VALIDATE_KG_CERT='no'
-e LOG_LEVEL=DEBUG
-e KG_REQUEST_TIMEOUT=40
-e KG_CONNECT_TIMEOUT=40
-v ${HOME}/notebooks/:/tmp/notebooks
-w /tmp/notebooks
elyra/nb2kg

[mihirkapadiap]

Another issue I see is when starting EG in Swarm it needs to start on manager node (sometimes when I do docker ps if it is running on non leader node) then nothing works (as EG can not run command like docker.service.list on non manager node etc.)!

docker stack deploy -c docker-compose.yml enterprise-gateway

[mihirkapadiap]

This is my understanding at high level!

1. On Notebook server user requests a new Python on Docker kernel.
2. Notebook server sends request for new kernel (http)
3. JEG (Enterprise Gateway) launches new kernel (using launch_docker.py script! passing kernel-id and response socket (ip and port))
4. JEG receives connection info for kernel (id, host, various ports etc.)
5. JEG passes kernel_id back to Notebook server!
6. Notebook server requests further kernel info and connects using ws!

You mentioned "There, Enterprise Gateway (built on Jupyter Kernel Gateway), listens for those requests to retrieve kernelspecs and manage kernels.
In addition, all traffic to/from the kernel itself, also gets sent via a websocket to the gateway server, where its split
into/from the appropriate of the 5 ZeroMQ ports."

Can you shed some light on how notebook server is getting info from EG?

Thanks in advance.

[kevin-bates]

2867t16cjx3k guest-0cba85a3-1df3-4515-a74b-71268a78c2ca replicated 0/1 elyra/kernel-py:dev
se89667ot341 guest-0d31d807-cb08-42c5-8d71-b9c81c6b5876 replicated 0/1 elyra/kernel-py:dev
k2ybqapgpmg0 guest-0dffc888-c70b-4022-9cd5-52e7dcf4c031 replicated 0/1 elyra/kernel-py:dev

Could you try getting at the logs from one of these containers?

How do I run notebook 6. Just download notebook version 6 and run! Even with version 6 it will still use EG to route request and do management (like before version 6).

Since you're already using elyra/nb2kg, it might be easier to stick with that. However, if you had a pure notebook image (w/o the NB2KG extension) you could just set -e JUPYTER_GATEWAY_URL='http://:8888' -e JUPYTER_GATEWAY_HTTP_USER=guest - JUPYTER_GATEWAY_HTTP_PWD=guest-password. If you just ran notebook from the command line, jupyter notebook --gateway-url=http://:8888 should do the trick also, but let's not worry about that unless you really need 6.0 functionality not provider in < 6.0.

Another issue I see is when starting EG in Swarm it needs to start on manager node (sometimes when I do docker ps if it is running on non leader node) then nothing works (as EG can not run command like docker.service.list on non manager node etc.)!

Yeah, we honestly haven't spent a lot of time with the Swarm offering. It mostly "came for free" after getting the Kubernetes work done. Having the ability to run service list is a key function in order to discover where the kernels "landed" and to perform lifecycle management. I believe you can configure multiple manager nodes if multiple EGs couldn't "fit" on a single manager. If you see places that should be changed, please feel free to open issues for discussion and contribute pull requests. We'd be happy to help out.

Can you shed some light on how notebook server is getting info from EG?

For HTTP requests (requests to start, stop, interrupt, restart kernels and GET kernelspecs), the responses are returned via HTTP responses. For websocket operations, they're returned via the websocket. All responses (and requests) relay through the NB2KG layer since that's what performs the redirection.

[mihirkapadiap]

@kevin-bates
Hi,

Tried with bigger swarm cluster.

I get Too many open files error.
I changed ulimit -n from 1024 to 4096 on all hosts but still get same error after I open about 210 kernels!
I added following to /etc/security/limits.conf

•   hard    nofile  4096
  

•   soft    nofile  4096
  

ulimit -n is very high (1 mn) value for enterprise gateway service.

[D 2019-09-04 21:46:52.247 EnterpriseGatewayApp] Connecting to: tcp://10.0.0.214:60683
[E 190904 21:46:52 web:1788] Uncaught exception POST /api/kernels (172.18.0.1)
HTTPServerRequest(protocol='http', host=':8888', method='POST', uri='/api/kernels', version='HTTP/1.1', remote_ip='')
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/tornado/web.py", line 1699, in _execute
result = await result
File "/opt/conda/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/opt/conda/lib/python3.7/site-packages/kernel_gateway/services/kernels/handlers.py", line 71, in post
yield super(MainKernelHandler, self).post()
File "/opt/conda/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/opt/conda/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/opt/conda/lib/python3.7/site-packages/notebook/services/kernels/handlers.py", line 47, in post
kernel_id = yield gen.maybe_future(km.start_kernel(kernel_name=model['name']))
File "/opt/conda/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/opt/conda/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/kernels/remotemanager.py", line 64, in start_kernel
kernel_id = yield gen.maybe_future(super(RemoteMappingKernelManager, self).start_kernel(*args, **kwargs))
File "/opt/conda/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/opt/conda/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/opt/conda/lib/python3.7/site-packages/kernel_gateway/services/kernels/manager.py", line 81, in start_kernel
kernel_id = yield gen.maybe_future(super(SeedingMappingKernelManager, self).start_kernel(*args, **kwargs))
File "/opt/conda/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/opt/conda/lib/python3.7/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/opt/conda/lib/python3.7/site-packages/notebook/services/kernels/kernelmanager.py", line 160, in start_kernel
super(MappingKernelManager, self).start_kernel(**kwargs)
File "/opt/conda/lib/python3.7/site-packages/jupyter_client/multikernelmanager.py", line 110, in start_kernel
km.start_kernel(**kwargs)
File "/opt/conda/lib/python3.7/site-packages/enterprise_gateway/services/kernels/remotemanager.py", line 225, in start_kernel
super(RemoteKernelManager, self).start_kernel(**kwargs)
File "/opt/conda/lib/python3.7/site-packages/jupyter_client/manager.py", line 261, in start_kernel
self._connect_control_socket()
File "/opt/conda/lib/python3.7/site-packages/jupyter_client/manager.py", line 210, in _connect_control_socket
self._control_socket = self.connect_control()
File "/opt/conda/lib/python3.7/site-packages/jupyter_client/connect.py", line 571, in connect_control
return self._create_connected_socket('control', identity=identity)
File "/opt/conda/lib/python3.7/site-packages/jupyter_client/connect.py", line 543, in _create_connected_socket
sock = self.context.socket(socket_type)
File "/opt/conda/lib/python3.7/site-packages/zmq/sugar/context.py", line 146, in socket
s = self._socket_class(self, socket_type, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/zmq/sugar/socket.py", line 59, in init
super(Socket, self).init(*a, **kw)
File "zmq/backend/cython/socket.pyx", line 328, in zmq.backend.cython.socket.Socket.init
zmq.error.ZMQError: Too many open files
[E 190904 21:46:52 web:2246] 500 POST /api/kernels (172.18.0.1) 5810.20ms

[kevin-bates]

Thanks for the update (sigh). Yeah, given the reports of various folks running into this outside of EG and the fact that EG is built on that same code base, I wouldn't be surprised there's a leak somewhere.

I also suspect this isn't relegated to Swarm.

EDIT:
Hmm - but you're not terminating kernels - correct? You're just seeing how many can be running concurrently. If so, then there's really nothing to leak - unless there's some kind of connection thing happening for each kernel.

[mihirkapadiap]

@kevin-bates

Any idea....

Tried creating 25 notebook server each opening 10 kernels. I see lots of kernels stuck or throwing kernel error.
In notebook server logs I see timeouts.

I did increase timeout to 180.
-e KG_REQUEST_TIMEOUT=180
-e KG_CONNECT_TIMEOUT=180

EG logs: (Part of EG logs)
[D 2019-09-09 19:59:00.796 EnterpriseGatewayApp] Received connection info for KernelID '93b5f2bc-6999-4886-835d-ac00dd33f91b' from host 'guest-93b5f2bc-6999-4886-835d-ac00dd33f91b': {'shell_port': 46489, 'iopub_port': 37879, 'stdin_port': 54059, 'control_port': 49080, 'hb_port': 40465, 'ip': '10.0.0.52', 'key': '0542c70a-77ff-4352-9aeb-cc4294cfce72', 'transport': 'tcp', 'signature_scheme': 'hmac-sha256', 'kernel_name': '', 'comm_port': 51033}...
[D 2019-09-09 19:59:00.799 EnterpriseGatewayApp] Connecting to: tcp://10.0.0.52:49080
[D 2019-09-09 19:59:00.800 EnterpriseGatewayApp] Connecting to: tcp://10.0.0.52:37879
[I 2019-09-09 19:59:00.802 EnterpriseGatewayApp] Kernel started: 93b5f2bc-6999-4886-835d-ac00dd33f91b
[D 2019-09-09 20:02:01.469 EnterpriseGatewayApp] Initializing websocket connection /api/kernels/93b5f2bc-6999-4886-835d-ac00dd33f91b/channels
[W 2019-09-09 20:04:04.630 EnterpriseGatewayApp] Timeout waiting for kernel_info reply from 93b5f2bc-6999-4886-835d-ac00dd33f91b

Notetook server logs:
[I 20:00:40.756 NotebookApp]^[(B^[[m Connecting to ws://<eg_host>:8888/api/kernels/93b5f2bc-6999-4886-835d-ac00dd33f91b/channels
Exception in callback KernelGatewayWSClient._connection_done()
handle: <Handle KernelGatewayWSClient._connection_done()>
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/asyncio/events.py", line 145, in _run
self._callback(*self._args)
File "/opt/conda/lib/python3.6/site-packages/nb2kg/handlers.py", line 169, in _connection_done
self.ws = fut.result()
File "/opt/conda/lib/python3.6/site-packages/tornado/stack_context.py", line 315, in wrapped
ret = fn(*args, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/tornado/simple_httpclient.py", line 271, in _on_timeout
raise HTTPError(599, error_message)
tornado.httpclient.HTTPError: HTTP 599: Timeout during request