Graph tab of dashboard gives 500 error

[birdsarah]

This only happens on my EMR cluster, not locally.

Here's the traceback.

distributed version 1.25.2

distributed.utils - ERROR - '<' not supported between instances of 'NoneType' and 'NoneType'
Traceback (most recent call last):
  File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/distributed/utils.py", line 648, in log_errors
    yield
  File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/distributed/bokeh/scheduler.py", line 1139, in graph_doc
    graph = GraphPlot(scheduler, sizing_mode='stretch_both')
  File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/distributed/bokeh/scheduler.py", line 615, in __init__
    self.layout = GraphLayout(scheduler)
  File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/distributed/diagnostics/graph_layout.py", line 38, in __init__
    priority=priority)
  File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/distributed/diagnostics/graph_layout.py", line 42, in update_graph
    stack = sorted(dependencies, key=lambda k: priority.get(k, 0), reverse=True)
TypeError: '<' not supported between instances of 'NoneType' and 'NoneType'
tornado.application - ERROR - Uncaught exception GET /graph (127.0.0.1)
HTTPServerRequest(protocol='http', host='localhost:8282', method='GET', uri='/graph', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/tornado/web.py", line 1592, in _execute
    result = yield result
  File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/tornado/gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/bokeh/server/views/doc_handler.py", line 55, in get
    session = yield self.get_session()
  File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/tornado/gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/bokeh/server/views/session_handler.py", line 77, in get_session
    session = yield self.application_context.create_session_if_needed(session_id, self.request)
  File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/bokeh/server/contexts.py", line 215, in create_session_if_needed
    self._application.initialize_document(doc)
  File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/bokeh/application/application.py", line 178, in initialize_document
    h.modify_document(doc)
  File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/bokeh/application/handlers/function.py", line 133, in modify_document
    self._func(doc)
  File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/distributed/bokeh/scheduler.py", line 1139, in graph_doc
    graph = GraphPlot(scheduler, sizing_mode='stretch_both')
  File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/distributed/bokeh/scheduler.py", line 615, in __init__
    self.layout = GraphLayout(scheduler)
  File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/distributed/diagnostics/graph_layout.py", line 38, in __init__
    priority=priority)
  File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/distributed/diagnostics/graph_layout.py", line 42, in update_graph
    stack = sorted(dependencies, key=lambda k: priority.get(k, 0), reverse=True)
TypeError: '<' not supported between instances of 'NoneType' and 'NoneType'
Exception ignored in: <bound method GraphPlot.__del__ of <distributed.bokeh.scheduler.GraphPlot object at 0x7fd29d9e4b00>>
Traceback (most recent call last):
  File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/distributed/bokeh/scheduler.py", line 750, in __del__
    self.scheduler.remove_plugin(self.layout)
AttributeError: 'GraphPlot' object has no attribute 'layout'

[mrocklin]

The right place to fix this is probably here: stack = sorted(dependencies, key=lambda k: priority.get(k, 0), reverse=True) The key function only works if the priorities are the same type. It fails if there are ints vs tuples vs None. Ideally we would find some function that was both very fast to run and also was able to compare safely between these types. Maybe just making the value `(type(value), value)` instead.

…

On Mon, Jan 21, 2019 at 7:54 PM Sarah Bird ***@***.***> wrote: This only happens on my EMR cluster, not locally. Here's the traceback. distributed version 1.25.2 distributed.utils - ERROR - '<' not supported between instances of 'NoneType' and 'NoneType' Traceback (most recent call last): File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/distributed/utils.py", line 648, in log_errors yield File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/distributed/bokeh/scheduler.py", line 1139, in graph_doc graph = GraphPlot(scheduler, sizing_mode='stretch_both') File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/distributed/bokeh/scheduler.py", line 615, in __init__ self.layout = GraphLayout(scheduler) File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/distributed/diagnostics/graph_layout.py", line 38, in __init__ priority=priority) File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/distributed/diagnostics/graph_layout.py", line 42, in update_graph stack = sorted(dependencies, key=lambda k: priority.get(k, 0), reverse=True)TypeError: '<' not supported between instances of 'NoneType' and 'NoneType' tornado.application - ERROR - Uncaught exception GET /graph (127.0.0.1) HTTPServerRequest(protocol='http', host='localhost:8282', method='GET', uri='/graph', version='HTTP/1.1', remote_ip='127.0.0.1') Traceback (most recent call last): File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/tornado/web.py", line 1592, in _execute result = yield result File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run value = future.result() File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/tornado/gen.py", line 1141, in run yielded = self.gen.throw(*exc_info) File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/bokeh/server/views/doc_handler.py", line 55, in get session = yield self.get_session() File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run value = future.result() File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/tornado/gen.py", line 1141, in run yielded = self.gen.throw(*exc_info) File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/bokeh/server/views/session_handler.py", line 77, in get_session session = yield self.application_context.create_session_if_needed(session_id, self.request) File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run value = future.result() File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/bokeh/server/contexts.py", line 215, in create_session_if_needed self._application.initialize_document(doc) File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/bokeh/application/application.py", line 178, in initialize_document h.modify_document(doc) File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/bokeh/application/handlers/function.py", line 133, in modify_document self._func(doc) File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/distributed/bokeh/scheduler.py", line 1139, in graph_doc graph = GraphPlot(scheduler, sizing_mode='stretch_both') File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/distributed/bokeh/scheduler.py", line 615, in __init__ self.layout = GraphLayout(scheduler) File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/distributed/diagnostics/graph_layout.py", line 38, in __init__ priority=priority) File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/distributed/diagnostics/graph_layout.py", line 42, in update_graph stack = sorted(dependencies, key=lambda k: priority.get(k, 0), reverse=True)TypeError: '<' not supported between instances of 'NoneType' and 'NoneType' Exception ignored in: <bound method GraphPlot.__del__ of <distributed.bokeh.scheduler.GraphPlot object at 0x7fd29d9e4b00>> Traceback (most recent call last): File "/mnt/anaconda2/envs/daskyarn/lib/python3.6/site-packages/distributed/bokeh/scheduler.py", line 750, in __del__ self.scheduler.remove_plugin(self.layout)AttributeError: 'GraphPlot' object has no attribute 'layout' — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2474>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszBhMCOgpcQsStJNKJpKX97qurHKNks5vFouEgaJpZM4aL4tT> .

[zyxue]

I am new to dask.

I am seeing the same problem on localhost on a Mac.

Any update on why this is happening?

After this error, the code is still running, which is confusing... I am working inside a jupyter notebook.

distributed.utils - ERROR - '<' not supported between instances of 'NoneType' and 'tuple'
Traceback (most recent call last):
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/distributed/utils.py", line 693, in log_errors
    yield
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/distributed/bokeh/scheduler.py", line 1139, in graph_doc
    graph = GraphPlot(scheduler, sizing_mode='stretch_both')
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/distributed/bokeh/scheduler.py", line 615, in __init__
    self.layout = GraphLayout(scheduler)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/distributed/diagnostics/graph_layout.py", line 38, in __init__
    priority=priority)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/distributed/diagnostics/graph_layout.py", line 42, in update_graph
    stack = sorted(dependencies, key=lambda k: priority.get(k, 0), reverse=True)
TypeError: '<' not supported between instances of 'NoneType' and 'tuple'
tornado.application - ERROR - Uncaught exception GET /graph (::1)
HTTPServerRequest(protocol='http', host='localhost:8787', method='GET', uri='/graph', version='HTTP/1.1', remote_ip='::1')
Traceback (most recent call last):
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/tornado/web.py", line 1699, in _execute
    result = await result
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/tornado/gen.py", line 736, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/bokeh/server/views/doc_handler.py", line 55, in get
    session = yield self.get_session()
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/tornado/gen.py", line 729, in run
    value = future.result()
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/tornado/gen.py", line 736, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/bokeh/server/views/session_handler.py", line 77, in get_session
    session = yield self.application_context.create_session_if_needed(session_id, self.request)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/tornado/gen.py", line 729, in run
    value = future.result()
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
    yielded = self.gen.send(value)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/bokeh/server/contexts.py", line 215, in create_session_if_needed
    self._application.initialize_document(doc)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/bokeh/application/application.py", line 178, in initialize_document
    h.modify_document(doc)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/bokeh/application/handlers/function.py", line 133, in modify_document
    self._func(doc)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/distributed/bokeh/scheduler.py", line 1139, in graph_doc
    graph = GraphPlot(scheduler, sizing_mode='stretch_both')
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/distributed/bokeh/scheduler.py", line 615, in __init__
    self.layout = GraphLayout(scheduler)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/distributed/diagnostics/graph_layout.py", line 38, in __init__
    priority=priority)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/distributed/diagnostics/graph_layout.py", line 42, in update_graph
    stack = sorted(dependencies, key=lambda k: priority.get(k, 0), reverse=True)
TypeError: '<' not supported between instances of 'NoneType' and 'tuple'
Exception ignored in: <function GraphPlot.__del__ at 0x1c1ffcdf28>
Traceback (most recent call last):
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/distributed/bokeh/scheduler.py", line 750, in __del__
    self.scheduler.remove_plugin(self.layout)
AttributeError: 'GraphPlot' object has no attribute 'layout'
distributed.nanny - WARNING - Worker exceeded 95% memory budget. Restarting
tornado.application - ERROR - Exception in callback <bound method Nanny.memory_monitor of <Nanny: tcp://127.0.0.1:50896, threads: 2>>
Traceback (most recent call last):
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_common.py", line 341, in wrapper
    ret = self._cache[fun]
AttributeError: 'Process' object has no attribute '_cache'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_common.py", line 341, in wrapper
    ret = self._cache[fun]
AttributeError: _cache

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_psosx.py", line 359, in catch_zombie
    yield
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_psosx.py", line 400, in _get_pidtaskinfo
    ret = cext.proc_pidtaskinfo_oneshot(self.pid)
ProcessLookupError: [Errno 3] No such process

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/tornado/ioloop.py", line 907, in _run
    return self.callback()
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/distributed/nanny.py", line 274, in memory_monitor
    memory = proc.memory_info().rss
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_common.py", line 344, in wrapper
    return fun(self)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/__init__.py", line 1166, in memory_info
    return self._proc.memory_info()
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_psosx.py", line 339, in wrapper
    return fun(self, *args, **kwargs)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_psosx.py", line 469, in memory_info
    rawtuple = self._get_pidtaskinfo()
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_psosx.py", line 339, in wrapper
    return fun(self, *args, **kwargs)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_common.py", line 344, in wrapper
    return fun(self)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_psosx.py", line 400, in _get_pidtaskinfo
    ret = cext.proc_pidtaskinfo_oneshot(self.pid)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_psosx.py", line 372, in catch_zombie
    raise AccessDenied(proc.pid, proc._name)
psutil.AccessDenied: psutil.AccessDenied (pid=65344)
tornado.application - ERROR - Exception in callback <bound method Nanny.memory_monitor of <Nanny: tcp://127.0.0.1:50896, threads: 2>>
Traceback (most recent call last):
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_common.py", line 341, in wrapper
    ret = self._cache[fun]
AttributeError: 'Process' object has no attribute '_cache'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_common.py", line 341, in wrapper
    ret = self._cache[fun]
AttributeError: _cache

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_psosx.py", line 359, in catch_zombie
    yield
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_psosx.py", line 400, in _get_pidtaskinfo
    ret = cext.proc_pidtaskinfo_oneshot(self.pid)
ProcessLookupError: [Errno 3] No such process

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/tornado/ioloop.py", line 907, in _run
    return self.callback()
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/distributed/nanny.py", line 274, in memory_monitor
    memory = proc.memory_info().rss
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_common.py", line 344, in wrapper
    return fun(self)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/__init__.py", line 1166, in memory_info
    return self._proc.memory_info()
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_psosx.py", line 339, in wrapper
    return fun(self, *args, **kwargs)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_psosx.py", line 469, in memory_info
    rawtuple = self._get_pidtaskinfo()
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_psosx.py", line 339, in wrapper
    return fun(self, *args, **kwargs)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_common.py", line 344, in wrapper
    return fun(self)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_psosx.py", line 400, in _get_pidtaskinfo
    ret = cext.proc_pidtaskinfo_oneshot(self.pid)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_psosx.py", line 372, in catch_zombie
    raise AccessDenied(proc.pid, proc._name)
psutil.AccessDenied: psutil.AccessDenied (pid=65344)
tornado.application - ERROR - Exception in callback <bound method Nanny.memory_monitor of <Nanny: tcp://127.0.0.1:50896, threads: 2>>
Traceback (most recent call last):
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_common.py", line 341, in wrapper
    ret = self._cache[fun]
AttributeError: 'Process' object has no attribute '_cache'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_common.py", line 341, in wrapper
    ret = self._cache[fun]
AttributeError: _cache

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_psosx.py", line 359, in catch_zombie
    yield
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_psosx.py", line 400, in _get_pidtaskinfo
    ret = cext.proc_pidtaskinfo_oneshot(self.pid)
ProcessLookupError: [Errno 3] No such process

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/tornado/ioloop.py", line 907, in _run
    return self.callback()
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/distributed/nanny.py", line 274, in memory_monitor
    memory = proc.memory_info().rss
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_common.py", line 344, in wrapper
    return fun(self)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/__init__.py", line 1166, in memory_info
    return self._proc.memory_info()
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_psosx.py", line 339, in wrapper
    return fun(self, *args, **kwargs)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_psosx.py", line 469, in memory_info
    rawtuple = self._get_pidtaskinfo()
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_psosx.py", line 339, in wrapper
    return fun(self, *args, **kwargs)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_common.py", line 344, in wrapper
    return fun(self)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_psosx.py", line 400, in _get_pidtaskinfo
    ret = cext.proc_pidtaskinfo_oneshot(self.pid)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/zyxue/miniconda3/envs/ml3.2/lib/python3.7/site-packages/psutil/_psosx.py", line 372, in catch_zombie
    raise AccessDenied(proc.pid, proc._name)
psutil.AccessDenied: psutil.AccessDenied (pid=65344)
distributed.nanny - WARNING - Worker process 65344 was killed by signal 15
distributed.nanny - WARNING - Restarting worker

[songqiqqq]

same problem, how to fix it ?

[mrocklin]

There is a suggestion two comments above if you want to try to fix things

#2474 (comment)

[jrbourbeau]

@songqiqqq any interest in submitting a PR with the suggested fix?

[songqiqqq]

of course yes, but i am messed up with lot's of work at hand these days. If I could fix it, a PR would be made. But i can't promise a deadline, it depends on my progress of work at hand. @jrbourbeau

[MrRobot2211]

I am having the exact same issue

[quasiben]

@MrRobot2211 are you also using EMR ?