Files pane is glitchy, lots of `/dev/null`s opened by orchestrator

[etnguyen03]

In production, the orchestrator is opening so many /dev/nulls that it hits the process limit on how many file descriptors are open at once.

Traceback (most recent call last):
  File "DIRECTOR_ROOT_PATH/orchestrator/.venv/lib/python3.8/site-packages/websockets/server.py", line 191, in handler
    await self.ws_handler(self, path)
  File "DIRECTOR_ROOT_PATH/orchestrator/orchestrator/ws.py", line 80, in route
    await handler(websock, params, stop_event)
  File "DIRECTOR_ROOT_PATH/orchestrator/orchestrator/consumers/status.py", line 89, in status_handler
    await log_follower.start(since_time=time.time())
  File "DIRECTOR_ROOT_PATH/orchestrator/orchestrator/logs.py", line 45, in start
    self.proc = await asyncio.create_subprocess_exec(
  File "/usr/lib/python3.8/asyncio/subprocess.py", line 236, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1630, in subprocess_exec
    transport = await self._make_subprocess_transport(
  File "/usr/lib/python3.8/asyncio/unix_events.py", line 197, in _make_subprocess_transport
    transp = _UnixSubprocessTransport(self, protocol, args, shell,
  File "/usr/lib/python3.8/asyncio/base_subprocess.py", line 36, in __init__
    self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
  File "/usr/lib/python3.8/asyncio/unix_events.py", line 789, in _start
    self._proc = subprocess.Popen(
  File "/usr/lib/python3.8/subprocess.py", line 804, in __init__
    errread, errwrite) = self._get_handles(stdin, stdout, stderr)
  File "/usr/lib/python3.8/subprocess.py", line 1470, in _get_handles
    p2cread = self._get_devnull()
  File "/usr/lib/python3.8/subprocess.py", line 952, in _get_devnull
    self._devnull = os.open(os.devnull, os.O_RDWR)
OSError: [Errno 24] Too many open files: '/dev/null'

and:

ls -la /proc/$WEBSOCK_PID/fd

yields many instances (about 900) of

lrwx------ 1 USER GROUP 64 Nov 23 11:31 1002 -> /dev/null

Proposed resolution

• Add a "Loading..." message to the files pane when it is loading
• TBD...

For the record, this has only occurred on one of several appservers in production, and it cannot be reproduced in a development environment.