improve how hupper determines what is system code or user code

[mmerickel]

Hupper tries to avoid monitoring all system code loaded into a process due to #40. We do want to monitor user code however. The four ways I'm aware of right now to develop "user" code:

• ensure folder is in PYTHONPATH
• pip install -e . installs a .pth file in site-packages which means the code continues to live properly wherever it is checked out.
• developing inside a virtualenv - some people do this so they can use bin/python
• symlinking the folder containing the code into site-packages which is done manually or using flit install --symlink and I imagine poetry install will also do this.

We want to try to count all of these scenarios as user code that is monitored and only ignore stuff that is third party installed directly into site-packages or stuff that is part of the virtualenv / system python.

related: #40 #41 #42 #60 #61

[merwok]

Notes from #61:

Adding prints to in_system_paths function (return true → in system, false → not system) shows this:

*** $venv/lib/python3.7/_bootlocale.py not system
*** /usr/lib/python3.7/contextlib.py not system

_bootlocale is symlinked from $venv/lib/python3.7 to /usr/lib, but the check from realpath prefix in system_paths fails because get_system_path uses sysconfig.get_path which never returns paths from real Python, only virtualenv paths. So the change you did for the flit use case ends up watching many sytem modules!

I think hupper will need virtualenv awareness, checking sys.real_prefix (virtualenv) or sys.base_prefix (venv) to add these paths to the exclude list. But this is to offer a complete system paths exclusion, not to fix my issue of not excluding package dir symlinked in site-packages. (Also de-duplicating list of system paths could help performance!)

[mmerickel]

I think this makes sense - I'm not sure I have the energy to sort it out though and would appreciate the help if you wanted to contribute. I think it's pretty focused on just improving the hupper.worker.get_system_paths function. It sounds to me like when hupper is running in a virtualenv it's only filtering out paths in the virtualenv, and not paths that are linked back to the main python installation.

[merwok]

A note that what I wrote is also tied to a specific implementation of virtualenv. Environments created with venv don’t have these symlinks, and virtualenv 20 is a big rewrite that (IIRC) uses a different symlink/copy strategy. So the problem may go away by itself in time, but it could still be worth it to implement it for current virtualenvs.

[mmerickel]

Yeah I mean it's not an issue when using venv I don't think, which is all I use on py3+. I'm open to specific suggestions on how to deal with it but I'm not going to dive into it myself in the near term.