Merge pull request #177 from auto-pi-lot/dev

v0.5.0a0 - The Data Modeling Edition

A lot. Namely a whole rewriting of the {mod}`autopilot.data` module.

This is being released as an alpha version while we finish working out a few of the kinks in the data modeling 
system because Jonny is finishing their dissertation and has a deadline determined by the viscera of 
institutional science rather than software development!

## Upgrading From <v0.5.0

- The subject class will attempt to update any of your existing
  .h5 files to the new format -- it should be nondestructive (making a backup of the whole file first
  and then making backups up any tables that it can't create a new version of that fully preserves
  the data from the previous version), but don't be alarmed when your subject files start updating!
  We advise, as we always do, making a full backup of your data directory before making this update.
- Change references in `launch_autopilot.sh` scripts from `autopilot.core.terminal` or `autopilot.core.pilot` 
  to `autopilot.agents.terminal` or `autopilot.agents.pilot`, respectively
- Update any `TrialData` declarations in plugin tasks to use the new {class}`.Trial_Data` model. See {attr}`.Nafc.TrialData` for an example.
  The old pytables IsDescriptor style descriptions will still  work, but you will be given a warning every time.
- We advise updating pigpio to at least [`3c23715`](sneakers-the-rat/pigpio@3c23715) to
  avoid warnings and make use of some new functionality.

## Package Structure

- The `autopilot.core` module was removed entirely, and was split into
  - {mod}`autopilot.agent`: which contains the {class}`.Terminal` and {class}`.Pilot` classes and a stub metaclass
  - {mod}`autopilot.utils.loggers`: formerly `autopilot.core.loggers`
  - {mod}`autopilot.gui` (described in module documentation) now has all the GUI modules, split out by type rather than all in one file.
  - {mod}`autopilot.data` was created from `autopilot.core.subject`, see below.
- {mod}`autopilot.root` contains root classes: 
  - {class}`.Autopilot_Type` - types and data models for handling and saving live data
  - {class}`.Autopilot_Object` - General root metaclass, beneath which will be top-level metaclasses for each of the major
    class hierarchies that are currently separate.
  - {class}`.Autopilot_Pref` - Stub for later converting prefs and environmental configuration parameters from their scattered state.
- {mod}`autopilot.exceptions` will be used for more explicit exception handling.
- Code files were given `755` permissions to allow execution (as opposed to `644`)
- The unused git submodules for pigpio and an old temperature sensor were removed.

## Major updates

- {mod}`autopilot.data` is the major change in this release! See the module-level doc page for more details
  - `autopilot.core.subject` was moved to {mod}`autopilot.data.subject` and remains the main interface to autopilot data.
    It was effectively rewritten to accomodate the use of data models, and many of its public methods were made private to
    make its use more obvious.
    - The subject structure that determines which things are located where has been made into its own class {class}`.Subject_Structure`
    - Making new subjects is now done with the {meth}`.Subject.new` class method rather than from the `__init__` method
    - The `open_hdf` method has been replaced with the `_h5f` private method that is a more explicit context manager allowing
      locking read/write access and nonlocking read access
    - The subject file now explicitly handles updates between different versions of the Subject class. 
    - Subject will check if the protocol `.json` file has been updated from when it was assigned and automatically update to the new version,
      logging the change.
    - All trial data is now saved with a `session_uuid` unique identifier column that is an always-unique key in case of any
      overlapping session IDs from reassignments, etc.
    - Many attributes were replaced by properties that return data models:
      - {attr}`.Subject.bio` and {attr}`.Subject.info` ->  {class}`.Biography`
      - {attr}`.Subject.protocol` -> {class}`.Protocol_Status`, which manages the current trial, session, step, etc. rather than being treated separately
  - {mod}`autopilot.data.interfaces` contains classes to relate abstract data models to different representations and formats
    - {mod}`.interfaces.datajoint` allows creating datajoint schema definitions from autopilot models using the
      companion [datajoint-babel](https://github.com/auto-pi-lot/datajoint-babel) project
    - {mod}`.interfaces.tables` translates pydantic models to HDF5 files
  - {mod}`autopilot.data.modeling` has basic types for use in data models
  - {mod}`autopilot.data.models` has the models themselves
  - {mod}`autopilot.data.units` stub module for using explicit units instead of anonymous floats
- A new set of general model filling widgets - {mod}`.widgets.input` and {mod}`.widgets.model` - were created that will
  eventually replace much of the existing GUI which suffers from code duplication problems from minor variations between
  representations of parameters, etc.
- {class}`~.loggers.Log`, {class}`.Log_Format`, {class}`.LogEntry`, {class}`.ParseError`, 
  {data}`.MESSAGE_FORMATS`, {data}`.LOG_FORMATS` were added to allow programmatic loading of logfiles, and {mod}`.utils.log_parsers` 
  was added to allow recovery of structured data (eg. from logged trial data). Loggers now use the [rich logging handler](https://rich.readthedocs.io/en/stable/logging.html)
  for much more readable logs in stderr.
- {class}`.Jack_Sound` classes now put all frames at once into the processing queue, rather than one at a time for much less
  variability in sound onset jitter. The {class}`.JackClient` class was updated to reflect that as well by adding a `play_q_size`
  argument that controls the size of the deque used to store frames to play (rather than pulling each from a Queue as was done before). {class}`.JackClient`
  also has an optional `disable_gc` (False by default) parameter that allows the garbage collector to be disabled
  in the process for further (unquantified) jitter reduction.

## Minor updates

- The bandwidth test ({class}`.Bandwidth_Test` widget and {meth}`.Pilot.l_bandwidth` listen) was updated to allow selecting
  compression, use of a random array, or preserialized messages
- The {mod}`.Terminal` now closes cleanly (still with a warning of leaked semaphors) without hanging.
- External processes {func}`.external.start_pigpiod` and {func}`.external.start_jackd` first check if the process is already started
- {meth}`.Digital_Out.set` now has a ``result`` argument for faster setting that doesn't confirm the result when ``False``
- {class}`.Message` s now provide clearer support for compression in serialization, and automatically detect and handle
  a compressed message when deserializing. This is also carried through to the {meth}`.Net_Node.send` method.
- {meth}`.Station.l_kill` was moved up to the top level {class}`.Station` class, rather than being something the Terminal station owns.
- The {mod}`~.setup.run_script` functions no longer uses the totally unreadable white on green ANSI code combo that I have no idea why i ever used in the first place.
- A `picamera_legacy` script was added for enabling the picamera on bullseye.
- The {class}`.sounds.Gammatone` sound now accepts a `filter_kwargs` dictionary that is passed on to the {class}`.timeseries.Gammatone` filter.
- The {class}`.Task` and {class}`.Nafc` classes use the new {class}`.Trial_Data` style data declarations.
- Two utility functions {func}`.common.walk_dicts` and {func}`.common.flatten_dicts` were added to make handling nested dictionaries a bit easier.
- 

## Bugfixes

- The Subject class would incorrectly overwrite data given a mismatch in system times between the Terminal and Pilot.
  The subject class should now ideally no longer overwrite anything ever.
- The subject class would drop trial data silently if it was not in the TrialData description. En route to making the table
  automatically expand to accomodate unexpected data, dropped data is now logged as an exception.
- The prefs manager handles being launched from within ipython and other processes better, but is still a bit buggy.
  Now it uses the check used internally in the multiprocessing module to see if a manager can be launched,
  and falls back to using a standard dictionary if not.
- `jackd_source` script uses correct `https://` rather than `git://` protocol.
- A `_TASK_LIST` was added to {mod}`.utils.registry` to support deprecated task specifications.

## Regressions

- With additional checking for monotonic increases in `trial_num` and checks that prevent data overwriting,
  individual writes of trial data are now a bit slower, which should be optimized for when we complete the transition to
  uniform data models throughout the library.

## Prefs

- `PIGPIOD` - `bool` - if `True`, start pigpiod on pilot start
- Stubs were created for converting the prefs to using data models, but they have not been filled yet.
- A `AUTOPILOT_NO_PREFS_MANAGER` environment variable now controls the use of a multiprocessing manager explicit. Documentation
  for environmental variables is forthcoming.
- A `AUTOPILOT_WARN_DEFAULTS` environment variable controls whether warnings should be printed for when a default pref value
  is retrieved, because that warning is a good one but can be really annoying.

## Packaging & Dependencies

- Autopilot is now packaged with Poetry! This allows for fully deterministic installation with the poetry.lock file and
  updates from the old setuptools style of dependency specification.
- The source repository has moved from https://github.com/wehr-lab/autopilot to https://github.com/auto-pi-lot/autopilot
- `MANIFEST.in` has been replaced by the `include` field in `pyproject.toml`
- `autopilot.__version__` is now determined by `importlib.metadata` and specified in the `pyproject.toml` file rather than in the `__init__.py` file
- `blosc` was replaced with `blosc2`
- New dependencies
  - global
    - pydantic (^1.9.0)
    - parse (^1.19.0)
    - rich (^11.2.0)
    - validators (^0.18.2)
  - docs
    - autodoc_pydantic (^1.7.0)
    - myst_parser (^0.17.2)
- Version Bumps
  - pyzmq 18.1.* -> ^22.3.0
  - tornado >=5.0.0 -> ^6.1.0
  - numpy >=1.16.5 -> ^1.20.0
  - scipy >=1.6.0 -> ^1.7.0
  - pandas >=0.19.2 -> ^1.3.0 on python 3.7 and ^1.4.0 on >=3.8
  - tables >=3.4.2 -> ^3.7.0
  - Sphinx >=3.1.2 -> ^4.3.1
  - A lot more dependencies were taken from being implicit versions to explicit
    by the conversion to using Poetry...

## Docs

- [Configuration](configuration) was moved to its own page, documenting setting up the system as well as the contents of the user directory.
- A [faq](FAQ) page was stubbed out (but is still pretty skeletal)
- The overview was updated with some more information in the module tour
- Virtual environment usage was moved from the setup page to its own subpage linked from the FAQ.
- A `make serve` option was added to the docs makefile that makes use of [sphinx-autobuild](https://pypi.org/project/sphinx-autobuild/)
  to livereload docs while editing them.
- `autopilot_theme.css` was updated to be compatible with the new version of sphinx-rtd-theme that apparently changed the way
  that TOC buttons were made, as well as remove incorrect references to fonts no longer packaged.
- The `autodoc_pydantic` and `myst_parser` extensions were added -- and we will be moving towards using MyST rather than 
  hellish ReST for future narrative docs!
- Private methods and functions are now no  longer rendered in the main documentation, and the library will over time
  use the public/private distinction more systematically to make it more understandable.
- Examples was split off into its own folder and links to wiki plugins. Blink was moved with it

## Tests

- We have started importing some of the pigpio mocking tools from the [People's Ventilator Project](https://peoplesvent.org)
  to start writing GPIO tests!

Author: sneakers-the-rat

.gitmodules

@@ -19,6 +19,9 @@ formats:
 
 # Optionally set the version of Python and requirements required to build your docs
 python:
-  version: 3.7
+  version: 3.8
   install:
-    - requirements: requirements/requirements_docs.txt
+    - method: pip
+      path: .
+      extra_requirements:
+        - docs

.readthedocs.yml

@@ -38,13 +38,8 @@ env:
   - DISPLAY=":99.0" QT_DEBUG_PLUGINS=1  # https://pytest-qt.readthedocs.io/en/latest/troubleshooting.html?highlight=travis#xvfb-assertionerror-timeouterror-when-using-waituntil-waitexposed-and-ui-events
 
 install:
-  - pip install -U pip
-  - pip install -U pytest
-  - pip install -U pytest-cov
-  - pip install -U pylint
-  - pip install -U coveralls
-  - pip install -U pytest-qt~=3.3.0
-  - pip install -e .
+  - pip install --upgrade setuptools importlib_metadata
+  - pip install .[tests]
   - "/sbin/start-stop-daemon --start --quiet --pidfile /tmp/custom_xvfb_99.pid --make-pidfile --background --exec /usr/bin/Xvfb -- :99 -screen 0 1920x1200x24 -ac +extension GLX +render -noreset"
   - sleep 3
 
@@ -63,17 +58,14 @@ after_script:
   - coverage combine --append
   - coveralls
 
-before_deploy:
-  - pip install scikit-build
-
 deploy:
   provider: pypi
   username: __token__
   pasword:
     secure: mIaKgOMl4tjUvLlpSGwbuUj2A7QK0nMnQzlYygdVN5vCuvXnggScytqwXNqjSqjB3cn0R4EKllCfnREsIqBgYrEsRCRIexwLuz58KDo6qjSad/ycpCjmtz/6nvuJp3I2iJGudzCZ/W1MCpz0ExAvvHXm7k0U4J831bK+mFYqe8RHJxcAuYMJndEUaD5s/Ly+ozqJYx3H9+f/fggpVAlGu/oy5p77so9vQA0xvayodQKzL1XGvNHBXMEKtBtEN33pTKDbOY3bQ7GR0WLE7pd+bg1rZ/r5EsvlHAlHR/rSCWDNq4K0Qk3ZyUeHLX9UjKIImXbhMMyNSyz98DkgUd/TkfGG2K2jyKd7WR1KqOrNjy+clqTdyj1fcHtm8au6YsvYLvR9qmO1prwrGq1vMH0/qvOrcM3+OU3hGdUwtjth3F+TTsc57O22qHjpeq/xlWtFC8vAF92ZHzPWW6xBApYB3+MJZHh27LJL/rdbAv/WoYPZbQWiqxNm869GCaSzjBGZ/Zx5BPpGbJUb8N+pBHgow3gww2TMS1C7W3gDVPm42DL8ehhOEJZBT9FIfzAooeyxzyvUb6gSdtshGUuJPe8HdZhNn5n4rFuSoZ7edCbEEM7bQ/gqOJK9Ccr+4yi/9MHBXB7LsjB32tzlxbsEePG18ac85wJp1Z64QYMpzV5DAi0=
   on:
     tags: true
-    branch: master
+    branch: main
     condition: "$TRAVIS_PYTHON_VERSION = 3.9"
   distributions: "sdist"
 

.travis.yml

@@ -17,7 +17,7 @@ doi: "10.1101/807693"
 keywords: 
 license: "MPL-2.0"
 message: "If you use this software, please cite it using these metadata."
-repository-code: "https://github.com/wehr-lab/autopilot"
+repository-code: "https://github.com/auto-pi-lot/autopilot"
 title: "Autopilot: Automating behavioral experiments with lots of Raspberry Pis"
 version: "0.4.4"
 ...

CITATION.cff

@@ -9,8 +9,8 @@
 Status:
 
 [![Documentation Status](https://readthedocs.org/projects/auto-pi-lot/badge/?version=latest)](https://docs.auto-pi-lot.com/en/latest/?badge=latest)
-[![Travis (.com) branch](https://img.shields.io/travis/com/wehr-lab/autopilot/main)](https://app.travis-ci.com/github/wehr-lab/autopilot/branches)
-[![Coveralls branch](https://img.shields.io/coveralls/github/wehr-lab/autopilot/main)](https://coveralls.io/github/wehr-lab/autopilot)
+[![Travis (.com) branch](https://img.shields.io/travis/com/auto-pi-lot/autopilot/main)](https://app.travis-ci.com/github/auto-pi-lot/autopilot/branches)
+[![Coverage Status](https://coveralls.io/repos/github/auto-pi-lot/autopilot/badge.svg?branch=main)](https://coveralls.io/github/auto-pi-lot/autopilot?branch=main)
 ![Jonny Status](https://img.shields.io/badge/jonny-dissertating-critical)
 
 
@@ -19,9 +19,9 @@ Status:
 
 ![Autopilot Banner Logo](docs/_images/autopilot_logo_banner.png)
 
-| [Docs](https://docs.auto-pi-lot.com) | [Paper](https://www.biorxiv.org/content/10.1101/807693v1) | [Forum](https://github.com/wehr-lab/autopilot/discussions) | [Wiki](https://wiki.auto-pi-lot.com) |
+| [Docs](https://docs.auto-pi-lot.com) | [Paper](https://www.biorxiv.org/content/10.1101/807693v1) | [Forum](https://github.com/auto-pi-lot/autopilot/discussions) | [Wiki](https://wiki.auto-pi-lot.com) |
 | :-: | :-: | :-: | :-: |
-| [![Read the Docs](docs/_images/docs_link.png)](https://docs.auto-pi-lot.com) | [![Paper](docs/_images/paper_link.png)](https://www.biorxiv.org/content/10.1101/807693v1)  | [![Forum](docs/_images/discussion_link.png)](https://github.com/wehr-lab/autopilot/discussions) | [![Wiki](docs/_images/hardware_link.png)](https://wiki.auto-pi-lot.com)
+| [![Read the Docs](docs/_images/docs_link.png)](https://docs.auto-pi-lot.com) | [![Paper](docs/_images/paper_link.png)](https://www.biorxiv.org/content/10.1101/807693v1)  | [![Forum](docs/_images/discussion_link.png)](https://github.com/auto-pi-lot/autopilot/discussions) | [![Wiki](docs/_images/hardware_link.png)](https://wiki.auto-pi-lot.com)
 
 Autopilot is a Python framework for performing complex, hardware-intensive behavioral experiments with swarms of networked Raspberry Pis. 
 As a tool, it provides researchers with a toolkit of flexible modules to design experiments without rigid programming & API limitations. 
@@ -91,7 +91,7 @@ python3 -m autopilot.setup.setup
 ![Autopilot Setup Console](docs/_images/installer.png)
 
 All of Autopilot is quite new, so bugs, incomplete documentation, missing features are very much expected! Don't be shy about
-[raising issues](https://github.com/wehr-lab/autopilot/issues) or [asking questions in the forum](https://github.com/wehr-lab/autopilot/discussions).
+[raising issues](https://github.com/auto-pi-lot/autopilot/issues) or [asking questions in the forum](https://github.com/auto-pi-lot/autopilot/discussions).
 
 
 # Development Status
@@ -102,17 +102,17 @@ Jonny is trying to graduate! Autopilot will be slow and maybe a little chaotic u
 
 We're working on a formal contribution system, pardon the mess! Until we get that and our CI coverage up, `main` will lag a bit behind the development branches:
 
-* [`dev`](https://github.com/wehr-lab/autopilot/tree/dev) - main development branch that collects hotfixes, PRs, etc. Unstable but usually has lots of extra goodies
-* [`hotfix`](https://github.com/wehr-lab/autopilot/tree/hotfix) - branches from `dev` for building and testing hotfixes, PRs back to `dev`.
-* [`lab`](https://github.com/wehr-lab/autopilot/tree/lab) - branches from `dev` but doesn't necessarily PR back, the local branch used in the maintaining ([Wehr](http://uoneuro.uoregon.edu/wehr/)) lab
-* [`parallax`](https://github.com/wehr-lab/autopilot/tree/parallax) - experimental departure from `dev` to implement a particular experiment and rebuild a lot of components along the way, will eventually return to `dev` <3
+* [`dev`](https://github.com/auto-pi-lot/autopilot/tree/dev) - main development branch that collects hotfixes, PRs, etc. Unstable but usually has lots of extra goodies
+* [`hotfix`](https://github.com/auto-pi-lot/autopilot/tree/hotfix) - branches from `dev` for building and testing hotfixes, PRs back to `dev`.
+* [`lab`](https://github.com/auto-pi-lot/autopilot/tree/lab) - branches from `dev` but doesn't necessarily PR back, the local branch used in the maintaining ([Wehr](http://uoneuro.uoregon.edu/wehr/)) lab
+* [`parallax`](https://github.com/auto-pi-lot/autopilot/tree/parallax) - experimental departure from `dev` to implement a particular experiment and rebuild a lot of components along the way, will eventually return to `dev` <3
 
 ## Short-Term
 
 See the short-term development goals in our version milestones:
 
-* [`v0.4.0`](https://github.com/wehr-lab/autopilot/milestone/1) - Implement registries to separate user code extensions like tasks and local hardware devices in a user directory, preserve source code in produced data so local development isn't lost. 
-* [`v0.5.0`](https://github.com/wehr-lab/autopilot/milestone/2) - Make a unitary inheritance structure from a root Autopilot object such that a) common operations like logging and networking are implemented only once, b) the plugin system for `v0.4.0` can not only add new objects, but replace core objects while maintaining provenance (ie. no monkey patching needed), c) object behavior that requires coordination across multiple instances gets much easier, making some magical things like self-healing self-discovering networking possible. This will also include a major refactoring of the code structure, finally breaking up some of the truly monstrous thousand-line modules in `core` into an actually modular system we can build from <3
+* [`v0.4.0`](https://github.com/auto-pi-lot/autopilot/milestone/1) - Implement registries to separate user code extensions like tasks and local hardware devices in a user directory, preserve source code in produced data so local development isn't lost. 
+* [`v0.5.0`](https://github.com/auto-pi-lot/autopilot/milestone/2) - Make a unitary inheritance structure from a root Autopilot object such that a) common operations like logging and networking are implemented only once, b) the plugin system for `v0.4.0` can not only add new objects, but replace core objects while maintaining provenance (ie. no monkey patching needed), c) object behavior that requires coordination across multiple instances gets much easier, making some magical things like self-healing self-discovering networking possible. This will also include a major refactoring of the code structure, finally breaking up some of the truly monstrous thousand-line modules in `core` into an actually modular system we can build from <3
 
 ## Long-Term
 

MANIFEST.in

@@ -1,5 +1,20 @@
 __author__  = 'Jonny Saunders <[email protected]>'
-__version__ = '0.4.4'
+
+import sys
+if sys.version_info < (3,8):
+    from importlib_metadata import version
+    # monkeypatch typing
+    import typing
+    from typing_extensions import Literal
+    typing.Literal = Literal
+else:
+    from importlib.metadata import version
+
+
+__version__ = version("auto-pi-lot")
+
+from autopilot.root import Autopilot_Type, Autopilot_Pref
+
 
 from autopilot.setup import setup_autopilot
 from autopilot.utils.registry import get, get_task, get_hardware, get_names

README.md

@@ -0,0 +1,28 @@
+"""
+Base Agent class.
+
+Currently a stub just to get them in the object hierarchy
+"""
+
+from autopilot.root import Autopilot_Object
+
+
+class Agent(Autopilot_Object):
+    """
+    Metaclass for agent types.
+
+    Currently a stub, but will provide hooks for basic lifecycle methods of agents:
+
+    * ``pre_init`` - to be run before any other standard initialization
+    * ``init`` - main initialization hook
+    * ``init_external`` - initialize external processes
+    * ``post_init`` - to be run after other initialization
+    * ... to be continued
+
+    And core class and instance attributes:
+
+    * ``prefs`` - prefs that are needed to configure this agent
+    * ``processes`` - processes spawned by this agent
+    * ``listens`` - methods to handle messages sent to this agent
+    * ``dependencies`` - additional optional python packages or system configurations that this agent depends on.
+    """