Move Agent code into main library

[BrianJKoopman]

I wanted to capture some of the discussion that happened when this was brought up last week. The more I think about it, the more I think we should do this.

While I'm writing this on the ocs repo, I definitely am keeping the socs repo in mind on these points.

Reasons

I was going to make a pros/cons list, but there really aren't any cons other than needing to actually spend the time moving things into the library. Here are the reasons we'd do that.

PyPI Distribution

This adds the Agent code to the PyPI distributed package.

It really occurred to me while releasing socs v0.3.0 that if systems were to install via PyPI, they would not have access to the actual Agent code, so would still need to checkout a copy of the repo anyway.

Reduce Number of Docker Images

Instead of using one Docker image per Agent and having the image default to running the given Agent a single image with all the Agents in it could be deployed and configured with a different entrypoint or command in the docker-compose file. This would require installing all dependencies for all Agents in the base image.

Reduce Docker Build Complexity

Right now developers who are working on parts in both the Agent and in the main library need to to make sure to build the base image first, then their Agent image to make sure they have any changes while testing the new Docker image.

Other Reasons?

What other reasons do you have for this reorganization?

Questions

• Do we still build individual images that default to a specific Agent as an entrypoint? i.e. ENTRYPOINT ["dumb-init", "python3", "-u", "aggregator_agent.py"]

  This would allow us to still package images that come with a specific Agent's dependencies, without needing to install them all in the base image.

• How do we handle finding Agents now (in the style of ocs_plugin/agent-paths)?

  This plugins list is often forgotten about when adding Agents. If the Agents are all in the main library and importable, it'd be great to just auto-detect what Agents are available if given the name of a package containing Agents. So if we tell our system we have ocs and socs installed then it could look in ocs.agents and socs.agents for what it's able to import.

• How do we bundle driver code?

  Motivated mostly by testing early on, I often was splitting any driver code off from the Agent file into the main library in ocs/agent/<agent_name>.py. The Agent would then from ocs.agent.<agent_name> import <function1>, <class1> or similar. This can often result in quite short Agents, where the complexity may be in hidden in the driver (i.e. the influxdb publisher agent.)

  Even though this means editing two separate files to work on an Agent, this separation still makes some sense to me. If the driver code is externally developed the Agent will perform a similar import (i.e. the ACU Agent). And the internally developed module could be quite large (take the Lakeshore drivers we've written for example).

  I don't think we'd require all Agents perform this split, but we should likely plan for some to have it. Having ocs/agent/ (current driver code) and ocs/agents/ (place I'd imagine we move agents) would be confusing. Perhaps agents move to ocs/agents/ and drive code to ocs/drivers/ (or ocs/agents/drivers/)?

Ideas

Create agent-cli Entrypoint

I think @mhasself suggested this. Similar to how we have client-cli we could have an interface for starting up Agents within the library, so you could run agent-cli aggregator to startup the aggregator (likely given an instance-id?)

Handling Dependencies

Moving all Agent code into the library would naturally introduce all of the separate Agent's dependencies onto the library. We could handle this in the setup.py, with separate Agent's dependencies being listed under different extra_requires entries. We'd keep any core library dependencies in the install_requires and then for each Agent you want to install you'd need to pip install socs[agent1,agent2,agent3] to get the various dependencies.

The view Agents that need this (on the socs end are):

• pysmurf Archiver (which we're getting rid of anyway)
• Labjack
• SupRsync (but it's deps are in the main requirements file already...)
• pysmurf monitor
• Synaccess Agent
• pysmurf controller

These are mostly fairly standard libraries anyway. The exception being the labjack's dependencies, which needs to be installed via a script provided by LabJack. There's also the ACU Agent, which depends on a private library.

[mhasself]

Thanks for thinking this all through!

• How do we handle finding Agents now (in the style of ocs_plugin/agent-paths)?
  This plugins list is often forgotten about when adding Agents. If the Agents are all in the main library and importable, it'd be great to just auto-detect what Agents are available if given the name of a package containing Agents. So if we tell our system we have ocs and socs installed then it could look in ocs.agents and socs.agents for what it's able to import.

As mentioned on call today, python has a pretty reasonable system to support automatically detecting plugins. I suggest we explore using "namespace packages":

• Example of two packages that install into the same namespace: https://github.com/pypa/sample-namespace-packages/tree/master/native
• Code that discovers subpackages in a namespace: https://packaging.python.org/en/latest/guides/creating-and-discovering-plugins/#using-namespace-packages

The idea would be to designate submodule ocs.plugins (or, possibly, a new package called ocs_plugins) as the target space where 3rd party packages (I'm going to include socs in this) install a small submodule that registers their agents. The registration wouldn't involve an import of any packages -- just a dict with strings telling what module / function to call to launch the agent. Partial example:

   # Data installed by socs setup.py to live in ocs/plugins/socs/__init__.py
   package_name = 'socs'
   agents = {
     'HWPBBBAgent': {'module': 'socs.agents.chwp.hwpbbb_agent', 'entry_point': 'main'},
     ...
    }

When the ocs-agent script goes to launch some instance-id, it figures out the agent class then looks that up in all the ocs.plugins submodules; and does the appropriate import and function call.

[mhasself]

• Do we still build individual images that default to a specific Agent as an entrypoint? i.e. ENTRYPOINT ["dumb-init", "python3", "-u", "aggregator_agent.py"]
  This would allow us to still package images that come with a specific Agent's dependencies, without needing to install them all in the base image.

I don't see a convincing downside to including as many Agent dependencies as possible in our images, making a sort of universal ocs Docker image. ("As possible" can run out eventually ... some stuff might be mutually incompatible. But until that point, it's pretty convenient.) The hardware dependencies don't tend to update super quickly, so it's not like people would be pulling layers needlessly all the time (and in any case they'd still only need one new thing for all their agents).