feature: Standalone watcher

[fregataa]

Changes

• Implement a standalone watcher component as ai.backend.watcher
  • The watcher dynamically loads watcher plugins and ships several intrinsic ones (e.g., systemd service monitor).
  • Just like other service components, it accepts -f/--config option while searching watcher.toml
• Unify the mechanism to detect and load daemon configurations, as a follow-up to feat: Add scie-based single binary distributions #1680.
• Merge https://github.com/lablup/backend.ai-storage-watcher and https://github.com/lablup/backend.ai-agent-watcher into this Backend.AI repo

Follow-ups

The PRs below should be merged after this get merged

• https://github.com/lablup/backend.ai-storage-watcher/pull/1
• https://github.com/lablup/backend.ai-agent-watcher/pull/1
• https://github.com/lablup/backend.ai-storage-kmanilla/pull/5
• https://github.com/lablup/backend.ai-ktc-manila-proxy/pull/1

How to test

./backend.ai watcher start-server -f <CONFIG_TOML_PATH>

You can find a sample config toml file in each watcher repository.
storage watcher: https://github.com/lablup/backend.ai-storage-watcher
agent watcher: https://github.com/lablup/backend.ai-agent-watcher

If you want to run multiple watcher plugins in one watcher process, install the all watcher plugins and set all configs in one file and pass the path of the config file.

# Example config toml file to run agent-watcher and storage-watcher simultaneously

[watcher]
node-id = "i-something-special"
event-loop = "uvloop"
ipc-base-path = "/tmp/backend.ai/ipc"

[module.agent-watcher]
service-name = "backendai-agent"
do-mount-check = true
ipc-base-path = "/tmp/backend.ai/ipc"
ignore-mount-event = true

[module.agent-watcher.event]
connect-server = true
consumer-group = "agent-watcher"

[module.storage-watcher]
bai-uid = 502
bai-gid = 502
service-name = "backendai-storage"
do-mount-check = false

[module.agent-watcher.event]
connect-server = false
consumer-group = "storage-watcher"

#...

TODO

• Port agent-watcher APIs
• Bundle indivisual subapps e.g) agent watcher subapp, service watcher subapp

Checklist: (if applicable)

• Milestone metadata specifying the target backport version
• Documentation

[fregataa]

Since control panel would be the one that requests to watcher servers, I made an issue about supporting HTTPS in control panel.
https://github.com/lablup/backend.ai-control-panel/issues/674