feature: revamp dummy agent to have compute plugins indivisually

[fregataa]

follow-up: #1313

Let dummy agents to have compute plugins indivisually.

How to use

Set agent.toml

Check configs/agent/sample-dummy.toml for full content.

# File name "agent1.toml"
[agent]
mode = "dummy" # or backend = "dummy"
...

[dummy.agent.intrinsic.cpu]
core-indexes = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

[dummy.agent.intrinsic.memory]
size = 50000000000

[dummy.agent.delay]
scan-image = 0.1
pull-image = 0.1
destroy-kernel = 0.1
clean-kernel = 0.1
create-network = 0.1
destroy-network = 0.1


[dummy.agent.image.already-have]
'cr.backend.ai/multiarch/python:3.9-ubuntu20.04' = ''


[dummy.kernel-creation-ctx.delay]
prepare-scratch = 0.1
prepare-ssh = 0.1
spawn = 0.1
start-container = 2.0
mount-krunner = 0.1


[dummy.kernel]
use-fake-code-runner = true


[dummy.kernel.delay]
check-status = 0.1
get-completions = 0.1
get-logs = 0.1
interrupt-kernel = 0.1
start-service = 0.1
start-model-service = 0.1
shutdown-service = 0.1
commit = 10.0
get-service-apps = 0.1
accept-file = 0.1
download-file = 0.1
download-single = 0.1
list-files = 0.1

Then run agent.

./backend.ai ag start-server --debug -f agent1.toml

How to run multiple agents

If you want to run multiple dummy agents in one machine, you need multiple "agent.toml" files. Copy and paste the original "agent.toml" file and change rpc-listen-addr, agent-sock-port, id and pid-file.

# File name "agent2.toml"
[agent]
mode = "dummy"
rpc-listen-addr = { host = "127.0.0.1", port = 6012 }
agent-sock-port = 6008
id = "dummy-agent2"
pid-file = "./agent2.pid"
ipc-base-path = "/tmp/backend.ai/ipc"
var-base-path = "var/lib/backend.ai"
...

Then run agent.

./backend.ai ag start-server --debug -f agent2.toml

How to set compute devices.

Dummy agents can set up heterogeneous computing devices using mock accelerators just like regular agents. For more information about how to use mock accelerators, see src/ai/backend/accelerator/mock/README.md.

How it looks

Can check mock accelerator as "GPU" resource here. It is possible to {create/delete} session with delays we set in `agent.toml` file.

Expectation

• Easy load-test
• Easy to test GUI with multiple heterogeneous agents

Checklist: (if applicable)

• Milestone metadata specifying the target backport version
• Documentation

[kyujin-cho]

Perhaps we can find a way to integrate our mock accelerator plugin with dummy agent.

[jopemachine]

@fregataa Dynamically generating multiple agent.toml files for running multiple agents could be a cumbersome task.

I think it would be convenient if a single agent.toml file could be used to run multiple agents by enabling the necessary values for each agent to be overridden with environment variables.
(rpc-listen-addr, agent-sock-port, id and pid-file.)

However since there are many environment variable values that need to be passed, it might be better to look for a more efficient method.

What do you think about it?