Home
This page discuss the ways of introducing chaos in to docker ecosystem using Apocalypse . With "Apocalypse", one can introduce / emulate various adverse events (resource/network / stop/ reboot/ terminate) on any random running service aka docker container. The tool is configurable over command line or with a config file to control the chaos being introduced.
Apocalypse supports the following chaos events
These are the evnets which introduce stress on the system/system resources , the following events falls under this category
- Burn CPU - loads CPU to a specified percentage between 1-100 and for a given duration in seconds. The number of cores to be stressed could also be modified
- Burn RAM - Simulates high RAM usage, percentage of RAM to be consumed and the duration can be configured
- Kill Process - Not actually a resource chaos events, but stresses the system by killing a random process.
- Burn IO , Burn Disk are other possible candidates for this category which are not yet supported
Introduces a random network error condition using tc and netem commands. refer TC-NETEM and NETEM The following events are supported under this category
- Packet Loss → Causes a certain percentage of network packets to drop
- Packet Corrupt → Corrupts a certain percentage of network packets
- Packet Reorder → Re orders network packets randomly
- Packet Duplication → Duplicates a certain percentage of network packets
- Network Delay → Introduce latency/delay to the network
- Blackhole → Make an interface un reachable , nothing goes in nothing comes out
- Failing DNS, Network Partitioning are other possible candidates in pipeline but not yet supported
These are the events which alters the state of the container as a whole , they Include
- Stop → Stops a running container
- Reboot → Causes a container to reboot
- Teminate → Stops and removes a running container
The basic assumption is that the docker ecosystem is live and is brought up preferably using docker-compose. This way the service names are well defined and will be used as it is in docker-compose.yml file
Apocalypse can be used in two modes.
- Web server
- CLI
When Apocalypse is run as web server , desired chaos event could be introduced manually from the web browser. This is more of controlled version of introducing Chaos where in the user can determine the chaos events to be introduced against a service.
To start Apocalypse in web server mode
$ ./doomsday start server --network <network-name>
This command would download ubuntu:trusty image and dhoomakethu/apocalypse:latest and start the container running web server. The web server could be accessed @ http://localhost:5555 or http://:5555.
The default port 5555 could be changed to any other value by passing optional argument --port to the command
$ ./doomsday start server --port 5678 --network <network-name>
Apocalypse rely on docker network in which docker containers are running to identify the services , the default network assumed is "bridge" , in any case where docker containers are run with a different network , the same needs to be passed as part of the command line with --network arg
$ ./doomsday start server --network some_other_network_name
In this mode , random chaos events are generated at a specified time interval against a service ,which could be chosen randomly or could be selected before hand. To start Twister in CLI mode with default values
$ ./doomsday start chaos
This would run all the Choas events against randomly choosen service every 30second. To run only selected event group, run
$ ./doomsday start chaos --run-network-choas | --run-generic-choas | --run-resource-chaos
Or to run events of choice, run
$ ./doomsday start chaos --events <space seperated events to run>
The chaos events could also be configured using a config file . E,g config file could be found here
$ ./doomsday start chaos --config <absolute path to config file>
To stop the Chaos, run
$ ./doomsday stop