Add doc for Management Server Statistics and Peers (#455)

* Add doc for Management Server Statistics and Peers

* Fix notes

* Update pics

* add doc for db connection issue

Author: weizhouapache

source/_static/images/management-server-peers.png

@@ -587,3 +587,53 @@ To start the Management Server:
 
    # service cloudstack-management start
 
+
+Management Server Statistics and Peers
+--------------------------------------
+
+Administrators are able to view the statistics and peers information of management server.
+      
+#. Log in to the CloudStack UI as administrator
+
+#. In the left navigation bar, click Infrastructure.
+
+#. Click "Management servers", all management servers are listed.
+
+|management-servers-list.png|
+
+#. Click the management server you'd like to view. The statistics of the management server are displayed.
+
+|management-server-statistics.png|
+
+#. Navigate to the "Peers" tab. The peers of the management servers are listed
+
+|management-server-peers.png|
+
+
+Global settings for management servers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. cssclass:: table-striped table-bordered table-hover
+
+======================================= ========================
+Configuration                            Description
+======================================= ========================
+management.server.stats.interval        Time interval in seconds, for management servers stats collection. Set to <= 0 to disable management servers stats. Default value is 60
+cluster.heartbeat.interval              Interval (in milliseconds) to check for the heart beat between management server nodes. Default value is 1500
+cluster.heartbeat.threshold             Threshold (in milliseconds) before self-fence the management server. The threshold should be larger than management.server.stats.interval. Default value is 150000
+======================================= ========================
+
+.. note::
+   - Every 60 seconds (configuable via management.server.stats.interval setting) each management server collects its statistics and publishs to all other management server peers. When other management server receives the published stats, it will set the peer state (owner is the receiver and peer is the sender) to Up.
+   - Every 1.5 seconds (configurable via cluster.heartbeat.interval), each management server writes heartbeat to CloudStack database, and check the stats of other management servers.
+   - If in the past 150 seconds (configurable via cluster.heartbeat.threshold), a management server does not write heartbeat and its peer states, its state and peer states will be set to Down by other management servers.
+   - In case a management server cannot write heartbeat to the database due to connection issue to the database, the host is set to Down state by other management server, when the database connection is restored, the management server will perform self-fencing and exit with code 219.
+
+.. |management-servers-list.png| image:: /_static/images/management-servers-list.png
+   :alt: List of management servers
+
+.. |management-server-statistics.png| image:: /_static/images/management-server-statistics.png
+   :alt: Details of management server
+
+.. |management-server-peers.png| image:: /_static/images/management-server-peers.png
+   :alt: List of management server peers