Show more statistics in console

[abitmore]

User Story

As a node admin I want to see more statistics in console, E.G.

• number of peers that are currently connected to my node
• number of websocket API clients connected
• number of API requests served in last x seconds

Currently we only have these:

1188274ms th_a application.cpp:532 handle_block ] Got block: #39537050 025b499a44481e82d0baf058549db1de97ef3331 time: 2019-07-26T21:19:48 transaction(s): 26 latency: 274 ms from: bhuz irreversible: 39537033 (-17)
1189287ms th_a application.cpp:592 handle_transaction ] Got 13 transactions from network
1190343ms th_a application.cpp:592 handle_transaction ] Got 10 transactions from network

Impacts
Describe which portion(s) of BitShares Core may be impacted by your request. Please tick at least one box.

• API (the application programming interface)
• Build (the build process or something prior to compiled code)
• CLI (the command line wallet)
• Deployment (the deployment process after building such as Docker, Travis, etc.)
• DEX (the Decentralized EXchange, market engine, etc.)
• P2P (the peer-to-peer network for transaction/block propagation)
• Performance (system or user efficiency, etc.)
• Protocol (the blockchain logic, consensus, validation, etc.)
• Security (the security of system or user data, etc.)
• UX (the User Experience)
• Other (please add below)

Additional Context (optional)
Add any other context about your request here.

CORE TEAM TASK LIST

• Evaluate / Prioritize Feature Request
• Refine User Stories / Requirements
• Define Test Cases
• Design / Develop Solution
• Perform QA/Testing
• Update Documentation

[cogutvalera]

@ryanRfox I want to take this issue, attach it to me please

[Zapata]

Ideally I would have those statistics in my monitoring tool (ie Prometheus). So those information should be easely exportable from an eventual (prometheus) plugin.

[ryanRfox]

Let's have some additional discussions to define how we want Core to expose stats to node operators (likely we don't what most stats exposed to public clients). Currently the issue author states logging to console, which is one option. I agree with @Zapata that a plugin may be desirable to capture data for post processing by another tool(s). Perhaps another option is to create some stats APIs.

Console/Logging
Pros: built into Core, easy to observe, realtime
Cons: limited data persistence/search-ability

Plugin
Pros: Optional if node operator wants to capture this data
Cons: Unsure if/how data output can be generalized for ingestion by multiple tools

API
Pros: built into Core, common design pattern
Cons: likely need to be authenticated APIs

Please add to my rough three Options and relevant Pro/Con. I realize it's low priority, but it may give us some valuable insights into node utilizations for improvements elsewhere in the code. Please offer examples of the stats you'd like to see.

[pmconrad]

IMO the "Cons" of "Console/Logging" can easily be overcome with appropriate post-processing tools, like an ELK stack or Prometheus/Grafana.
We could facilitate this by adding a log appender that logs JSON-formatted log messages into rotating files, for better machine-readability.
We already have a gelf_appender that sends machine-readable logs to a network endpoint. I've never tried it out, unsure what happens if the receiving endpoint is offline.

[pmconrad]

Related: #1554