02 Elasticsearch FAQ

NTOPNG - Elasticsearch - Kibana FAQ

---

Q: Do I need Logstash to insert ntopng data in Elasticsearch?

A: No, ntopng is able to directly and properly feed data to Elasticsearch Bulk API

Example:
ntopng -F 'es;ntopng;ntopng-%Y.%m.%d;http://elasticsearch:9200/_bulk;'

Format:
         es;<idx type>;<idx name>;<es URL>;<http:auth>
Usage:
         es;ntopng;ntopng-%Y.%m.%d;http://localhost:9200/_bulk;
         Note: the <idx name> accepts the strftime() format.

---

Q: How often will ntopng push data to Elasticsearch?

A: Once started, ntopng will push ES flows that are expired or periodically send (every 5 mins) partial flows for long lasting flows. The @timestamp field will be derived by the host time settings.

---

Q: I need to use Authentication and/or HTTPS to reach my ES Cluster

A: ntopng natively supports both - use the optional parameters, ie:

ntopng -F 'es;ntopng;ntopng-%Y.%m.%d;https://elasticsearch:80/_bulk;http_user:password;'

---

Q: How will ntopng data be indexed?

A: ntopng will populate index and type fields based on its configuration:

Example:
         es;ntopng;ntopng-%Y.%m.%d;http://localhost:9200/_bulk;

{
  "_index": "ntopng-2015.09.26",
  "_type": "ntopng",
  "_id": "ykXCN6sqQCueiyEH-mSv-w",
  "_score": 1,
  "_source": {
    "IPV4_SRC_ADDR": "127.0.0.1",
    "L4_SRC_PORT": 60091,
    "IPV4_DST_ADDR": "127.0.0.1",
    "L4_DST_PORT": 3000,
    "PROTOCOL": 6,
    "L7_PROTO": 7,
    "L7_PROTO_NAME": "HTTP",
    "TCP_FLAGS": 27,
    "IN_PKTS": 5,
    "IN_BYTES": 908,
    "OUT_PKTS": 5,
    "OUT_BYTES": 415,
    "FIRST_SWITCHED": 1443299288,
    "LAST_SWITCHED": 1443299288,
    "CLIENT_NW_LATENCY_MS": 0.003,
    "SERVER_NW_LATENCY_MS": 0.002,
    "HTTP_HOST": "localhost",
    "HTTP_URL": "\/js\/jquery.js",
    "HTTP_METHOD": "GET",
    "HTTP_RET_CODE": 304,
    "@timestamp": "2015-09-26T20:28:08.0Z",
    "@version": 1,
    "type": "ntopng"
  }
}

Note: The content of each flow will vary depending on the protocol, sources, etc.

---

Q: How can I have ES/Kibana recognize correct data types from ntopng?

A: ntopng automatically pushes a template to specify the proper data type for any field.

---