Elastic search show only suricata and p0f data.

[TanveerAhmad2017]

Hi there
I have 20.6.2 Tpot version installed in VM.8Gb RAM ,140SSD and 2 cores.
*Note: /data folder is not empty i.e every honeypot have their respective logs inside.
Basically what I want is the whole data present in the "/data" folder to seen by the Elastic search but the problem is that it only shows p0f and suricata data. I tried to look for the logstash.config and found the input and output sections which show which data to take as input and which as output(elasticsearch) but when we visit elastic search web interface why it does not show the logs or data of other folders lets say cowrie etc. Am I missing something or I have to do something to achieve the required behaviour.
Below code is not complete and just for reference

input {
Cowrie
file {
path => ["/data/cowrie/log/cowrie.json"]
codec => json
type => "Cowrie"
}

_output {
elasticsearch {
hosts => ["elasticsearch:9200"]
# With templates now being legacy and ILM in place we need to set the daily index with its template manually. Otherwise a new index might be created with differents settings configured through Kibana.
index => "logstash-%{+YYYY.MM.dd}"
template => "/etc/logstash/tpot_es_template.json"

document_type => "doc"

}_

[Ganesh-403]

@TanveerAhmad2017
If the raw log files exist under /data but Elasticsearch only contains Suricata and p0f events, then the issue is most likely between Logstash and Elasticsearch, not with the honeypots themselves.

Since you're on T-Pot 20.06.2, the Logstash pipeline is already configured to ingest many honeypot logs, including Cowrie, Dionaea, Conpot, Heralding, etc. For example, the default pipeline contains inputs such as:

file {
  path => ["/data/cowrie/log/cowrie.json"]
  codec => json
  type => "Cowrie"
}

along with inputs for the other bundled honeypots. :contentReference[oaicite:0]{index=0}

So, in a default installation, you should not need to manually add Cowrie or the other honeypot inputs.

A few things I'd check:

1. Verify Logstash is processing the events

Check the Logstash logs for parsing or indexing errors:

docker logs logstash

In particular, look for:

• _jsonparsefailure
• mapping exceptions
• Elasticsearch indexing errors
• pipeline startup failures

---

2. Confirm the indices actually exist

Run:

curl http://localhost:9200/_cat/indices?v

or from Kibana Dev Tools:

GET _cat/indices?v

If the documents are indexed but Kibana doesn't display them, it may be a data view/index pattern issue rather than an ingestion problem.

---

3. Verify the Cowrie log is receiving new events

For example:

tail -f /data/cowrie/log/cowrie.json

If new JSON events appear but Logstash never indexes them, the issue is within the Logstash pipeline.

---

4. Remember how the file input behaves

The Logstash file input keeps track of its read position using sincedb. If the log files already contained historical events before Logstash started, it may consider them already processed and only ingest newly appended events. :contentReference[oaicite:1]{index=1}

A quick test is to generate a new event (for example, by connecting to the Cowrie SSH honeypot) and then see whether it appears in Elasticsearch.

---

5. Check whether events are being dropped by filters

The input section alone doesn't determine whether an event is indexed. If a filter later in the pipeline drops or fails to parse a particular event type, it won't reach Elasticsearch.

Inspect the Logstash logs for messages indicating dropped events or filter failures.

---

One question that would help narrow this down:

• When you query Elasticsearch directly, do you see only Suricata and P0f documents, or do you also see indices/documents for Cowrie and the other honeypots but they're simply missing from Kibana?

That will help determine whether the issue is with Logstash ingestion, Elasticsearch indexing, or Kibana visualization.

If this solves your problem, feel free to mark it as the accepted answer so others can find it easily.