Kibana: Elastic did not load properly. Check the server output for more information.

[atbohmer]

Hello,

Running for almost a year now on a VM the T-Pot iso installation. Over the weekend something died?! There is more then enough disk space. Any advise how to debug other then a fresh install appriciated!

cat /opt/tpot/version

22.04.0

/opt/tpot/bin/dps.sh

[ ========| System |======== ]
DATE: Mon 12 Jun 2023 02:58:46 PM UTC
UPTIME: 14:58:46 up 2:39, 2 users, load average: 0.27, 0.23, 0.19
T-POT: ACTIVE
BLACKHOLE: DISABLED

NAME STATUS PORTS
adbhoney Up 2 hours (healthy) 0.0.0.0:5555->5555/tcp
ciscoasa Up 2 hours 0.0.0.0:5000->5000/udp, 0.0.0.0:8443->8443/tcp
citrixhoneypot Up 2 hours 0.0.0.0:443->443/tcp
conpot_guardian_ast Up 2 hours (healthy) 0.0.0.0:10001->10001/tcp
conpot_iec104 Up 2 hours (healthy) 0.0.0.0:161->161/udp, 0.0.0.0:2404->2404/tcp
conpot_ipmi Up 2 hours (healthy) 0.0.0.0:623->623/udp
conpot_kamstrup_382 Up 2 hours (healthy) 0.0.0.0:1025->1025/tcp, 0.0.0.0:50100->50100/tcp
cowrie Up 2 hours 0.0.0.0:22-23->22-23/tcp
ddospot Up 2 hours 0.0.0.0:19->19/udp, 0.0.0.0:53->53/udp, 0.0.0.0:123->123/udp, 0.0.0.0:1900->1900/udp
dicompot Up 2 hours 0.0.0.0:11112->11112/tcp
dionaea Up 2 hours (healthy) 0.0.0.0:20-21->20-21/tcp, 0.0.0.0:42->42/tcp, 0.0.0.0:81->81/tcp, 0.0.0.0:135->135/tcp, 0.0.0.0:445->445/tcp, 0.0.0.0:1433->1433/tcp, 0.0.0.0:1723->1723/tcp, 0.0.0.0:1883->1883/tcp, 0.0.0.0:3306->3306/tcp, 0.0.0.0:27017->27017/tcp, 0.0.0.0:69->69/udp
elasticpot Up 2 hours 0.0.0.0:9200->9200/tcp
elasticsearch Up 2 hours (healthy) 127.0.0.1:64298->9200/tcp
fatt Up 2 hours
heralding Up 2 hours 0.0.0.0:110->110/tcp, 0.0.0.0:143->143/tcp, 0.0.0.0:465->465/tcp, 0.0.0.0:993->993/tcp, 0.0.0.0:995->995/tcp, 0.0.0.0:1080->1080/tcp, 0.0.0.0:5432->5432/tcp, 0.0.0.0:5900->5900/tcp
honeytrap Up 2 hours
ipphoney Up 2 hours 0.0.0.0:631->631/tcp
kibana Up 2 hours (healthy) 127.0.0.1:64296->5601/tcp
logstash Up 2 hours (healthy)
mailoney Up 2 hours 0.0.0.0:25->25/tcp
map_data Up 2 hours
map_redis Up 2 hours
map_web Up 2 hours 127.0.0.1:64299->64299/tcp
medpot Up 2 hours 0.0.0.0:2575->2575/tcp
nginx Up 2 hours
p0f Up 2 hours
redishoneypot Up 2 hours 0.0.0.0:6379->6379/tcp
sentrypeer Up 2 hours 0.0.0.0:5060->5060/udp
snare Up 2 hours 0.0.0.0:80->80/tcp
spiderfoot Up 2 hours (healthy) 127.0.0.1:64303->8080/tcp
suricata Up 2 hours
tanner Up 2 hours
tanner_api Up 2 hours
tanner_phpox Up 2 hours
tanner_redis Up 2 hours

Elasticsearch docker image:
elasticsearch@3e92191ec47e:/$ curl -s -XGET http://localhost:9200/_cluster/health?pretty
{
"cluster_name" : "tpotcluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 103,
"active_shards" : 103,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}

Kibana docker image log:
kibana@405c7a5ac173:/var/log/kibana$ grep -i elas kibana.log
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-06-12T12:30:04.338+00:00","message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...","log":{"level":"INFO","logger":"savedobjects-service"},"process":{"pid":1},"trace":{"id":"bf9a2117eea5b2b086f8a2bd4cb1993b"},"transaction":{"id":"0b950cb646a620e0"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-06-12T12:30:08.308+00:00","message":"Error Initializing ML saved objects {"name":"ResponseError","meta":{"body":{"error":"Incorrect HTTP method for uri [/_ml/anomaly_detectors] and method [GET], allowed: [POST]","status":405},"statusCode":405,"headers":{"x-opaque-id":"unknownId","allow":"POST","x-elastic-product":"Elasticsearch","content-type":"application/json","content-length":"113"},"meta":{"context":null,"request":{"params":{"method":"GET","path":"/_ml/anomaly_detectors","querystring":"","headers":{"user-agent":"Kibana/8.6.2","x-elastic-product-origin":"kibana","x-elastic-client-meta":"es=8.4.0p,js=16.18.1,t=8.2.0,hc=16.18.1","x-opaque-id":"unknownId","accept":"application/vnd.elasticsearch+json; compatible-with=8,text/plain"}},"options":{"opaqueId":"unknownId","headers":{"x-elastic-product-origin":"kibana","user-agent":"Kibana/8.6.2","x-elastic-client-meta":"es=8.4.0p,js=16.18.1,t=8.2.0,hc=16.18.1"}},"id":7},"name":"elasticsearch-js","connection":{"url":"http://elasticsearch:9200/\",\"id\":\"http://elasticsearch:9200/\",\"headers\":{},\"status\":\"alive\"},\"attempts\":0,\"aborted\":false},\"warnings\":null}}","log":{"level":"ERROR","logger":"plugins.ml"},"process":{"pid":1},"trace":{"id":"bf9a2117eea5b2b086f8a2bd4cb1993b"},"transaction":{"id":"0b950cb646a620e0"}}
{"tags":["ML:saved-objects-sync","ML:saved-objects-sync-task","task-run-failed"],"error":{"stack_trace":"ResponseError: {"error":"Incorrect HTTP method for uri [/_ml/datafeeds] and method [GET], allowed: [POST]","status":405}\n at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/src/create_transport.js:51:16)\n at Ml.getDatafeeds (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/ml.js:699:16)\n at async Promise.all (index 5)\n at checkStatus (/usr/share/kibana/x-pack/plugins/ml/server/saved_objects/checks.js:19:108)\n at initSavedObjects (/usr/share/kibana/x-pack/plugins/ml/server/saved_objects/sync.js:311:20)\n at Object.run (/usr/share/kibana/x-pack/plugins/ml/server/saved_objects/sync_task.js:68:19)\n at TaskManagerRunner.run (/usr/share/kibana/x-pack/plugins/task_manager/server/task_running/task_runner.js:266:22)"},"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-06-12T12:30:11.529+00:00","message":"Task ML:saved-objects-sync "ML:saved-objects-sync-task" failed: ResponseError: {"error":"Incorrect HTTP method for uri [/_ml/datafeeds] and method [GET], allowed: [POST]","status":405}","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":1},"trace":{"id":"bf9a2117eea5b2b086f8a2bd4cb1993b"},"transaction":{"id":"23884546b9795ed8"}}
{"tags":["ML:saved-objects-sync","ML:saved-objects-sync-task","task-run-failed"],"error":{"stack_trace":"ResponseError: {"error":"Incorrect HTTP method for uri [/_ml/anomaly_detectors] and method [GET], allowed: [POST]","status":405}\n at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)\n at runMicrotasks ()\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/src/create_transport.js:51:16)\n at Ml.getJobs (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/ml.js:808:16)\n at async Promise.all (index 4)\n at checkStatus (/usr/share/kibana/x-pack/plugins/ml/server/saved_objects/checks.js:19:108)\n at initSavedObjects (/usr/share/kibana/x-pack/plugins/ml/server/saved_objects/sync.js:311:20)\n at Object.run (/usr/share/kibana/x-pack/plugins/ml/server/saved_objects/sync_task.js:68:19)\n at TaskManagerRunner.run (/usr/share/kibana/x-pack/plugins/task_manager/server/task_running/task_runner.js:266:22)"},"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-06-12T13:30:11.545+00:00","message":"Task ML:saved-objects-sync "ML:saved-objects-sync-task" failed: ResponseError: {"error":"Incorrect HTTP method for uri [/_ml/anomaly_detectors] and method [GET], allowed: [POST]","status":405}","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":1},"trace":{"id":"bf9a2117eea5b2b086f8a2bd4cb1993b"},"transaction":{"id":"9762a98ccebe2a48"}}
{"tags":["ML:saved-objects-sync","ML:saved-objects-sync-task","task-run-failed"],"error":{"stack_trace":"ResponseError: {"error":"Incorrect HTTP method for uri [/_ml/anomaly_detectors] and method [GET], allowed: [POST]","status":405}\n at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)\n at runMicrotasks ()\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/target_node/src/create_transport.js:51:16)\n at Ml.getJobs (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/ml.js:808:16)\n at async Promise.all (index 4)\n at checkStatus (/usr/share/kibana/x-pack/plugins/ml/server/saved_objects/checks.js:19:108)\n at initSavedObjects (/usr/share/kibana/x-pack/plugins/ml/server/saved_objects/sync.js:311:20)\n at Object.run (/usr/share/kibana/x-pack/plugins/ml/server/saved_objects/sync_task.js:68:19)\n at TaskManagerRunner.run (/usr/share/kibana/x-pack/plugins/task_manager/server/task_running/task_runner.js:266:22)"},"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-06-12T14:30:11.674+00:00","message":"Task ML:saved-objects-sync "ML:saved-objects-sync-task" failed: ResponseError: {"error":"Incorrect HTTP method for uri [/_ml/anomaly_detectors] and method [GET], allowed: [POST]","status":405}","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":1},"trace":{"id":"bf9a2117eea5b2b086f8a2bd4cb1993b"},"transaction":{"id":"2bb61cdd634eb907"}}

[atbohmer]

And it works again without me doing anything, but ok thanks.

[Ganesh-403]

@atbohmer
Based on the information you've shared, I don't think this looks like a T-Pot failure.

A few observations:

• ✅ All containers are running, including Kibana and Elasticsearch.
• ✅ Elasticsearch reports a green cluster with all shards assigned.
• ✅ Kibana itself is marked healthy.
• ✅ The issue appears to be limited to the Kibana UI showing "Elastic did not load properly".

The log entries you posted:

Incorrect HTTP method for uri [/_ml/anomaly_detectors] and method [GET], allowed: [POST]

are related to Kibana's Machine Learning (ML) plugin. Similar messages have been reported by Elastic users when Kibana attempts to execute periodic ML tasks. On their own, they do not necessarily indicate that Kibana is broken, especially if the rest of the stack is healthy. :contentReference[oaicite:0]{index=0}

I'd recommend checking a few additional things before considering a reinstall:

1. Browser-side issue

   • Try an incognito/private window.
   • Clear the browser cache for the Kibana site.
   • Open the browser's Developer Tools (F12) and look for JavaScript or network errors.

2. Kibana logs
   Besides the ML warnings, look for any FATAL or ERROR messages around the time the UI failed:

   docker logs kibana

3. Nginx logs
   Since T-Pot serves Kibana through Nginx, verify that requests are successfully proxied:

   docker logs nginx

4. Health endpoints
   Confirm Kibana is responding from inside the host:

   curl http://localhost:64296/api/status

   If this returns a healthy status while the browser still shows "Elastic did not load properly", the issue is likely in the frontend or reverse proxy rather than Elasticsearch.

Interestingly, you mentioned that the issue resolved itself without any intervention. That makes me suspect a transient condition rather than data corruption—for example:

• Kibana temporarily waiting for background tasks to complete,
• a browser cache issue,
• or a short-lived communication problem between Kibana and Elasticsearch.

Since your Elasticsearch cluster remained green throughout, I wouldn't recommend a fresh installation unless the problem becomes reproducible.

If the issue happens again, collecting the following at the same time would be very helpful:

docker logs kibana
docker logs nginx
curl http://localhost:64296/api/status
curl http://localhost:9200/_cluster/health?pretty

That should quickly determine whether the problem is in Kibana itself, the reverse proxy, or the browser.

If this solves your problem, feel free to mark it as the accepted answer so others can find it easily.