max shards per node reached (1000 by default)

[nbittich]

Issue when indexing a lot of data (app-loket).

Current workaround:

curl -X PUT localhost:9200/_cluster/settings -H "Content-Type: application/json" -d '{ "persistent": { "cluster.max_shards_per_node": "7000" } }'
{"acknowledged":true,"persistent":{"cluster":{"max_shards_per_node":"3000"}},"transient":{}}

[nvdk]

it would probably be a good idea to run multiple nodes (even if only on the same server)

[nbittich]

@nvdk it seems to be the safe solution, however it can be fine tuned (https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html#field-count-recommendation) . I guess in some cases it could be fine tuned based on recommenfations of elasticsearch? FYI for a poc for loket, we had more than 1.8k indices: