Show "mustache.max_output_size_bytes" cluster settings defaults

[stefnestor]

Elasticsearch Version

8.17.3

Installed Plugins

NA

Java Version

bundled

OS Version

ESS

Problem Description

👋 howdy, team!

It appears @thecoop added mustache.max_output_size_bytes into Elasticsearch in v8.16.0 via #114002. Per #122947, this setting is editable to a certain limit somewhere under 2GB.

However this setting is not documented in Elasticsearch's docs (in current for v8 nor docs-content for main for v9) && this setting does not show under Cluster Settings API with defaults included. Therefore, users who immediately start erring post upgrade cannot find recovery information.

Steps to Reproduce

1. Create a Watcher with a large search response body.
2. Unrecoverably error like log below.
3. IF on-prem update elasticsearch.yml setting mustache.max_output_size_bytes to potentially recover, else narrow down response body. IF on ECH/ES3, unknown recovery path.

Logs (if relevant)

[2025-03-26T07:32:56,638][ERROR][org.elasticsearch.xpack.watcher.actions.webhook.ExecutableWebhookAction] [instance-0000000023] failed to execute action [xxxxx_xxxxxx/output_to_eleasticsearch]
org.elasticsearch.ElasticsearchParseException: Mustache script result size limit exceeded
    at org.elasticsearch.script.mustache.MustacheScriptEngine$MustacheExecutableScript.execute(MustacheScriptEngine.java:148) ~[?:?]
    at org.elasticsearch.xpack.watcher.common.text.TextTemplateEngine.render(TextTemplateEngine.java:64) ~[?:?]
    at org.elasticsearch.xpack.watcher.common.http.HttpRequestTemplate.render(HttpRequestTemplate.java:163) ~[?:?]
    at org.elasticsearch.xpack.watcher.notification.WebhookService.execute(WebhookService.java:126) ~[?:?]
    at org.elasticsearch.xpack.watcher.actions.webhook.ExecutableWebhookAction.execute(ExecutableWebhookAction.java:30) ~[?:?]
    at org.elasticsearch.xpack.core.watcher.actions.ActionWrapper.execute(ActionWrapper.java:175) ~[?:?]
    at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:559) ~[?:?]
    at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:341) ~[?:?]
    at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$6(ExecutionService.java:439) ~[?:?]
    at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:665) ~[?:?]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:956) ~[elasticsearch-8.17.3.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
    at java.lang.Thread.run(Thread.java:1575) ~[?:?]
Caused by: com.github.mustachejava.MustacheException: Failed to get value for xxxxx_xxxxxx.sum @[query-template:4]
    at com.github.mustachejava.codes.ValueCode.execute(ValueCode.java:74) ~[?:?]
    at com.github.mustachejava.codes.DefaultMustache.run(DefaultMustache.java:34) ~[?:?]
    at com.github.mustachejava.codes.DefaultCode.run(DefaultCode.java:214) ~[?:?]
    at com.github.mustachejava.codes.IterableCode.next(IterableCode.java:143) ~[?:?]
    at com.github.mustachejava.reflect.BaseObjectHandler.iterate(BaseObjectHandler.java:81) ~[?:?]
    at com.github.mustachejava.codes.IterableCode.execute(IterableCode.java:138) ~[?:?]
    at com.github.mustachejava.codes.IterableCode.handle(IterableCode.java:54) ~[?:?]
    at com.github.mustachejava.codes.IterableCode.execute(IterableCode.java:42) ~[?:?]
    at com.github.mustachejava.codes.DefaultMustache.run(DefaultMustache.java:34) ~[?:?]
    at com.github.mustachejava.codes.DefaultCode.execute(DefaultCode.java:162) ~[?:?]
    at com.github.mustachejava.codes.DefaultMustache.execute(DefaultMustache.java:57) ~[?:?]
    at com.github.mustachejava.Mustache.execute(Mustache.java:38) ~[?:?]
    at org.elasticsearch.script.mustache.MustacheScriptEngine$MustacheExecutableScript.execute(MustacheScriptEngine.java:143) ~[?:?]
    ... 13 more
Caused by: org.elasticsearch.common.text.SizeLimitingStringWriter$SizeLimitExceededException: String [
{ "index" : { "_ind...] has exceeded the size limit [1048576]
    at org.elasticsearch.common.text.SizeLimitingStringWriter.checkSizeLimit(SizeLimitingStringWriter.java:37) ~[elasticsearch-8.17.3.jar:?]
    at org.elasticsearch.common.text.SizeLimitingStringWriter.write(SizeLimitingStringWriter.java:58) ~[elasticsearch-8.17.3.jar:?]
    at com.github.mustachejava.codes.DefaultCode.appendText(DefaultCode.java:205) ~[?:?]
    at com.github.mustachejava.codes.DefaultCode.execute(DefaultCode.java:162) ~[?:?]
    at com.github.mustachejava.codes.ValueCode.execute(ValueCode.java:72) ~[?:?]
    at com.github.mustachejava.codes.DefaultMustache.run(DefaultMustache.java:34) ~[?:?]
    at com.github.mustachejava.codes.DefaultCode.run(DefaultCode.java:214) ~[?:?]
    at com.github.mustachejava.codes.IterableCode.next(IterableCode.java:143) ~[?:?]
    at com.github.mustachejava.reflect.BaseObjectHandler.iterate(BaseObjectHandler.java:81) ~[?:?]
    at com.github.mustachejava.codes.IterableCode.execute(IterableCode.java:138) ~[?:?]
    at com.github.mustachejava.codes.IterableCode.handle(IterableCode.java:54) ~[?:?]
    at com.github.mustachejava.codes.IterableCode.execute(IterableCode.java:42) ~[?:?]
    at com.github.mustachejava.codes.DefaultMustache.run(DefaultMustache.java:34) ~[?:?]
    at com.github.mustachejava.codes.DefaultCode.execute(DefaultCode.java:162) ~[?:?]
    at com.github.mustachejava.codes.DefaultMustache.execute(DefaultMustache.java:57) ~[?:?]
    at com.github.mustachejava.Mustache.execute(Mustache.java:38) ~[?:?]
    at org.elasticsearch.script.mustache.MustacheScriptEngine$MustacheExecutableScript.execute(MustacheScriptEngine.java:143) ~[?:?]
    ... 13 more

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[rjernst]

@stefnestor This mustache setting isn't a cluster setting, it's a node setting. The value should show up in the nodes info api.

However, one could argue it should be a cluster setting. Unfortunately it's a bit complex to do that right now, but we'll see if it's possible.

[stefnestor]

Thanks @rjernst ! 💚

Sorry, I assumed where it should output, but I confirm it also doesn't show in nodes.json. Let me take a step back and say for the internally linked example, I searched the Elasticsearch diagnostic for max_output_size_bytes and had no results, so AFAIK this setting does not show anywhere regardless if it's node-level or cluster-level.

🙂

[rjernst]

I suspect it's not showing up because the nodes info API only supports explicit values from elasticsearch.yml. It doesn't have the equivalent include_defaults parameter that the get cluster settings API has.