Skip to content

[BUG] carbonapi 0.15.3+ doesn't create separate targets for groupByNode #913

New issue
New issue
Open
Open
[BUG] carbonapi 0.15.3+ doesn't create separate targets for groupByNode#913
Labels
bug
@CTimmerman

Description

@CTimmerman
CTimmerman
opened on Dec 2, 2025

Describe the bug

I have working Grafana 11.5.2 dashboards using carbonapi 0.10.0.1.
Grafana 11.5.2 has a duplicated panel edit bug: grafana/grafana#98351
Grafana 11.6+ truncates labels with carbonapi 0.10.0.1: grafana/grafana#113265
carbonapi=0.15.3~1 fixes the label issue, but introduces a missing query result targets issue.

CarbonAPI Version
0.15.3~1 up to and including 0.18.2~1.

Logs
If applicable, add logs (please use log level debug) that shows the whole duration of the request. Request can be identified by UUID if needed.

WORKING CASE:
$ devvm down carbonapi; devvm build carbonapi; devvm up carbonapi; docker inspect docker-per-service-carbonapi-1 | grep IPAddress  # carbonapi=0.10.0.1
$ curl 'http://172.18.0.20:8085/render?target=groupByNode(aaa.*.*.*.bbb.ccc.*.count%2C%206%2C%20%22sum%22)&from=1764795955&until=1764796857&format=json&maxDataPoints=1144'
[{"target":"Main","datapoints":[[null,1764795960],[null,1764795970],[null,1764795980],[null,1764795990],[null,1764796000],[null,1764796010],[null,1764796020],[null,1764796030],[null,1764796040],[null,1764796050],[null,1764796060],[null,1764796070],[null,1764796080],[null,1764796090],[null,1764796100],[null,1764796110],[null,1764796120],[null,1764796130],[null,1764796140],[null,1764796150],[null,1764796160],[null,1764796170],[null,1764796180],[null,1764796190],[null,1764796200],[null,1764796210],[null,1764796220],[null,1764796230],[null,1764796240],[null,1764796250],[null,1764796260],[null,1764796270],[null,1764796280],[null,1764796290],[null,1764796300],[null,1764796310],[null,1764796320],[null,1764796330],[null,1764796340],[null,1764796350],[null,1764796360],[null,1764796370],[null,1764796380],[null,1764796390],[null,1764796400],[null,1764796410],[null,1764796420],[null,1764796430],[null,1764796440],[null,1764796450],[null,1764796460],[null,1764796470],[1,1764796480],[0,1764796490],[0,1764796500],[3,1764796510],[1,1764796520],[2,1764796530],[null,1764796540],[1,1764796550],[1,1764796560],[2,1764796570],[0,1764796580],[0,1764796590],[1,1764796600],[0,1764796610],[1,1764796620],[2,1764796630],[1,1764796640],[5,1764796650],[1,1764796660],[3,1764796670],[0,1764796680],[1,1764796690],[3,1764796700],[2,1764796710],[2,1764796720],[0,1764796730],[1,1764796740],[0,1764796750],[0,1764796760],[2,1764796770],[0,1764796780],[1,1764796790],[1,1764796800],[0,1764796810],[1,1764796820],[2,1764796830],[1,1764796840],[2,1764796850]]},{"target":"Mobile","datapoints":[[null,1764795960],[null,1764795970],[null,1764795980],[null,1764795990],[null,1764796000],[null,1764796010],[null,1764796020],[null,1764796030],[null,1764796040],[null,1764796050],[null,1764796060],[null,1764796070],[null,1764796080],[null,1764796090],[null,1764796100],[null,1764796110],[null,1764796120],[null,1764796130],[null,1764796140],[null,1764796150],[null,1764796160],[null,1764796170],[null,1764796180],[null,1764796190],[null,1764796200],[null,1764796210],[null,1764796220],[null,1764796230],[null,1764796240],[null,1764796250],[null,1764796260],[null,1764796270],[null,1764796280],[null,1764796290],[null,1764796300],[null,1764796310],[null,1764796320],[null,1764796330],[null,1764796340],[null,1764796350],[null,1764796360],[null,1764796370],[null,1764796380],[null,1764796390],[null,1764796400],[null,1764796410],[null,1764796420],[null,1764796430],[null,1764796440],[null,1764796450],[null,1764796460],[null,1764796470],[0,1764796480],[0,1764796490],[0,1764796500],[1,1764796510],[1,1764796520],[1,1764796530],[null,1764796540],[0,1764796550],[0,1764796560],[1,1764796570],[1,1764796580],[0,1764796590],[0,1764796600],[1,1764796610],[1,1764796620],[0,1764796630],[0,1764796640],[0,1764796650],[0,1764796660],[1,1764796670],[0,1764796680],[0,1764796690],[0,1764796700],[1,1764796710],[1,1764796720],[1,1764796730],[0,1764796740],[0,1764796750],[1,1764796760],[0,1764796770],[0,1764796780],[1,1764796790],[0,1764796800],[1,1764796810],[0,1764796820],[1,1764796830],[0,1764796840],[0,1764796850]]}]

$ devvm enter carbonapi
root@bcecbb736786:/# tail -f /var/log/carbonapi/carbonapi.log 
[2025-12-03T21:47:51.798Z] INFO [access] request served {"data": {"handler":"render","carbonapi_uuid":"3ee911b6-f1bf-4076-9141-e73eafdc115c","url":"/render?target=groupByNode(aaa.*.*.*.bbb.ccc.*.count%2C%206%2C%20%22sum%22)\u0026from=1764795955\u0026until=1764796857\u0026format=json\u0026maxDataPoints=1144","peer_ip":"172.18.0.1","peer_port":"52744","host":"172.18.0.20:8085","format":"json","use_cache":true,"targets":["groupByNode(aaa.*.*.*.bbb.ccc.*.count, 6, \"sum\")"],"cache_timeout":10,"metrics":["aaa.*.*.*.bbb.ccc.*.count"],"runtime":0.001139602,"http_code":200,"from":1764795955,"until":1764796857,"from_raw":"1764795955","until_raw":"1764796857","uri":"/render?target=groupByNode(aaa.*.*.*.bbb.ccc.*.count%2C%206%2C%20%22sum%22)\u0026from=1764795955\u0026until=1764796857\u0026format=json\u0026maxDataPoints=1144","zipper_requests":3}}
[2025-12-03T21:49:40.147Z] INFO [access] request served {"data": {"handler":"render","carbonapi_uuid":"a1a585a5-cef3-455f-aab0-8b19ca0ee665","url":"/render?target=groupByNode(aaa.*.*.*.bbb.ccc.*.count%2C%206%2C%20%22sum%22)\u0026from=1764795955\u0026until=1764796857\u0026format=json\u0026maxDataPoints=1144","peer_ip":"172.18.0.1","peer_port":"44248","host":"172.18.0.20:8085","format":"json","use_cache":true,"targets":["groupByNode(aaa.*.*.*.bbb.ccc.*.count, 6, \"sum\")"],"cache_timeout":10,"metrics":["aaa.*.*.*.bbb.ccc.*.count"],"runtime":0.000931206,"http_code":200,"from":1764795955,"until":1764796857,"from_raw":"1764795955","until_raw":"1764796857","uri":"/render?target=groupByNode(aaa.*.*.*.bbb.ccc.*.count%2C%206%2C%20%22sum%22)\u0026from=1764795955\u0026until=1764796857\u0026format=json\u0026maxDataPoints=1144","zipper_requests":2}}
[2025-12-03T21:51:31.009Z] INFO [access] request served {"data": {"handler":"render","carbonapi_uuid":"543ec41e-87ab-4bdf-81ff-0af7c36deab5","url":"/render?target=groupByNode(aaa.*.*.*.bbb.ccc.*.count%2C%206%2C%20%22sum%22)\u0026from=1764795955\u0026until=1764796857\u0026format=json\u0026maxDataPoints=1144","peer_ip":"172.18.0.1","peer_port":"48582","host":"172.18.0.20:8085","format":"json","use_cache":true,"targets":["groupByNode(aaa.*.*.*.bbb.ccc.*.count, 6, \"sum\")"],"cache_timeout":10,"metrics":["aaa.*.*.*.bbb.ccc.*.count"],"runtime":0.001503202,"http_code":200,"from":1764795955,"until":1764796857,"from_raw":"1764795955","until_raw":"1764796857","uri":"/render?target=groupByNode(aaa.*.*.*.bbb.ccc.*.count%2C%206%2C%20%22sum%22)\u0026from=1764795955\u0026until=1764796857\u0026format=json\u0026maxDataPoints=1144","zipper_requests":2}}


BROKEN CASES:
$ devvm down carbonapi; devvm build carbonapi; devvm up carbonapi; docker inspect docker-per-service-carbonapi-1 | grep IPAddress  # carbonapi=0.15.3~1
$ curl 'http://172.18.0.20:8085/render?target=groupByNode(aaa.*.*.*.bbb.ccc.*.count%2C%206%2C%20%22sum%22)&from=1764795955&until=1764796857&format=json&maxDataPoints=1144'
[{"target":"Mobile","datapoints":[[null,1764795960],[null,1764795970],[null,1764795980],[null,1764795990],[null,1764796000],[null,1764796010],[null,1764796020],[null,1764796030],[null,1764796040],[null,1764796050],[null,1764796060],[null,1764796070],[null,1764796080],[null,1764796090],[null,1764796100],[null,1764796110],[null,1764796120],[null,1764796130],[null,1764796140],[null,1764796150],[null,1764796160],[null,1764796170],[null,1764796180],[null,1764796190],[null,1764796200],[null,1764796210],[null,1764796220],[null,1764796230],[null,1764796240],[null,1764796250],[null,1764796260],[null,1764796270],[null,1764796280],[null,1764796290],[null,1764796300],[null,1764796310],[null,1764796320],[null,1764796330],[null,1764796340],[null,1764796350],[null,1764796360],[null,1764796370],[null,1764796380],[null,1764796390],[null,1764796400],[null,1764796410],[null,1764796420],[null,1764796430],[null,1764796440],[null,1764796450],[null,1764796460],[null,1764796470],[0,1764796480],[0,1764796490],[0,1764796500],[1,1764796510],[1,1764796520],[1,1764796530],[null,1764796540],[0,1764796550],[0,1764796560],[1,1764796570],[1,1764796580],[0,1764796590],[0,1764796600],[1,1764796610],[1,1764796620],[0,1764796630],[0,1764796640],[0,1764796650],[0,1764796660],[1,1764796670],[0,1764796680],[0,1764796690],[0,1764796700],[1,1764796710],[1,1764796720],[1,1764796730],[0,1764796740],[0,1764796750],[1,1764796760],[0,1764796770],[0,1764796780],[1,1764796790],[0,1764796800],[1,1764796810],[0,1764796820],[1,1764796830],[0,1764796840],[0,1764796850]],"tags":{"name":"Mobile"}}]

$ devvm enter carbonapi
root@13dd7c007f8a:/# tail -f /var/log/carbonapi/carbonapi.log 
[2025-12-03T21:54:37.294Z] DEBUG [functionInit] no config file specified {"function": "graphiteWeb", "message": "this function requrires config file to work properly"}
[2025-12-03T21:54:37.295Z] DEBUG [zipper] initializing zipper {}
[2025-12-03T21:54:37.295Z] DEBUG [zipper] creating lb group {"name": "backends", "servers": ["http://go-carbon:8080"], "type": "broadcast"}
[2025-12-03T21:54:37.295Z] DEBUG [dns] no caching dns initialized, will return typical DialContext {}
[2025-12-03T21:54:37.295Z] DEBUG [zipper] zipper config {"config": {"Buckets":10,"SlowLogThreshold":1000000000,"ConcurrencyLimitPerServer":0,"MaxIdleConnsPerHost":100,"Backends":["http://go-carbon:8080"],"BackendsV2":{"Backends":[{"GroupName":"backends","Protocol":"carbonapi_v2_pb","LBMethod":"broadcast","Servers":["http://go-carbon:8080"],"Timeouts":{"Find":2000000000,"Render":10000000000000,"Connect":9900000000},"ConcurrencyLimit":0,"KeepAliveInterval":30000000000,"MaxIdleConnsPerHost":100,"MaxTries":0,"MaxBatchSize":1,"BackendOptions":null,"ForceAttemptHTTP2":false,"DoMultipleRequestsIfSplit":true}],"MaxIdleConnsPerHost":100,"ConcurrencyLimitPerServer":0,"Timeouts":{"Find":2000000000,"Render":10000000000000,"Connect":9900000000},"KeepAliveInterval":30000000000,"MaxTries":0,"MaxBatchSize":1},"MaxBatchSize":1,"FallbackMaxBatchSize":1,"MaxTries":0,"DoMultipleRequestsIfSplit":true,"CarbonSearch":{"Backend":"http://127.0.0.1:8070","Prefix":"virt.v1.*"},"CarbonSearchV2":{"Backends":[{"GroupName":"http://127.0.0.1:8070","Protocol":"carbonapi_v2_pb","LBMethod":"roundrobin","Servers":["http://127.0.0.1:8070"],"Timeouts":{"Find":2000000000,"Render":10000000000000,"Connect":9900000000},"ConcurrencyLimit":0,"KeepAliveInterval":30000000000,"MaxIdleConnsPerHost":100,"MaxTries":0,"MaxBatchSize":1,"BackendOptions":null,"ForceAttemptHTTP2":false,"DoMultipleRequestsIfSplit":true}],"MaxIdleConnsPerHost":100,"ConcurrencyLimitPerServer":0,"Timeouts":{"Find":2000000000,"Render":10000000000000,"Connect":9900000000},"KeepAliveInterval":30000000000,"MaxTries":0,"MaxBatchSize":1,"Prefix":"virt.v1.*"},"ExpireDelaySec":0,"TLDCacheDisabled":false,"InternalRoutingCache":600000000000,"Timeouts":{"Find":2000000000,"Render":10000000000000,"Connect":9900000000},"KeepAliveInterval":30000000000,"ScaleToCommonStep":true}}
[2025-12-03T21:54:37.295Z] DEBUG [zipper] doing request {"type": "protoV2Group", "name": "http://go-carbon:8080", "function": "prober", "request": ["*"]}
[2025-12-03T21:54:37.295Z] DEBUG [zipper] will do query {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "find", "request": ["*"], "carbonapi_uuid": "", "query": "*"}
[2025-12-03T21:54:37.295Z] DEBUG [zipper] trying to get slot {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "find", "request": ["*"], "carbonapi_uuid": "", "function": "HttpQuery.doRequest", "server": "http://go-carbon:8080", "name": "http://go-carbon:8080", "uri": "http://go-carbon:8080/metrics/find/?format=protobuf&query=%2A", "name": "http://go-carbon:8080"}
[2025-12-03T21:54:37.295Z] DEBUG [zipper] got slot for server {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "find", "request": ["*"], "carbonapi_uuid": "", "function": "HttpQuery.doRequest", "server": "http://go-carbon:8080", "name": "http://go-carbon:8080", "uri": "http://go-carbon:8080/metrics/find/?format=protobuf&query=%2A", "name": "http://go-carbon:8080"}
[2025-12-03T21:54:37.297Z] DEBUG [zipper] will return data {"type": "protoV2Group", "name": "http://go-carbon:8080", "function": "prober", "tlds": ["carbon", "aaa"]}
[2025-12-03T21:58:01.092Z] DEBUG [zipper] will try to fetch data {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09"}
[2025-12-03T21:58:01.092Z] DEBUG [zipper] single fetch {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "client": {}}
[2025-12-03T21:58:01.092Z] DEBUG [zipper] will do query with timeout {"type": "broadcastGroup", "groupName": "backends", "type": "find", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "backends": [{}], "timeout": 2}
[2025-12-03T21:58:01.092Z] DEBUG [zipper] single fetch {"type": "broadcastGroup", "groupName": "backends", "type": "find", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "client": {}}
[2025-12-03T21:58:01.092Z] DEBUG [zipper] waiting for a slot {"type": "broadcastGroup", "groupName": "backends", "type": "find", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "group_name": "backends", "backend_name": "http://go-carbon:8080"}
[2025-12-03T21:58:01.092Z] DEBUG [zipper] got slot {"type": "broadcastGroup", "groupName": "backends", "type": "find", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "group_name": "backends", "backend_name": "http://go-carbon:8080"}
[2025-12-03T21:58:01.092Z] DEBUG [zipper] will do query {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "find", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "query": "aaa.*.*.*.bbb.ccc.*.count"}
[2025-12-03T21:58:01.092Z] DEBUG [zipper] trying to get slot {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "find", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "function": "HttpQuery.doRequest", "server": "http://go-carbon:8080", "name": "http://go-carbon:8080", "uri": "http://go-carbon:8080/metrics/find/?format=protobuf&query=aaa.%2A.%2A.%2A.bbb.ccc.%2A.count", "name": "http://go-carbon:8080"}
[2025-12-03T21:58:01.092Z] DEBUG [zipper] got slot for server {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "find", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "function": "HttpQuery.doRequest", "server": "http://go-carbon:8080", "name": "http://go-carbon:8080", "uri": "http://go-carbon:8080/metrics/find/?format=protobuf&query=aaa.%2A.%2A.%2A.bbb.ccc.%2A.count", "name": "http://go-carbon:8080"}
[2025-12-03T21:58:01.093Z] DEBUG [zipper] fetched response {"type": "broadcastGroup", "groupName": "backends", "type": "find", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "group_name": "backends", "backend_name": "http://go-carbon:8080", "response": {"Server":"http://go-carbon:8080","Response":{"metrics":[{"name":"aaa.*.*.*.bbb.ccc.*.count","matches":[{"path":"aaa.ddd.dev.gateway.bbb.ccc.Main.count","isLeaf":true},{"path":"aaa.ddd.dev.gateway.bbb.ccc.Mobile.count","isLeaf":true}]}]},"Stats":{"Timeouts":0,"FindRequests":1,"FindErrors":0,"FindTimeouts":0,"RenderRequests":0,"RenderErrors":0,"RenderTimeouts":0,"InfoRequests":0,"InfoErrors":0,"InfoTimeouts":0,"SearchRequests":0,"SearchCacheHits":0,"SearchCacheMisses":0,"ZipperRequests":0,"TotalMetricsCount":0,"MemoryUsage":0,"CacheMisses":0,"CacheHits":0,"Servers":["http://go-carbon:8080"],"FailedServers":null},"Err":null}}
[2025-12-03T21:58:01.093Z] DEBUG [zipper] got some find responses {"type": "broadcastGroup", "groupName": "backends", "type": "find", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "backends_count": 1, "response_count": 1, "have_errors": false, "errors": null, "response": "&MultiGlobResponse{Metrics:[{aaa.*.*.*.bbb.ccc.*.count [{aaa.ddd.dev.gateway.bbb.ccc.Main.count true} {aaa.ddd.dev.gateway.bbb.ccc.Mobile.count true}]}],}"}
[2025-12-03T21:58:01.093Z] DEBUG [zipper] waiting for slot {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "multi_fetch": true, "backend_name": "backends", "max_connections": 0}
[2025-12-03T21:58:01.093Z] DEBUG [zipper] got slot {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "multi_fetch": true, "backend_name": "backends"}
[2025-12-03T21:58:01.093Z] DEBUG [zipper] sending request {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "multi_fetch": true, "backend_name": "backends"}
[2025-12-03T21:58:01.093Z] DEBUG [zipper] waiting for slot {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "multi_fetch": true, "backend_name": "backends", "max_connections": 0}
[2025-12-03T21:58:01.093Z] DEBUG [zipper] will try to fetch data {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Mobile.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09"}
[2025-12-03T21:58:01.093Z] DEBUG [zipper] got slot {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "multi_fetch": true, "backend_name": "backends"}
[2025-12-03T21:58:01.093Z] DEBUG [zipper] sending request {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "multi_fetch": true, "backend_name": "backends"}
[2025-12-03T21:58:01.093Z] DEBUG [zipper] will try to fetch data {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Main.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09"}
[2025-12-03T21:58:01.093Z] DEBUG [zipper] single fetch {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Mobile.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "client": {}}
[2025-12-03T21:58:01.093Z] DEBUG [zipper] single fetch {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Main.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "client": {}}
[2025-12-03T21:58:01.093Z] DEBUG [zipper] waiting for slot {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Mobile.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "multi_fetch": true, "backend_name": "http://go-carbon:8080", "max_connections": 0}
[2025-12-03T21:58:01.093Z] DEBUG [zipper] got slot {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Mobile.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "multi_fetch": true, "backend_name": "http://go-carbon:8080"}
[2025-12-03T21:58:01.093Z] DEBUG [zipper] sending request {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Mobile.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "multi_fetch": true, "backend_name": "http://go-carbon:8080"}
[2025-12-03T21:58:01.093Z] DEBUG [zipper] waiting for slot {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Main.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "multi_fetch": true, "backend_name": "http://go-carbon:8080", "max_connections": 0}
[2025-12-03T21:58:01.093Z] DEBUG [zipper] got slot {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Main.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "multi_fetch": true, "backend_name": "http://go-carbon:8080"}
[2025-12-03T21:58:01.093Z] DEBUG [zipper] sending request {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Main.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "multi_fetch": true, "backend_name": "http://go-carbon:8080"}
[2025-12-03T21:58:01.093Z] DEBUG [zipper] trying to get slot {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[{aaa.ddd.dev.gateway.bbb.ccc.Main.count 1764795955 1764796857 false aaa.*.*.*.bbb.ccc.*.count [] 0}],}", "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "function": "HttpQuery.doRequest", "server": "http://go-carbon:8080", "name": "http://go-carbon:8080", "uri": "http://go-carbon:8080/render/?format=protobuf&from=1764795955&target=aaa.ddd.dev.gateway.bbb.ccc.Main.count&until=1764796857", "name": "http://go-carbon:8080"}
[2025-12-03T21:58:01.093Z] DEBUG [zipper] trying to get slot {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[{aaa.ddd.dev.gateway.bbb.ccc.Mobile.count 1764795955 1764796857 false aaa.*.*.*.bbb.ccc.*.count [] 0}],}", "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "function": "HttpQuery.doRequest", "server": "http://go-carbon:8080", "name": "http://go-carbon:8080", "uri": "http://go-carbon:8080/render/?format=protobuf&from=1764795955&target=aaa.ddd.dev.gateway.bbb.ccc.Mobile.count&until=1764796857", "name": "http://go-carbon:8080"}
[2025-12-03T21:58:01.093Z] DEBUG [zipper] got slot for server {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[{aaa.ddd.dev.gateway.bbb.ccc.Main.count 1764795955 1764796857 false aaa.*.*.*.bbb.ccc.*.count [] 0}],}", "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "function": "HttpQuery.doRequest", "server": "http://go-carbon:8080", "name": "http://go-carbon:8080", "uri": "http://go-carbon:8080/render/?format=protobuf&from=1764795955&target=aaa.ddd.dev.gateway.bbb.ccc.Main.count&until=1764796857", "name": "http://go-carbon:8080"}
[2025-12-03T21:58:01.093Z] DEBUG [zipper] got slot for server {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[{aaa.ddd.dev.gateway.bbb.ccc.Mobile.count 1764795955 1764796857 false aaa.*.*.*.bbb.ccc.*.count [] 0}],}", "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "function": "HttpQuery.doRequest", "server": "http://go-carbon:8080", "name": "http://go-carbon:8080", "uri": "http://go-carbon:8080/render/?format=protobuf&from=1764795955&target=aaa.ddd.dev.gateway.bbb.ccc.Mobile.count&until=1764796857", "name": "http://go-carbon:8080"}
[2025-12-03T21:58:01.094Z] DEBUG [zipper] got response {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Mobile.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "multi_fetch": true, "backend_name": "http://go-carbon:8080"}
[2025-12-03T21:58:01.094Z] DEBUG [zipper] got some fetch responses {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Mobile.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "backends_count": 1, "response_count": 1, "have_errors": false, "errors": null, "response_count": 1}
[2025-12-03T21:58:01.094Z] DEBUG [zipper] got response {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "multi_fetch": true, "backend_name": "backends"}
[2025-12-03T21:58:01.094Z] DEBUG [zipper] got some fetch responses {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "backends_count": 1, "response_count": 1, "have_errors": false, "errors": null, "response_count": 1}
[2025-12-03T21:58:01.094Z] DEBUG [zipper] errors while fetching data from backends {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Main.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "httpCode": 503, "errors": ["timeout while fetching Response"]}
[2025-12-03T21:58:01.094Z] DEBUG [zipper] got response {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "multi_fetch": true, "backend_name": "backends"}
[2025-12-03T21:58:01.094Z] DEBUG [zipper] error fetching result {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[{aaa.ddd.dev.gateway.bbb.ccc.Main.count 1764795955 1764796857 false aaa.*.*.*.bbb.ccc.*.count [] 0}],}", "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "function": "HttpQuery.doRequest", "server": "http://go-carbon:8080", "name": "http://go-carbon:8080", "uri": "http://go-carbon:8080/render/?format=protobuf&from=1764795955&target=aaa.ddd.dev.gateway.bbb.ccc.Main.count&until=1764796857", "error": "Get \"http://go-carbon:8080/render/?format=protobuf&from=1764795955&target=aaa.ddd.dev.gateway.bbb.ccc.Main.count&until=1764796857\": context canceled"}
[2025-12-03T21:58:01.094Z] DEBUG [zipper] have errors {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[{aaa.ddd.dev.gateway.bbb.ccc.Main.count 1764795955 1764796857 false aaa.*.*.*.bbb.ccc.*.count [] 0}],}", "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "error": "error while fetching Response", "errorVerbose": "error while fetching Response\n\ngithub.com/go-graphite/carbonapi/zipper/types.init\n\t/root/go/src/github.com/go-graphite/carbonapi/zipper/types/errors.go:34\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6309\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:208\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371\n"}
[2025-12-03T21:58:01.094Z] WARN [zipper] errors occurred while getting results {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[{aaa.ddd.dev.gateway.bbb.ccc.Main.count 1764795955 1764796857 false aaa.*.*.*.bbb.ccc.*.count [] 0}],}", "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "errors": "max tries exceeded: failed to fetch data from server/group: error while fetching Response", "errorsVerbose": "max tries exceeded\n\ngithub.com/go-graphite/carbonapi/zipper/types.init\n\t/root/go/src/github.com/go-graphite/carbonapi/zipper/types/errors.go:25\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6309\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:208\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371\n\n\nCaused By: failed to fetch data from server/group\n\ngithub.com/go-graphite/carbonapi/zipper/types.init\n\t/root/go/src/github.com/go-graphite/carbonapi/zipper/types/errors.go:27\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6309\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:208\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371\n\n\nCaused By: error while fetching Response\n\ngithub.com/go-graphite/carbonapi/zipper/types.init\n\t/root/go/src/github.com/go-graphite/carbonapi/zipper/types/errors.go:34\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6309\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6286\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:208\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371\n"}
[2025-12-03T21:58:01.094Z] DEBUG [zipper] got response {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Main.count"], "carbonapi_uuid": "19bd2d54-6de0-4e99-82bf-809cfed31d09", "multi_fetch": true, "backend_name": "http://go-carbon:8080"}
[2025-12-03T21:58:01.094Z] INFO [access] request served {"data": {"handler":"render","carbonapi_uuid":"19bd2d54-6de0-4e99-82bf-809cfed31d09","url":"/render?target=groupByNode(aaa.*.*.*.bbb.ccc.*.count%2C%206%2C%20%22sum%22)\u0026from=1764795955\u0026until=1764796857\u0026format=json\u0026maxDataPoints=1144","peer_ip":"172.18.0.1","peer_port":"42604","host":"172.18.0.20:8085","format":"json","use_cache":true,"targets":["groupByNode(aaa.*.*.*.bbb.ccc.*.count, 6, \"sum\")"],"cache_timeout":10,"metrics":["groupByNode(aaa.*.*.*.bbb.ccc.*.count, 6, \"sum\")"],"runtime":0.002417083,"http_code":200,"carbonzipper_response_size_bytes":810,"carbonapi_response_size_bytes":1570,"from":1764795955,"until":1764796857,"max_data_points":1144,"from_raw":"1764795955","until_raw":"1764796857","uri":"/render?target=groupByNode(aaa.*.*.*.bbb.ccc.*.count%2C%206%2C%20%22sum%22)\u0026from=1764795955\u0026until=1764796857\u0026format=json\u0026maxDataPoints=1144","from_cache":false,"used_backend_cache":false,"request_headers":{}}}


$ devvm down carbonapi; devvm build carbonapi; devvm up carbonapi; docker inspect docker-per-service-carbonapi-1 | grep IPAddress  # 0.18.2
$ curl 'http://172.18.0.20:8085/render?target=groupByNode(aaa.*.*.*.bbb.ccc.*.count%2C%206%2C%20%22sum%22)&from=1764795955&until=1764796857&format=json&maxDataPoints=1144'
[{"target":"Mobile","datapoints":[[null,1764795960],[null,1764795970],[null,1764795980],[null,1764795990],[null,1764796000],[null,1764796010],[null,1764796020],[null,1764796030],[null,1764796040],[null,1764796050],[null,1764796060],[null,1764796070],[null,1764796080],[null,1764796090],[null,1764796100],[null,1764796110],[null,1764796120],[null,1764796130],[null,1764796140],[null,1764796150],[null,1764796160],[null,1764796170],[null,1764796180],[null,1764796190],[null,1764796200],[null,1764796210],[null,1764796220],[null,1764796230],[null,1764796240],[null,1764796250],[null,1764796260],[null,1764796270],[null,1764796280],[null,1764796290],[null,1764796300],[null,1764796310],[null,1764796320],[null,1764796330],[null,1764796340],[null,1764796350],[null,1764796360],[null,1764796370],[null,1764796380],[null,1764796390],[null,1764796400],[null,1764796410],[null,1764796420],[null,1764796430],[null,1764796440],[null,1764796450],[null,1764796460],[null,1764796470],[0,1764796480],[0,1764796490],[0,1764796500],[1,1764796510],[1,1764796520],[1,1764796530],[null,1764796540],[0,1764796550],[0,1764796560],[1,1764796570],[1,1764796580],[0,1764796590],[0,1764796600],[1,1764796610],[1,1764796620],[0,1764796630],[0,1764796640],[0,1764796650],[0,1764796660],[1,1764796670],[0,1764796680],[0,1764796690],[0,1764796700],[1,1764796710],[1,1764796720],[1,1764796730],[0,1764796740],[0,1764796750],[1,1764796760],[0,1764796770],[0,1764796780],[1,1764796790],[0,1764796800],[1,1764796810],[0,1764796820],[1,1764796830],[0,1764796840],[0,1764796850]],"tags":{"aggregatedBy":"sum","name":"Mobile"}}]

$ devvm enter carbonapi
root@8ed7c5cabb9d:/# tail -f /var/log/carbonapi/carbonapi.log 
[2025-12-03T22:05:01.831Z] DEBUG [zipper] initializing zipper {}
[2025-12-03T22:05:01.831Z] DEBUG [zipper] creating lb group {"name": "backends", "servers": ["http://go-carbon:8080"], "type": "broadcast"}
[2025-12-03T22:05:01.832Z] DEBUG [dns] no caching dns initialized, will return typical DialContext {}
[2025-12-03T22:05:01.832Z] ERROR [zipper] DEBUG ERROR LOGGGGG {"cfg": {"SumBuckets":false,"Buckets":10,"BucketsWidth":[],"BucketsLabels":[],"ExtendedStat":false,"SlowLogThreshold":1000000000,"ConcurrencyLimitPerServer":0,"MaxIdleConnsPerHost":100,"Backends":["http://go-carbon:8080"],"BackendsV2":{"Backends":[{"GroupName":"backends","Protocol":"carbonapi_v2_pb","LBMethod":"broadcast","Servers":["http://go-carbon:8080"],"Timeouts":{"Find":2000000000,"Render":10000000000,"Connect":9900000000},"ConcurrencyLimit":0,"KeepAliveInterval":30000000000,"MaxIdleConnsPerHost":100,"MaxTries":0,"MaxBatchSize":1,"BackendOptions":null,"ForceAttemptHTTP2":false,"DoMultipleRequestsIfSplit":true,"IdleConnectionTimeout":3600000000000,"TLSClientConfig":null}],"MaxIdleConnsPerHost":100,"ConcurrencyLimitPerServer":0,"Timeouts":{"Find":2000000000,"Render":10000000000,"Connect":9900000000},"KeepAliveInterval":30000000000,"MaxTries":0,"MaxBatchSize":1},"MaxBatchSize":1,"FallbackMaxBatchSize":1,"MaxTries":0,"DoMultipleRequestsIfSplit":true,"RequireSuccessAll":false,"ExpireDelaySec":0,"TLDCacheDisabled":false,"InternalRoutingCache":600000000000,"Timeouts":{"Find":2000000000,"Render":10000000000,"Connect":9900000000},"KeepAliveInterval":30000000000,"ScaleToCommonStep":true}}
[2025-12-03T22:05:01.832Z] DEBUG [zipper] zipper config {"config": {"SumBuckets":false,"Buckets":10,"BucketsWidth":[],"BucketsLabels":[],"ExtendedStat":false,"SlowLogThreshold":1000000000,"ConcurrencyLimitPerServer":0,"MaxIdleConnsPerHost":100,"Backends":["http://go-carbon:8080"],"BackendsV2":{"Backends":[{"GroupName":"backends","Protocol":"carbonapi_v2_pb","LBMethod":"broadcast","Servers":["http://go-carbon:8080"],"Timeouts":{"Find":2000000000,"Render":10000000000,"Connect":9900000000},"ConcurrencyLimit":0,"KeepAliveInterval":30000000000,"MaxIdleConnsPerHost":100,"MaxTries":0,"MaxBatchSize":1,"BackendOptions":null,"ForceAttemptHTTP2":false,"DoMultipleRequestsIfSplit":true,"IdleConnectionTimeout":3600000000000,"TLSClientConfig":null}],"MaxIdleConnsPerHost":100,"ConcurrencyLimitPerServer":0,"Timeouts":{"Find":2000000000,"Render":10000000000,"Connect":9900000000},"KeepAliveInterval":30000000000,"MaxTries":0,"MaxBatchSize":1},"MaxBatchSize":1,"FallbackMaxBatchSize":1,"MaxTries":0,"DoMultipleRequestsIfSplit":true,"RequireSuccessAll":false,"ExpireDelaySec":0,"TLDCacheDisabled":false,"InternalRoutingCache":600000000000,"Timeouts":{"Find":2000000000,"Render":10000000000,"Connect":9900000000},"KeepAliveInterval":30000000000,"ScaleToCommonStep":true}}
[2025-12-03T22:05:01.832Z] DEBUG [zipper] doing request {"type": "protoV2Group", "name": "http://go-carbon:8080", "function": "prober", "request": ["*"]}
[2025-12-03T22:05:01.832Z] DEBUG [zipper] will do query {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "find", "request": ["*"], "carbonapi_uuid": "", "query": "*"}
[2025-12-03T22:05:01.832Z] DEBUG [zipper] trying to get slot {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "find", "request": ["*"], "carbonapi_uuid": "", "function": "HttpQuery.doRequest", "server": "http://go-carbon:8080", "name": "http://go-carbon:8080", "uri": "http://go-carbon:8080/metrics/find/?format=protobuf&query=%2A", "name": "http://go-carbon:8080"}
[2025-12-03T22:05:01.832Z] DEBUG [zipper] got slot for server {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "find", "request": ["*"], "carbonapi_uuid": "", "function": "HttpQuery.doRequest", "server": "http://go-carbon:8080", "name": "http://go-carbon:8080", "uri": "http://go-carbon:8080/metrics/find/?format=protobuf&query=%2A", "name": "http://go-carbon:8080"}
[2025-12-03T22:05:01.834Z] DEBUG [zipper] will return data {"type": "protoV2Group", "name": "http://go-carbon:8080", "function": "prober", "tlds": ["carbon", "aaa"]}
[2025-12-03T22:06:15.703Z] DEBUG [zipper] will try to fetch data {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29"}
[2025-12-03T22:06:15.703Z] DEBUG [zipper] single fetch {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "client": {}}
[2025-12-03T22:06:15.703Z] DEBUG [zipper] will do query with timeout {"type": "broadcastGroup", "groupName": "backends", "type": "find", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "backends": [{}], "timeout": 2}
[2025-12-03T22:06:15.703Z] DEBUG [zipper] single fetch {"type": "broadcastGroup", "groupName": "backends", "type": "find", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "client": {}}
[2025-12-03T22:06:15.703Z] DEBUG [zipper] waiting for a slot {"type": "broadcastGroup", "groupName": "backends", "type": "find", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "group_name": "backends", "backend_name": "http://go-carbon:8080"}
[2025-12-03T22:06:15.703Z] DEBUG [zipper] got slot {"type": "broadcastGroup", "groupName": "backends", "type": "find", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "group_name": "backends", "backend_name": "http://go-carbon:8080"}
[2025-12-03T22:06:15.703Z] DEBUG [zipper] will do query {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "find", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "query": "aaa.*.*.*.bbb.ccc.*.count"}
[2025-12-03T22:06:15.703Z] DEBUG [zipper] trying to get slot {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "find", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "function": "HttpQuery.doRequest", "server": "http://go-carbon:8080", "name": "http://go-carbon:8080", "uri": "http://go-carbon:8080/metrics/find/?format=protobuf&query=aaa.%2A.%2A.%2A.bbb.ccc.%2A.count", "name": "http://go-carbon:8080"}
[2025-12-03T22:06:15.703Z] DEBUG [zipper] got slot for server {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "find", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "function": "HttpQuery.doRequest", "server": "http://go-carbon:8080", "name": "http://go-carbon:8080", "uri": "http://go-carbon:8080/metrics/find/?format=protobuf&query=aaa.%2A.%2A.%2A.bbb.ccc.%2A.count", "name": "http://go-carbon:8080"}
[2025-12-03T22:06:15.704Z] DEBUG [zipper] fetched response {"type": "broadcastGroup", "groupName": "backends", "type": "find", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "group_name": "backends", "backend_name": "http://go-carbon:8080", "response_size": 181}
[2025-12-03T22:06:15.704Z] DEBUG [zipper] got some find responses {"type": "broadcastGroup", "groupName": "backends", "type": "find", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "backends_count": 1, "response_count": 1, "have_errors": false, "errors": null, "response": "&MultiGlobResponse{Metrics:[]GlobResponse{GlobResponse{Name:aaa.*.*.*.bbb.ccc.*.count,Matches:[]GlobMatch{GlobMatch{Path:aaa.ddd.dev.gateway.bbb.ccc.Main.count,IsLeaf:true,},GlobMatch{Path:aaa.ddd.dev.gateway.bbb.ccc.Mobile.count,IsLeaf:true,},},},},}"}
[2025-12-03T22:06:15.704Z] DEBUG [zipper] waiting for slot {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "multi_fetch": true, "backend_name": "backends", "max_connections": 0}
[2025-12-03T22:06:15.704Z] DEBUG [zipper] got slot {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "multi_fetch": true, "backend_name": "backends", "backend_name": "backends"}
[2025-12-03T22:06:15.704Z] DEBUG [zipper] waiting for slot {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "multi_fetch": true, "backend_name": "backends", "backend_name": "backends", "max_connections": 0}
[2025-12-03T22:06:15.704Z] DEBUG [zipper] sending request {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "multi_fetch": true, "backend_name": "backends", "backend_name": "backends"}
[2025-12-03T22:06:15.704Z] DEBUG [zipper] got slot {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "multi_fetch": true, "backend_name": "backends", "backend_name": "backends"}
[2025-12-03T22:06:15.704Z] DEBUG [zipper] will try to fetch data {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Mobile.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29"}
[2025-12-03T22:06:15.704Z] DEBUG [zipper] sending request {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "multi_fetch": true, "backend_name": "backends", "backend_name": "backends"}
[2025-12-03T22:06:15.704Z] DEBUG [zipper] single fetch {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Mobile.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "client": {}}
[2025-12-03T22:06:15.704Z] DEBUG [zipper] will try to fetch data {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Main.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29"}
[2025-12-03T22:06:15.704Z] DEBUG [zipper] single fetch {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Main.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "client": {}}
[2025-12-03T22:06:15.704Z] DEBUG [zipper] waiting for slot {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Mobile.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "multi_fetch": true, "backend_name": "http://go-carbon:8080", "max_connections": 0}
[2025-12-03T22:06:15.704Z] DEBUG [zipper] got slot {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Mobile.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "multi_fetch": true, "backend_name": "http://go-carbon:8080"}
[2025-12-03T22:06:15.704Z] DEBUG [zipper] sending request {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Mobile.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "multi_fetch": true, "backend_name": "http://go-carbon:8080"}
[2025-12-03T22:06:15.704Z] DEBUG [zipper] waiting for slot {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Main.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "multi_fetch": true, "backend_name": "http://go-carbon:8080", "max_connections": 0}
[2025-12-03T22:06:15.704Z] DEBUG [zipper] got slot {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Main.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "multi_fetch": true, "backend_name": "http://go-carbon:8080"}
[2025-12-03T22:06:15.704Z] DEBUG [zipper] trying to get slot {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[]FetchRequest{FetchRequest{Name:aaa.ddd.dev.gateway.bbb.ccc.Mobile.count,StartTime:1764795955,StopTime:1764796857,HighPrecisionTimestamps:false,PathExpression:aaa.*.*.*.bbb.ccc.*.count,FilterFunctions:[]*FilteringFunction{},MaxDataPoints:0,},},}", "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "function": "HttpQuery.doRequest", "server": "http://go-carbon:8080", "name": "http://go-carbon:8080", "uri": "http://go-carbon:8080/render/?format=protobuf&from=1764795955&target=aaa.ddd.dev.gateway.bbb.ccc.Mobile.count&until=1764796857", "name": "http://go-carbon:8080"}
[2025-12-03T22:06:15.704Z] DEBUG [zipper] sending request {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Main.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "multi_fetch": true, "backend_name": "http://go-carbon:8080"}
[2025-12-03T22:06:15.704Z] DEBUG [zipper] got slot for server {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[]FetchRequest{FetchRequest{Name:aaa.ddd.dev.gateway.bbb.ccc.Mobile.count,StartTime:1764795955,StopTime:1764796857,HighPrecisionTimestamps:false,PathExpression:aaa.*.*.*.bbb.ccc.*.count,FilterFunctions:[]*FilteringFunction{},MaxDataPoints:0,},},}", "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "function": "HttpQuery.doRequest", "server": "http://go-carbon:8080", "name": "http://go-carbon:8080", "uri": "http://go-carbon:8080/render/?format=protobuf&from=1764795955&target=aaa.ddd.dev.gateway.bbb.ccc.Mobile.count&until=1764796857", "name": "http://go-carbon:8080"}
[2025-12-03T22:06:15.704Z] DEBUG [zipper] trying to get slot {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[]FetchRequest{FetchRequest{Name:aaa.ddd.dev.gateway.bbb.ccc.Main.count,StartTime:1764795955,StopTime:1764796857,HighPrecisionTimestamps:false,PathExpression:aaa.*.*.*.bbb.ccc.*.count,FilterFunctions:[]*FilteringFunction{},MaxDataPoints:0,},},}", "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "function": "HttpQuery.doRequest", "server": "http://go-carbon:8080", "name": "http://go-carbon:8080", "uri": "http://go-carbon:8080/render/?format=protobuf&from=1764795955&target=aaa.ddd.dev.gateway.bbb.ccc.Main.count&until=1764796857", "name": "http://go-carbon:8080"}
[2025-12-03T22:06:15.704Z] DEBUG [zipper] got slot for server {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[]FetchRequest{FetchRequest{Name:aaa.ddd.dev.gateway.bbb.ccc.Main.count,StartTime:1764795955,StopTime:1764796857,HighPrecisionTimestamps:false,PathExpression:aaa.*.*.*.bbb.ccc.*.count,FilterFunctions:[]*FilteringFunction{},MaxDataPoints:0,},},}", "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "function": "HttpQuery.doRequest", "server": "http://go-carbon:8080", "name": "http://go-carbon:8080", "uri": "http://go-carbon:8080/render/?format=protobuf&from=1764795955&target=aaa.ddd.dev.gateway.bbb.ccc.Main.count&until=1764796857", "name": "http://go-carbon:8080"}
[2025-12-03T22:06:15.705Z] DEBUG [zipper] got response {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Mobile.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "multi_fetch": true, "backend_name": "http://go-carbon:8080", "metrics_in_response": 1, "errors_count": 0, "timeouts_count": 0, "render_requests_count": 1, "render_errors_count": 0, "render_timeouts_count": 0, "zipper_requests_count": 0, "total_metric_count": 0, "servers_count": 0, "failed_servers_count": 0}
[2025-12-03T22:06:15.705Z] DEBUG [zipper] got some fetch responses {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Mobile.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "backends_count": 1, "response_count": 1, "have_errors": false, "errors": null, "metrics_in_response": 1}
[2025-12-03T22:06:15.705Z] DEBUG [zipper] got response {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "multi_fetch": true, "backend_name": "backends", "backend_name": "backends", "metrics_in_response": 1, "errors_count": 0, "timeouts_count": 0, "render_requests_count": 1, "render_errors_count": 0, "render_timeouts_count": 0, "zipper_requests_count": 0, "total_metric_count": 0, "servers_count": 0, "failed_servers_count": 0}
[2025-12-03T22:06:15.705Z] DEBUG [zipper] got some fetch responses {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "backends_count": 1, "response_count": 1, "have_errors": false, "errors": null, "metrics_in_response": 1}
[2025-12-03T22:06:15.705Z] DEBUG [zipper] errors while fetching data from backends {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Main.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "httpCode": 503, "errors": ["timeout while fetching Response"]}
[2025-12-03T22:06:15.705Z] DEBUG [zipper] got response {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["aaa.*.*.*.bbb.ccc.*.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "multi_fetch": true, "backend_name": "backends", "backend_name": "backends", "response_is_nil": true, "stats_is_nil": false, "err": "timeout while fetching Response", "errVerbose": "timeout while fetching Response\nHTTP Code: 503\n\ngithub.com/go-graphite/carbonapi/zipper/types.init\n\t/root/go/src/github.com/go-graphite/carbonapi/zipper/types/errors.go:27\nruntime.doInit1\n\t/root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:7350\nruntime.doInit\n\t/root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:7317\nruntime.main\n\t/root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:254\nruntime.goexit\n\t/root/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1700"}
[2025-12-03T22:06:15.705Z] DEBUG [zipper] error fetching result {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[]FetchRequest{FetchRequest{Name:aaa.ddd.dev.gateway.bbb.ccc.Main.count,StartTime:1764795955,StopTime:1764796857,HighPrecisionTimestamps:false,PathExpression:aaa.*.*.*.bbb.ccc.*.count,FilterFunctions:[]*FilteringFunction{},MaxDataPoints:0,},},}", "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "function": "HttpQuery.doRequest", "server": "http://go-carbon:8080", "name": "http://go-carbon:8080", "uri": "http://go-carbon:8080/render/?format=protobuf&from=1764795955&target=aaa.ddd.dev.gateway.bbb.ccc.Main.count&until=1764796857", "error": "Get \"http://go-carbon:8080/render/?format=protobuf&from=1764795955&target=aaa.ddd.dev.gateway.bbb.ccc.Main.count&until=1764796857\": context canceled"}
[2025-12-03T22:06:15.705Z] DEBUG [zipper] have errors {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[]FetchRequest{FetchRequest{Name:aaa.ddd.dev.gateway.bbb.ccc.Main.count,StartTime:1764795955,StopTime:1764796857,HighPrecisionTimestamps:false,PathExpression:aaa.*.*.*.bbb.ccc.*.count,FilterFunctions:[]*FilteringFunction{},MaxDataPoints:0,},},}", "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "error": "error while fetching Response", "server": "http://go-carbon:8080"}
[2025-12-03T22:06:15.705Z] INFO [access] request served {"data": {"handler":"render","carbonapi_uuid":"0750efeb-0dbc-4c12-a5e8-dc4cf0726f29","url":"/render?target=groupByNode(aaa.*.*.*.bbb.ccc.*.count%2C%206%2C%20%22sum%22)\u0026from=1764795955\u0026until=1764796857\u0026format=json\u0026maxDataPoints=1144","peer_ip":"172.18.0.1","peer_port":"44318","host":"172.18.0.20:8085","format":"json","use_cache":true,"targets":["groupByNode(aaa.*.*.*.bbb.ccc.*.count, 6, \"sum\")"],"cache_timeout":10,"metrics":["groupByNode(aaa.*.*.*.bbb.ccc.*.count, 6, \"sum\")"],"runtime":0.00162052,"http_code":200,"carbonzipper_response_size_bytes":810,"carbonapi_response_size_bytes":1591,"from":1764795955,"until":1764796857,"max_data_points":1144,"from_raw":"1764795955","until_raw":"1764796857","uri":"/render?target=groupByNode(aaa.*.*.*.bbb.ccc.*.count%2C%206%2C%20%22sum%22)\u0026from=1764795955\u0026until=1764796857\u0026format=json\u0026maxDataPoints=1144","from_cache":false,"used_backend_cache":false,"request_headers":{}}}
[2025-12-03T22:06:15.705Z] WARN [zipper] errors occurred while getting results {"type": "protoV2Group", "name": "http://go-carbon:8080", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[]FetchRequest{FetchRequest{Name:aaa.ddd.dev.gateway.bbb.ccc.Main.count,StartTime:1764795955,StopTime:1764796857,HighPrecisionTimestamps:false,PathExpression:aaa.*.*.*.bbb.ccc.*.count,FilterFunctions:[]*FilteringFunction{},MaxDataPoints:0,},},}", "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "errors": "max tries exceeded", "errorsVerbose": "max tries exceeded\nHTTP Code: 500\n\ngithub.com/go-graphite/carbonapi/zipper/types.init\n\t/root/go/src/github.com/go-graphite/carbonapi/zipper/types/errors.go:25\nruntime.doInit1\n\t/root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:7350\nruntime.doInit\n\t/root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:7317\nruntime.main\n\t/root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:254\nruntime.goexit\n\t/root/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1700\n\nCaused By: failed to fetch data from server/group\nHTTP Code: 500\n\ngithub.com/go-graphite/carbonapi/zipper/types.init\n\t/root/go/src/github.com/go-graphite/carbonapi/zipper/types/errors.go:27\nruntime.doInit1\n\t/root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:7350\nruntime.doInit\n\t/root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:7317\nruntime.main\n\t/root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:254\nruntime.goexit\n\t/root/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1700\n\nCaused By: error while fetching Response\n\ngithub.com/go-graphite/carbonapi/zipper/types.init\n\t/root/go/src/github.com/go-graphite/carbonapi/zipper/types/errors.go:34\nruntime.doInit1\n\t/root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:7350\nruntime.doInit\n\t/root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:7317\nruntime.main\n\t/root/go/pkg/mod/golang.org/[email protected]/src/runtime/proc.go:254\nruntime.goexit\n\t/root/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1700"}
[2025-12-03T22:06:15.705Z] DEBUG [zipper] got response {"type": "broadcastGroup", "groupName": "backends", "type": "fetch", "request": ["aaa.ddd.dev.gateway.bbb.ccc.Main.count"], "carbonapi_uuid": "0750efeb-0dbc-4c12-a5e8-dc4cf0726f29", "multi_fetch": true, "backend_name": "http://go-carbon:8080", "metrics_in_response": 0, "errors_count": 1, "timeouts_count": 0, "render_requests_count": 1, "render_errors_count": 1, "render_timeouts_count": 0, "zipper_requests_count": 0, "total_metric_count": 0, "servers_count": 0, "failed_servers_count": 0}


$ devvm down carbonapi; devvm build carbonapi; devvm up carbonapi; docker inspect docker-per-service-carbonapi-1 | grep IPAddress  # carbonapi=0.18.2? Different target so nondeterministic!
$ curl 'http://172.18.0.20:8085/render?target=groupByNode(aaa.*.*.*.bbb.ccc.*.count%2C%206%2C%20%22sum%22)&from=1764795955&until=1764796857&format=json&maxDataPoints=1144'
[{"target":"Main","datapoints":[[null,1764795960],[null,1764795970],[null,1764795980],[null,1764795990],[null,1764796000],[null,1764796010],[null,1764796020],[null,1764796030],[null,1764796040],[null,1764796050],[null,1764796060],[null,1764796070],[null,1764796080],[null,1764796090],[null,1764796100],[null,1764796110],[null,1764796120],[null,1764796130],[null,1764796140],[null,1764796150],[null,1764796160],[null,1764796170],[null,1764796180],[null,1764796190],[null,1764796200],[null,1764796210],[null,1764796220],[null,1764796230],[null,1764796240],[null,1764796250],[null,1764796260],[null,1764796270],[null,1764796280],[null,1764796290],[null,1764796300],[null,1764796310],[null,1764796320],[null,1764796330],[null,1764796340],[null,1764796350],[null,1764796360],[null,1764796370],[null,1764796380],[null,1764796390],[null,1764796400],[null,1764796410],[null,1764796420],[null,1764796430],[null,1764796440],[null,1764796450],[null,1764796460],[null,1764796470],[1,1764796480],[0,1764796490],[0,1764796500],[3,1764796510],[1,1764796520],[2,1764796530],[null,1764796540],[1,1764796550],[1,1764796560],[2,1764796570],[0,1764796580],[0,1764796590],[1,1764796600],[0,1764796610],[1,1764796620],[2,1764796630],[1,1764796640],[5,1764796650],[1,1764796660],[3,1764796670],[0,1764796680],[1,1764796690],[3,1764796700],[2,1764796710],[2,1764796720],[0,1764796730],[1,1764796740],[0,1764796750],[0,1764796760],[2,1764796770],[0,1764796780],[1,1764796790],[1,1764796800],[0,1764796810],[1,1764796820],[2,1764796830],[1,1764796840],[2,1764796850]],"tags":{"aggregatedBy":"sum","name":"Main"}}]

CarbonAPI Configuration:
Please include carbonapi config file that you are using (please replace all information you don't want to share, like domain names or IP addresses).

# Listen address, should always include hostname or ip address and a port.
listen: "0.0.0.0:8085"
# Max concurrent requests to CarbonZipper
concurency: 20
cache:
   # Type of caching. Valid: "mem", "memcache", "null"
   type: "mem"
   # Cache limit in megabytes
   size_mb: 0
   # Default cache timeout value. Identical to DEFAULT_CACHE_DURATION in graphite-web.
   defaultTimeoutSec: 10
# Amount of CPUs to use. 0 - unlimited
cpus: 0
# Timezone, default - local
tz: ""
# If 'true', carbonapi will send requests as is, with globs and braces
# Otherwise for each request it will generate /metrics/find and then /render
# individual metrics.
# true --- faster, but will cause carbonzipper to consume much more RAM.
#
# For some backends (e.x. graphite-clickhouse) this *MUST* be set to true in order
# to get reasonable performance
#
# For go-carbon --- it depends on how you use it.
sendGlobsAsIs: false
# If sendGlobsAsIs is set and resulting response will be larger than maxBatchSize
# carbonapi will revert to old behavir. This allows you to use benifits of passing
# globs as is but keep memory usage in sane limits.
#
# For go-carbon you might want it to keep in some reasonable limits, 100 is good "safe" defaults
#
# For some backends (e.x. graphite-clickhouse) you might want to set it to some insanly high value, like 100000
maxBatchSize: 100
graphite:
    # Host:port where to send internal metrics
    # Empty = disabled
    host: "carbon-c-relay:2023"
    interval: "60s"
    prefix: "carbon.api"
# Maximium idle connections to carbonzipper
idleConnections: 10
pidFile: ""
upstreams:
    # Number of 100ms buckets to track request distribution in. Used to build
    # 'carbon.zipper.hostname.requests_in_0ms_to_100ms' metric and friends.
    # Requests beyond the last bucket are logged as slow
    # (default of 10 implies "slow" is >1 second).
    buckets: 10
    timeouts:
        # Maximum total backend requesting timeout in ms.
        # ( How long we may spend making requests. )
        global: "15s"
        # Timeout, in ms, once the final backend has been contacted.
        # ( [Effectively] How long we'll wait for the slowest response. )
        afterStarted: "25000ms"
        # Timeout to connect to the server
        connect: "9900ms"
    # Number of concurrent requests to any given backend - default is no limit.
    # If set, you likely want >= MaxIdleConnsPerHost
    concurrencyLimit: 0
    # Configures how often keep alive packets will be sent out
    keepAliveInterval: "30s"
    # Control http.MaxIdleConnsPerHost. Large values can lead to more idle
    # connections on the backend servers which may bump into limits; tune with care.
    maxIdleConnsPerHost: 100
    # "http://host:port" array of instances of carbonserver stores
    # This is the *ONLY* config element in this section that MUST be specified.
    # loclhost:8080 - go-carbon
    backends:
      - "http://go-carbon:8080"
    carbonsearch:
        # Instance of carbonsearch backend
        backend: "http://127.0.0.1:8070"
        # carbonsearch prefix to reserve/register
        prefix: "virt.v1.*"
    # Enable compatibility with graphite-web 0.9
    # This will affect graphite-web 1.0+ with multiple cluster_servers
    # Default: disabled
    graphite09compat: false
# If not zero, enabled cache for find requests
# This parameter controls when it will expire (in seconds)
# Default: 600 (10 minutes)
expireDelaySec: 0
# See https://github.com/go-graphite/carbonzipper/blob/master/example.conf#L70-L108 for format explanation
logger:
    - logger: ""
      file: "/var/log/carbonapi/carbonapi.log"
      level: "debug"
      encoding: "mixed"
      encodingTime: "iso8601"
      encodingDuration: "seconds"

Simplified query (if applicable)

See above: curl 'http://172.18.0.20:8085/render?target=groupByNode(aaa.*.*.*.bbb.ccc.*.count%2C%206%2C%20%22sum%22)&from=1764795955&until=1764796857&format=json&maxDataPoints=1144'

Testing with Grafana 12.3.0 and carbonapi=0.15.3~1 datasource graphite version 1.1.x, i found my labels are good, but some dashboard graph panel queries are missing targets. A new panel does contain multiple targets, though. Claude claims multiple data series are working correctly for queries with multiple separate target parameters, so the bug is specific to groupByNode() producing multiple series within a single target, not multiple targets.

Working pattern (separate targets):

target=aliasSub(alias(...), ...)
&target=aliasSub(alias(...), ...)
&target=aliasSub(alias(...), ...)

→ Each target returns one series, carbonapi 0.15.3+ handles this fine ✓

Broken pattern (groupByNode creates multiple series):

target=aliasSub(groupByNode(aaa.*.*.*.bbb.ccc.*.count, 6, 'sum'), ...)

→ groupByNode should create multiple series (Main, Mobile, etc.), but carbonapi 0.15.3+ only returns one ✗

The fix: Stay on carbonapi 0.10.0.1 because truncated labels are less bad than missing data.

Backend metric retention and aggregation schemas
Please provide backend's schema (most important thing - if query cross retention period or not), aggregation function, xFilesFactor (if applicable).

I have no idea where to look for that.

Backend response (if possible)

See above.

If that's possible - please share some sample set of backend responses. Most important thing here:

  1. Are all backend responses the same? If not, what's the difference? 0.10.0.1 returns both Main and Mobile targets. 0.15.3+ returns either Main or Mobile target, and the carbonapi log shows zipper is getting errors while fetching data from backends [...] "httpCode": 503, "errors": ["timeout while fetching Response"]} which is strange because 0.10.0.1 gets both targets, and 0.18.2 still gets one, well within 1s while the carbonapi config timeouts are "10s" or more.
  2. Do they contain special values, like Inf/NaN values? No. Unless null counts, but even the broken version returns that.
  3. Do all of them have same step? I don't know what that means.

Additional context
Upgrading Grafana/carbonapi isn't in our sprint anymore.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bug

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions