Enabling retention is incompatible with setting up a proxy through env vars in SingleBinary and SimpleScalable modes

[rurod]

Describe the bug
Loki forwards ":9095" requests to HTTP Proxy when both ruler and retention are enabled.

To Reproduce
Steps to reproduce the behavior:

1. Deploy Loki 3.4.2 using HelmChart 6.29.0 in a Simple Scalable Architecture
2. Deploy a basic squid proxy
3. Configure Loki to access S3 storage through the squid proxy using env vars (HTTP_PROXY, HTTPS_PROXY, NO_PROXY)
4. Configure the compactor to enable retention and set delete_request_store value to "s3".
5. Set up AlertingRules using the sidecar and local storage.

Issue does not occur if no alerting rule is configured.
Issue disappears if retention is disabled

Error message :

ts=2025-04-10T08:45:46.888537767Z caller=spanlogger.go:111 user=fake caller=log.go:168 level=error msg="failed loading deletes for user" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: failed to do connect handshake, response: \\\"HTTP/1.1 400 Bad Request........"

Expected behavior

1. More precise log on which Loki component is trying to forward :9095 requests through the proxy.
2. No component trying to reach GRPC endpoint through the HTTP Proxy

Environment:

• Infrastructure: Kubernetes/OpenShift
• Deployment tool: Helm
• Storage : s3 accessible through an HTTP Proxy for the chunk storage, local storage for the ruler

Screenshots, Promtail config, or terminal output

Backend pod logs :

level=info ts=2025-04-10T08:44:40.494573597Z caller=main.go:126 msg="Starting Loki" version="(version=3.4.2, branch=release-3.4.x, revision=4fa045d3)"
level=info ts=2025-04-10T08:44:40.494606468Z caller=main.go:127 msg="Loading configuration file" filename=/etc/loki/config/config.yaml
level=info ts=2025-04-10T08:44:40.495374054Z caller=server.go:351 msg="server listening on addresses" http=[::]:3100 grpc=[::]:9095
level=info ts=2025-04-10T08:44:40.497105263Z caller=memberlist_client.go:439 msg="Using memberlist cluster label and node name" cluster_label= node=loki-backend-1-7bffd8a7
level=info ts=2025-04-10T08:44:40.497385437Z caller=memberlist_client.go:549 msg="memberlist fast-join starting" nodes_found=1 to_join=4
level=info ts=2025-04-10T08:44:40.501024005Z caller=table_manager.go:456 index-store=tsdb-2024-04-01 msg="loading local table loki_index_20188"
level=info ts=2025-04-10T08:44:40.509754357Z caller=memberlist_client.go:569 msg="memberlist fast-join finished" joined_nodes=3 elapsed_time=12.370763ms
level=info ts=2025-04-10T08:44:40.509803089Z caller=memberlist_client.go:581 phase=startup msg="joining memberlist cluster" join_members=loki-memberlist
level=info ts=2025-04-10T08:44:40.516948647Z caller=memberlist_client.go:588 phase=startup msg="joining memberlist cluster succeeded" reached_nodes=3 elapsed_time=7.137975ms
level=info ts=2025-04-10T08:44:40.541731251Z caller=table_manager.go:300 index-store=tsdb-2024-04-01 msg="query readiness setup completed" duration=3.267µs distinct_users_len=0 distinct_users=
level=info ts=2025-04-10T08:44:40.54176804Z caller=shipper.go:165 index-store=tsdb-2024-04-01 msg="starting index shipper in RO mode"
level=info ts=2025-04-10T08:44:40.542982804Z caller=mapper.go:47 msg="cleaning up mapped rules directory" path=/var/loki/rules-temp
level=info ts=2025-04-10T08:44:40.547399787Z caller=module_service.go:82 msg=starting module=analytics
level=info ts=2025-04-10T08:44:40.547476931Z caller=module_service.go:82 msg=starting module=server
level=info ts=2025-04-10T08:44:40.547550038Z caller=module_service.go:82 msg=starting module=runtime-config
level=info ts=2025-04-10T08:44:40.547746015Z caller=module_service.go:82 msg=starting module=memberlist-kv
level=info ts=2025-04-10T08:44:40.547997565Z caller=module_service.go:82 msg=starting module=ring
level=info ts=2025-04-10T08:44:40.547997865Z caller=module_service.go:82 msg=starting module=index-gateway-ring
level=info ts=2025-04-10T08:44:40.548022271Z caller=module_service.go:82 msg=starting module=query-scheduler-ring
level=info ts=2025-04-10T08:44:40.548041277Z caller=module_service.go:82 msg=starting module=compactor
level=info ts=2025-04-10T08:44:40.548158726Z caller=basic_lifecycler.go:299 msg="instance not found in the ring" instance=loki-backend-1 ring=compactor
level=info ts=2025-04-10T08:44:40.548171059Z caller=basic_lifecycler_delegates.go:63 msg="not loading tokens from file, tokens file path is empty"
level=info ts=2025-04-10T08:44:40.548291986Z caller=basic_lifecycler.go:299 msg="instance not found in the ring" instance=loki-backend-1 ring=scheduler
level=info ts=2025-04-10T08:44:40.548301835Z caller=basic_lifecycler_delegates.go:63 msg="not loading tokens from file, tokens file path is empty"
level=info ts=2025-04-10T08:44:40.54837986Z caller=module_service.go:82 msg=starting module=ingester-querier
level=info ts=2025-04-10T08:44:40.548474978Z caller=ringmanager.go:186 msg="waiting until scheduler is JOINING in the ring"
level=info ts=2025-04-10T08:44:40.548487342Z caller=ringmanager.go:190 msg="scheduler is JOINING in the ring"
level=info ts=2025-04-10T08:44:40.548485388Z caller=basic_lifecycler.go:299 msg="instance not found in the ring" instance=loki-backend-1 ring=index-gateway
level=info ts=2025-04-10T08:44:40.548493052Z caller=basic_lifecycler_delegates.go:63 msg="not loading tokens from file, tokens file path is empty"
level=info ts=2025-04-10T08:44:40.548804495Z caller=ringmanager.go:186 msg="waiting until index-gateway is JOINING in the ring"
level=info ts=2025-04-10T08:44:40.54881326Z caller=ringmanager.go:190 msg="index-gateway is JOINING in the ring"
level=info ts=2025-04-10T08:44:40.548909762Z caller=compactor.go:414 msg="waiting until compactor is JOINING in the ring"
level=info ts=2025-04-10T08:44:40.548920392Z caller=compactor.go:418 msg="compactor is JOINING in the ring"
level=info ts=2025-04-10T08:44:41.548841865Z caller=ringmanager.go:199 msg="waiting until scheduler is ACTIVE in the ring"
level=info ts=2025-04-10T08:44:41.548915482Z caller=ringmanager.go:199 msg="waiting until index-gateway is ACTIVE in the ring"
level=info ts=2025-04-10T08:44:41.549886369Z caller=compactor.go:428 msg="waiting until compactor is ACTIVE in the ring"
level=info ts=2025-04-10T08:44:41.707565323Z caller=ringmanager.go:203 msg="index-gateway is ACTIVE in the ring"
level=info ts=2025-04-10T08:44:41.707650502Z caller=module_service.go:82 msg=starting module=store
level=info ts=2025-04-10T08:44:41.707678775Z caller=module_service.go:82 msg=starting module=rule-evaluator
level=info ts=2025-04-10T08:44:41.707690437Z caller=module_service.go:82 msg=starting module=index-gateway
level=info ts=2025-04-10T08:44:41.707713841Z caller=module_service.go:82 msg=starting module=ruler
level=info ts=2025-04-10T08:44:41.707754848Z caller=ruler.go:533 msg="ruler up and running"
level=info ts=2025-04-10T08:44:41.709825851Z caller=manager.go:164 user=fake msg="Starting rule manager..."
level=info ts=2025-04-10T08:44:41.723187899Z caller=ringmanager.go:203 msg="scheduler is ACTIVE in the ring"
level=info ts=2025-04-10T08:44:41.723283129Z caller=module_service.go:82 msg=starting module=query-scheduler
level=info ts=2025-04-10T08:44:41.749980074Z caller=compactor.go:432 msg="compactor is ACTIVE in the ring"
level=info ts=2025-04-10T08:44:41.750065494Z caller=loki.go:545 msg="Loki started" startup_time=1.268247982s
level=info ts=2025-04-10T08:44:44.723841976Z caller=scheduler.go:653 msg="this scheduler is in the ReplicationSet, will now accept requests."
level=info ts=2025-04-10T08:44:46.75083674Z caller=compactor.go:493 msg="this instance has been chosen to run the compactor, starting compactor"
level=info ts=2025-04-10T08:44:46.750919173Z caller=compactor.go:522 msg="waiting 10m0s for ring to stay stable and previous compactions to finish before starting compactor"
level=info ts=2025-04-10T08:45:41.709803903Z caller=mapper.go:163 msg="updating rule file" file=/var/loki/rules-temp/fake/namespace_loki.configmap_ruler-rules.rules.yaml
level=info ts=2025-04-10T08:45:41.750795311Z caller=compactor.go:502 msg="this instance should no longer run the compactor, stopping compactor"
level=info ts=2025-04-10T08:45:41.750847469Z caller=compactor.go:507 msg="compactor stopped"
level=error ts=2025-04-10T08:45:41.751026523Z caller=cached_client.go:189 msg="failed to build table names cache" err="RequestCanceled: request context canceled\ncaused by: context canceled"
level=error ts=2025-04-10T08:45:41.751096325Z caller=cached_client.go:189 msg="failed to build table names cache" err="RequestCanceled: request context canceled\ncaused by: context canceled"
level=error ts=2025-04-10T08:45:41.751146318Z caller=cached_client.go:189 msg="failed to build table names cache" err="RequestCanceled: request context canceled\ncaused by: context canceled"
level=error ts=2025-04-10T08:45:41.751172117Z caller=compactor.go:534 msg="failed to run compaction" err="failed to list tables: RequestCanceled: request context canceled\ncaused by: context canceled"
level=info ts=2025-04-10T08:45:41.751184068Z caller=compactor.go:592 msg="compactor started"
level=info ts=2025-04-10T08:45:41.751190691Z caller=marker.go:177 msg="mark processor started" workers=150 delay=2h0m0s
level=info ts=2025-04-10T08:45:41.751231146Z caller=expiration.go:78 msg="overall smallest retention period 1744274741.751, default smallest retention period 1744274741.751"
level=error ts=2025-04-10T08:45:41.751330453Z caller=cached_client.go:189 msg="failed to build table names cache" err="RequestCanceled: request context canceled\ncaused by: context canceled"
level=error ts=2025-04-10T08:45:41.751400383Z caller=cached_client.go:189 msg="failed to build table names cache" err="RequestCanceled: request context canceled\ncaused by: context canceled"
level=error ts=2025-04-10T08:45:41.751494109Z caller=cached_client.go:189 msg="failed to build table names cache" err="RequestCanceled: request context canceled\ncaused by: context canceled"
level=error ts=2025-04-10T08:45:41.751508195Z caller=compactor.go:561 msg="failed to apply retention" err="failed to list tables: RequestCanceled: request context canceled\ncaused by: context canceled"
level=info msg="request timings" insight=true source=loki_ruler rule_name=http-credentials-leaked rule_type=alerting total=0.003209054 total_bytes=0 query_hash=3305400941
level=info ts=2025-04-10T08:45:46.887492993Z caller=compat.go:67 user=fake rule_name=http-credentials-leaked rule_type=alerting query="sum by (cluster,job,pod)((count_over_time({namespace=\"prod\"} |~ \"http(s?)://(\\\\w+):(\\\\w+)@\"[5m]) > 0))" query_hash=3305400941 msg="evaluating rule"
level=info ts=2025-04-10T08:45:46.88828808Z caller=engine.go:263 component=ruler evaluation_mode=local org_id=fake traceID=64c74ba86adf0c67 msg="executing query" query="sum by (cluster,job,pod)((count_over_time({namespace=\"prod\"} |~ \"http(s?)://(\\\\w+):(\\\\w+)@\"[5m]) > 0))" query_hash=3305400941 type=instant
ts=2025-04-10T08:45:46.888537767Z caller=spanlogger.go:111 user=fake caller=log.go:168 level=error msg="failed loading deletes for user" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: failed to do connect handshake, response: \\\"HTTP/1.1 400 Bad Request\\\\r\\\\nConnection: close\\\\r\\\\nContent-Length: 3516\\\\r\\\\nContent-Language: en\\\\r\\\\nContent-Type: text/html;charset=utf-8\\\\r\\\\nDate: Thu, 10 Apr 2025 08:45:44 GMT\\\\r\\\\nMime-Version: 1.0\\\\r\\\\nServer: squid/5.5\\\\r\\\\nVary: Accept-Language\\\\r\\\\nVia: 1.1 squid-68d7ff79b-kxc5g (squid/5.5)\\\\r\\\\nX-Cache: MISS from squid-68d7ff79b-kxc5g\\\\r\\\\nX-Cache-Lookup: NONE from squid-68d7ff79b-kxc5g:3128\\\\r\\\\nX-Squid-Error: ERR_INVALID_URL 0\\\\r\\\\n\\\\r\\\\n<!DOCTYPE html PUBLIC \\\\\\\"-//W3C//DTD HTML 4.01//EN\\\\\\\" \\\\\\\"http://www.w3.org/TR/html4/strict.dtd\\\\\\\">\\\\n<html><head>\\\\n<meta type=\\\\\\\"copyright\\\\\\\" content=\\\\\\\"Copyright (C) 1996-2022 The Squid Software Foundation and contributors\\\\\\\">\\\\n<meta http-equiv=\\\\\\\"Content-Type\\\\\\\" content=\\\\\\\"text/html; charset=utf-8\\\\\\\">\\\\n<title>ERROR: The requested URL could not be retrieved</title>\\\\n<style type=\\\\\\\"text/css\\\\\\\"><!-- \\\\n /*\\\\n * Copyright (C) 1996-2022 The Squid Software Foundation and contributors\\\\n *\\\\n * Squid software is distributed under GPLv2+ license and includes\\\\n * contributions from numerous individuals and organizations.\\\\n * Please see the COPYING and CONTRIBUTORS files for details.\\\\n */\\\\n\\\\n/*\\\\n Stylesheet for Squid Error pages\\\\n Adapted from design by Free CSS Templates\\\\n http://www.freecsstemplates.org\\\\n Released for free under a Creative Commons Attribution 2.5 License\\\\n*/\\\\n\\\\n/* Page basics */\\\\n* {\\\\n\\\\tfont-family: verdana, sans-serif;\\\\n}\\\\n\\\\nhtml body {\\\\n\\\\tmargin: 0;\\\\n\\\\tpadding: 0;\\\\n\\\\tbackground: #efefef;\\\\n\\\\tfont-size: 12px;\\\\n\\\\tcolor: #1e1e1e;\\\\n}\\\\n\\\\n/* Page displayed title area */\\\\n#titles {\\\\n\\\\tmargin-left: 15px;\\\\n\\\\tpadding: 10px;\\\\n\\\\tpadding-left: 100px;\\\\n\\\\tbackground: url('/squid-internal-static/icons/SN.png') no-repeat left;\\\\n}\\\\n\\\\n/* initial title */\\\\n#titles h1 {\\\\n\\\\tcolor: #000000;\\\\n}\\\\n#titles h2 {\\\\n\\\\tcolor: #000000;\\\\n}\\\\n\\\\n/* special event: FTP success page titles */\\\\n#titles ftpsuccess {\\\\n\\\\tbackground-color:#00ff00;\\\\n\\\\twidth:100%;\\\\n}\\\\n\\\\n/* Page displayed body content area */\\\\n#content {\\\\n\\\\tpadding: 10px;\\\\n\\\\tbackground: #ffffff;\\\\n}\\\\n\\\\n/* General text */\\\\np {\\\\n}\\\\n\\\\n/* error brief description */\\\\n#error p {\\\\n}\\\\n\\\\n/* some data which may have caused the problem */\\\\n#data {\\\\n}\\\\n\\\\n/* the error message received from the system or other software */\\\\n#sysmsg {\\\\n}\\\\n\\\\npre {\\\\n}\\\\n\\\\n/* special event: FTP directory listing */\\\\n#dirmsg {\\\\n    font-family: courier, monospace;\\\\n    color: black;\\\\n    font-size: 10pt;\\\\n}\\\\n#dirlisting {\\\\n    margin-left: 2%;\\\\n    margin-right: 2%;\\\\n}\\\\n#dirlisting tr.entry td.icon,td.filename,td.size,td.date {\\\\n    border-bottom: groove;\\\\n}\\\\n#dirlisting td.size {\\\\n    width: 50px;\\\\n    text-align: right;\\\\n    padding-right: 5px;\\\\n}\\\\n\\\\n/* horizontal lines */\\\\nhr {\\\\n\\\\tmargin: 0;\\\\n}\\\\n\\\\n/* page displayed footer area */\\\\n#footer {\\\\n\\\\tfont-size: 9px;\\\\n\\\\tpadding-left: 10px;\\\\n}\\\\n\\\\n\\\\nbody\\\\n:lang(fa) { direction: rtl; font-size: 100%; font-family: Tahoma, Roya, sans-serif; float: right; }\\\\n:lang(he) { direction: rtl; }\\\\n --></style>\\\\n</head><body id=ERR_INVALID_URL>\\\\n<div id=\\\\\\\"titles\\\\\\\">\\\\n<h1>ERROR</h1>\\\\n<h2>The requested URL could not be retrieved</h2>\\\\n</div>\\\\n<hr>\\\\n\\\\n<div id=\\\\\\\"content\\\\\\\">\\\\n<p>The following error was encountered while trying to retrieve the URL: <a href=\\\\\\\":9095\\\\\\\">:9095</a></p>\\\\n\\\\n<blockquote id=\\\\\\\"error\\\\\\\">\\\\n<p><b>Invalid URL</b></p>\\\\n</blockquote>\\\\n\\\\n<p>Some aspect of the requested URL is incorrect.</p>\\\\n\\\\n<p>Some possible problems are:</p>\\\\n<ul>\\\\n<li><p>Missing or incorrect access protocol (should be <q>http://</q> or similar)</p></li>\\\\n<li><p>Missing hostname</p></li>\\\\n<li><p>Illegal double-escape in the URL-Path</p></li>\\\\n<li><p>Illegal character in hostname; underscores are not allowed.</p></li>\\\\n</ul>\\\\n\\\\n<p>Your cache administrator is <a href=\\\\\\\"mailto:root?subject=CacheErrorInfo%20-%20ERR_INVALID_URL&amp;body=CacheHost%3A%20squid-68d7ff79b-kxc5g%0D%0AErrPage%3A%20ERR_INVALID_URL%0D%0AErr%3A%20%5Bnone%5D%0D%0ATimeStamp%3A%20Thu,%2010%20Apr%202025%2008%3A45%3A44%20GMT%0D%0A%0D%0AClientIP%3A%2010.135.0.64%0D%0A%0D%0AHTTP%20Request%3A%0D%0A%0D%0A%0D%0A\\\\\\\">root</a>.</p>\\\\n<br>\\\\n</div>\\\\n\\\\n<hr>\\\\n<div id=\\\\\\\"footer\\\\\\\">\\\\n<p>Generated Thu, 10 Apr 2025 08:45:44 GMT by squid-68d7ff79b-kxc5g (squid/5.5)</p>\\\\n<!-- ERR_INVALID_URL -->\\\\n</div>\\\\n</body></html>\\\\n\\\"\""
level=info ts=2025-04-10T08:45:46.890315754Z caller=table_manager.go:195 index-store=tsdb-2024-04-01 msg="get or create table" found=true table=loki_index_20188 wait_for_lock=2.134µs
level=info ts=2025-04-10T08:45:46.891573046Z caller=metrics.go:237 component=ruler evaluation_mode=local org_id=fake traceID=64c74ba86adf0c67 latency=fast query="sum by (cluster,job,pod)((count_over_time({namespace=\"prod\"} |~ \"http(s?)://(\\\\w+):(\\\\w+)@\"[5m]) > 0))" query_hash=3305400941 query_type=metric range_type=instant length=0s start_delta=4.424701ms end_delta=4.424862ms step=0s duration=3.209054ms status=200 limit=0 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=0s splits=0 shards=0 query_referenced_structured_metadata=false pipeline_wrapper_filtered_lines=0 chunk_refs_fetch_time=174.446µs cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_volume_results_req=0 cache_volume_results_hit=0 cache_volume_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s cache_result_query_length_served=0s cardinality_estimate=0 ingester_chunk_refs=0 ingester_chunk_downloaded=0 ingester_chunk_matches=0 ingester_requests=2 ingester_chunk_head_bytes=0B ingester_chunk_compressed_bytes=0B ingester_chunk_decompressed_bytes=0B ingester_post_filter_lines=0 congestion_control_latency=0s index_total_chunks=0 index_post_bloom_filter_chunks=0 index_bloom_filter_ratio=0.00 index_used_bloom_filters=false index_shard_resolver_duration=0s disable_pipeline_wrappers=false has_labelfilter_before_parser=false
level=info msg="request timings" insight=true source=loki_ruler rule_name=HighPercentageError rule_type=alerting total=0.002248156 total_bytes=0 query_hash=3932776858
level=info ts=2025-04-10T08:46:11.053166265Z caller=compat.go:67 user=fake rule_name=HighPercentageError rule_type=alerting query="((sum by (job)(rate({app=\"foo\", env=\"production\"} |= \"error\"[5m])) / sum by (job)(rate({app=\"foo\", env=\"production\"}[5m]))) > 0.05)" query_hash=3932776858 msg="evaluating rule"
level=info ts=2025-04-10T08:46:11.053967944Z caller=engine.go:263 component=ruler evaluation_mode=local org_id=fake traceID=458db7fba79c4ad0 msg="executing query" query="((sum by (job)(rate({app=\"foo\", env=\"production\"} |= \"error\"[5m])) / sum by (job)(rate({app=\"foo\", env=\"production\"}[5m]))) > 0.05)" query_hash=3932776858 type=instant
ts=2025-04-10T08:46:11.054218654Z caller=spanlogger.go:111 user=fake caller=log.go:168 level=error msg="failed loading deletes for user" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: failed to do connect handshake, response: \\\"HTTP/1.1 400 Bad Request\\\\r\\\\nConnection: close\\\\r\\\\nContent-Length: 3516\\\\r\\\\nContent-Language: en\\\\r\\\\nContent-Type: text/html;charset=utf-8\\\\r\\\\nDate: Thu, 10 Apr 2025 08:46:07 GMT\\\\r\\\\nMime-Version: 1.0\\\\r\\\\nServer: squid/5.5\\\\r\\\\nVary: Accept-Language\\\\r\\\\nVia: 1.1 squid-68d7ff79b-kxc5g (squid/5.5)\\\\r\\\\nX-Cache: MISS from squid-68d7ff79b-kxc5g\\\\r\\\\nX-Cache-Lookup: NONE from squid-68d7ff79b-kxc5g:3128\\\\r\\\\nX-Squid-Error: ERR_INVALID_URL 0\\\\r\\\\n\\\\r\\\\n<!DOCTYPE html PUBLIC \\\\\\\"-//W3C//DTD HTML 4.01//EN\\\\\\\" \\\\\\\"http://www.w3.org/TR/html4/strict.dtd\\\\\\\">\\\\n<html><head>\\\\n<meta type=\\\\\\\"copyright\\\\\\\" content=\\\\\\\"Copyright (C) 1996-2022 The Squid Software Foundation and contributors\\\\\\\">\\\\n<meta http-equiv=\\\\\\\"Content-Type\\\\\\\" content=\\\\\\\"text/html; charset=utf-8\\\\\\\">\\\\n<title>ERROR: The requested URL could not be retrieved</title>\\\\n<style type=\\\\\\\"text/css\\\\\\\"><!-- \\\\n /*\\\\n * Copyright (C) 1996-2022 The Squid Software Foundation and contributors\\\\n *\\\\n * Squid software is distributed under GPLv2+ license and includes\\\\n * contributions from numerous individuals and organizations.\\\\n * Please see the COPYING and CONTRIBUTORS files for details.\\\\n */\\\\n\\\\n/*\\\\n Stylesheet for Squid Error pages\\\\n Adapted from design by Free CSS Templates\\\\n http://www.freecsstemplates.org\\\\n Released for free under a Creative Commons Attribution 2.5 License\\\\n*/\\\\n\\\\n/* Page basics */\\\\n* {\\\\n\\\\tfont-family: verdana, sans-serif;\\\\n}\\\\n\\\\nhtml body {\\\\n\\\\tmargin: 0;\\\\n\\\\tpadding: 0;\\\\n\\\\tbackground: #efefef;\\\\n\\\\tfont-size: 12px;\\\\n\\\\tcolor: #1e1e1e;\\\\n}\\\\n\\\\n/* Page displayed title area */\\\\n#titles {\\\\n\\\\tmargin-left: 15px;\\\\n\\\\tpadding: 10px;\\\\n\\\\tpadding-left: 100px;\\\\n\\\\tbackground: url('/squid-internal-static/icons/SN.png') no-repeat left;\\\\n}\\\\n\\\\n/* initial title */\\\\n#titles h1 {\\\\n\\\\tcolor: #000000;\\\\n}\\\\n#titles h2 {\\\\n\\\\tcolor: #000000;\\\\n}\\\\n\\\\n/* special event: FTP success page titles */\\\\n#titles ftpsuccess {\\\\n\\\\tbackground-color:#00ff00;\\\\n\\\\twidth:100%;\\\\n}\\\\n\\\\n/* Page displayed body content area */\\\\n#content {\\\\n\\\\tpadding: 10px;\\\\n\\\\tbackground: #ffffff;\\\\n}\\\\n\\\\n/* General text */\\\\np {\\\\n}\\\\n\\\\n/* error brief description */\\\\n#error p {\\\\n}\\\\n\\\\n/* some data which may have caused the problem */\\\\n#data {\\\\n}\\\\n\\\\n/* the error message received from the system or other software */\\\\n#sysmsg {\\\\n}\\\\n\\\\npre {\\\\n}\\\\n\\\\n/* special event: FTP directory listing */\\\\n#dirmsg {\\\\n    font-family: courier, monospace;\\\\n    color: black;\\\\n    font-size: 10pt;\\\\n}\\\\n#dirlisting {\\\\n    margin-left: 2%;\\\\n    margin-right: 2%;\\\\n}\\\\n#dirlisting tr.entry td.icon,td.filename,td.size,td.date {\\\\n    border-bottom: groove;\\\\n}\\\\n#dirlisting td.size {\\\\n    width: 50px;\\\\n    text-align: right;\\\\n    padding-right: 5px;\\\\n}\\\\n\\\\n/* horizontal lines */\\\\nhr {\\\\n\\\\tmargin: 0;\\\\n}\\\\n\\\\n/* page displayed footer area */\\\\n#footer {\\\\n\\\\tfont-size: 9px;\\\\n\\\\tpadding-left: 10px;\\\\n}\\\\n\\\\n\\\\nbody\\\\n:lang(fa) { direction: rtl; font-size: 100%; font-family: Tahoma, Roya, sans-serif; float: right; }\\\\n:lang(he) { direction: rtl; }\\\\n --></style>\\\\n</head><body id=ERR_INVALID_URL>\\\\n<div id=\\\\\\\"titles\\\\\\\">\\\\n<h1>ERROR</h1>\\\\n<h2>The requested URL could not be retrieved</h2>\\\\n</div>\\\\n<hr>\\\\n\\\\n<div id=\\\\\\\"content\\\\\\\">\\\\n<p>The following error was encountered while trying to retrieve the URL: <a href=\\\\\\\":9095\\\\\\\">:9095</a></p>\\\\n\\\\n<blockquote id=\\\\\\\"error\\\\\\\">\\\\n<p><b>Invalid URL</b></p>\\\\n</blockquote>\\\\n\\\\n<p>Some aspect of the requested URL is incorrect.</p>\\\\n\\\\n<p>Some possible problems are:</p>\\\\n<ul>\\\\n<li><p>Missing or incorrect access protocol (should be <q>http://</q> or similar)</p></li>\\\\n<li><p>Missing hostname</p></li>\\\\n<li><p>Illegal double-escape in the URL-Path</p></li>\\\\n<li><p>Illegal character in hostname; underscores are not allowed.</p></li>\\\\n</ul>\\\\n\\\\n<p>Your cache administrator is <a href=\\\\\\\"mailto:root?subject=CacheErrorInfo%20-%20ERR_INVALID_URL&amp;body=CacheHost%3A%20squid-68d7ff79b-kxc5g%0D%0AErrPage%3A%20ERR_INVALID_URL%0D%0AErr%3A%20%5Bnone%5D%0D%0ATimeStamp%3A%20Thu,%2010%20Apr%202025%2008%3A46%3A07%20GMT%0D%0A%0D%0AClientIP%3A%2010.135.0.64%0D%0A%0D%0AHTTP%20Request%3A%0D%0A%0D%0A%0D%0A\\\\\\\">root</a>.</p>\\\\n<br>\\\\n</div>\\\\n\\\\n<hr>\\\\n<div id=\\\\\\\"footer\\\\\\\">\\\\n<p>Generated Thu, 10 Apr 2025 08:46:07 GMT by squid-68d7ff79b-kxc5g (squid/5.5)</p>\\\\n<!-- ERR_INVALID_URL -->\\\\n</div>\\\\n</body></html>\\\\n\\\"\""

Squid proxy access logs :

1744275670.596  90052 10.135.0.64 TCP_TUNNEL/200 8058 CONNECT s3-REDACTED:443 - HIER_DIRECT/10.10.10.101 -
1744275670.636  90125 10.135.0.64 TCP_TUNNEL/200 6719 CONNECT s3-REDACTED:443 - HIER_DIRECT/10.10.10.101 -
1744275670.839      0 10.132.2.57 NONE_NONE/400 3918 CONNECT :9095 - HIER_NONE/- text/html
1744275671.434      0 10.135.0.64 NONE_NONE/400 3918 CONNECT :9095 - HIER_NONE/- text/html
1744275672.386      0 10.135.0.64 NONE_NONE/400 3918 CONNECT :9095 - HIER_NONE/- text/html
1744275673.297      0 10.132.2.57 NONE_NONE/400 3918 CONNECT :9095 - HIER_NONE/- text/html

SimpleScalableValues.yaml

---
loki:
  schemaConfig:
    configs:
      - from: 2024-04-01
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h
  ingester:
    chunk_encoding: snappy
  tracing:
    enabled: true
  querier:
    max_concurrent: 4

  auth_enabled: false
  storage:
    bucketNames:
      chunks: loki-chunks-9868e731-2f16-49ce-bcb7-38d74f139f58
    type: s3
    s3:
      endpoint: https://REDACTED
      secretAccessKey: REDACTED
      accessKeyId: REDACTED
      insecure: false
      s3ForcePathStyle: true
      http_config:
        insecure_skip_verify: true
  compactor:
    working_directory: /var/loki/data/retention
    compaction_interval: 10m
    retention_enabled: true
    retention_delete_delay: 2h
    retention_delete_worker_count: 150
    delete_request_store: s3
  rulerConfig:
    wal:
      dir: /var/loki/ruler-wal
    storage:
      type: local
      local:
        directory: /rules

deploymentMode: SimpleScalable

backend:
  replicas: 2
  extraEnv:
    - name: HTTP_PROXY
      value: http://172.31.228.11:3128
    - name: HTTPS_PROXY
      value: http://172.31.228.11:3128
    - name: NO_PROXY
      value: "172.31.0.0/16,10.132.0.0/14,.svc,.cluster.local,10.10.10.0/24,127.0.0.1,localhost"
read:
  replicas: 2
write:
  replicas: 2

minio:
  enabled: false

# Zero out replica counts of other deployment modes
singleBinary:
  replicas: 0
ingester:
  replicas: 0
querier:
  replicas: 0
queryFrontend:
  replicas: 0
queryScheduler:
  replicas: 0
distributor:
  replicas: 0
compactor:
  replicas: 0
indexGateway:
  replicas: 0
bloomCompactor:
  replicas: 0
bloomGateway:
  replicas: 0

serviceAccount:
  # -- Specifies whether a ServiceAccount should be created
  create: true
  automountServiceAccountToken: true
# RBAC configuration
rbac:
  pspEnabled: false
  sccEnabled: true
  pspAnnotations: {}
  namespaced: false

ingress:
  enabled: true
  paths:
    distributor:
      - /api/prom/push
      - /loki/api/v1/push
      - /otlp/v1/logs
    queryFrontend:
      - /api/prom/query
      - /api/prom/label
      - /api/prom/series
      - /api/prom/tail
      - /loki/api/v1/query
      - /loki/api/v1/query_range
      - /loki/api/v1/tail
      - /loki/api/v1/label
      - /loki/api/v1/labels
      - /loki/api/v1/series
      - /loki/api/v1/index/stats
      - /loki/api/v1/index/volume
      - /loki/api/v1/index/volume_range
      - /loki/api/v1/format_query
      - /loki/api/v1/detected_field
      - /loki/api/v1/detected_fields
      - /loki/api/v1/detected_labels
      - /loki/api/v1/patterns
    ruler:
      - /api/prom/rules
      - /api/prom/api/v1/rules
      - /api/prom/api/v1/alerts
      - /loki/api/v1/rules
      - /prometheus/api/v1/rules
      - /prometheus/api/v1/alerts
  hosts:
    - loki.example.com

sidecar:
  enableUniqueFilenames: true
  rules:
    enabled: true
    folder: /rules/fake
memcachedExporter:
  enabled: false
resultsCache:
  # -- Specifies whether memcached based results-cache should be enabled
  enabled: false
chunksCache:
  # -- Specifies whether memcached based results-cache should be enabled
  enabled: false
test:
  enabled: false
lokiCanary: 
  enabled: false

Ruler rules configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: ruler-rules
  namespace: loki
  labels:
    loki_rule: ""
data:
  rules.yaml: |
    groups:
    - name: should_fire
      rules:
        - alert: HighPercentageError
          expr: |
            sum(rate({app="foo", env="production"} |= "error" [5m])) by (job)
              /
            sum(rate({app="foo", env="production"}[5m])) by (job)
              > 0.05
          for: 10m
          labels:
            severity: page
          annotations:
            summary: High request latency
    - name: credentials_leak
      rules:
        - alert: http-credentials-leaked
          annotations:
            message: "{{ $labels.job }} is leaking http basic auth credentials."
          expr: 'sum by (cluster, job, pod) (count_over_time({namespace="prod"} |~ "http(s?)://(\\w+):(\\w+)@" [5m]) > 0)'
          for: 10m
          labels:
            severity: critical 

[rurod]

Hi,

So digging a little bit deeper, it seems that ":9095" requests are still being forwarded to the proxy even with ruler disabled. Only thing is that the backend pod is not complaining about it unless some rules are loaded in the ruler...
Those requests disappear the moment the retention is turned off.

Looking in the code base related to the compactor, I found this :
https://github.com/grafana/loki/blob/main/pkg/loki/modules.go#L1258

func (t *Loki) compactorAddress() (string, bool, error) {
	legacyReadMode := t.Cfg.LegacyReadTarget && t.Cfg.isTarget(Read)
	if t.Cfg.isTarget(All) || legacyReadMode || t.Cfg.isTarget(Backend) {
		// In single binary or read modes, this module depends on Server
		return net.JoinHostPort(t.Cfg.Server.GRPCListenAddress, strconv.Itoa(t.Cfg.Server.GRPCListenPort)), true, nil
	}

	if t.Cfg.Common.CompactorAddress == "" && t.Cfg.Common.CompactorGRPCAddress == "" {
		return "", false, errors.New("query filtering for deletes requires 'compactor_grpc_address' or 'compactor_address' to be configured")
	}

	if t.Cfg.Common.CompactorGRPCAddress != "" {
		return t.Cfg.Common.CompactorGRPCAddress, true, nil
	}

	return t.Cfg.Common.CompactorAddress, false, nil
}

In my case, the backend pod enters the first condition, as the target is "Backend" :

if t.Cfg.isTarget(All) || legacyReadMode || t.Cfg.isTarget(Backend) {
	// In single binary or read modes, this module depends on Server
	return net.JoinHostPort(t.Cfg.Server.GRPCListenAddress, strconv.Itoa(t.Cfg.Server.GRPCListenPort)), true, nil
}

The address returned is ":9095", without a hostname, because the grpc_listen_address is set to "" in the configuration.
In distributed architecture mode it works because the compactorAddress is computed from the compactor_grpc_address parameters from the config.

But in SingleBinary and SimpleScalableSetup the GRPC client try to connect to ":9095", falling into the first condition.
And this seems to fail because all requests made to ":9095" seems to ignore the NO_PROXY directive.

Not sure if it's expected.
IMO, when the compactor_grpc_address or compactor_address is defined, then it should be prioritized over the first condition of the compactorAddress() function.

[rurod]

Renaming the Issue according to the latest findings.