Skip to content

Conversation

@dcoppa
Copy link

@dcoppa dcoppa commented Dec 8, 2025

What this PR does / why we need it:

When trying to use a k8s-sidecar version ≥ 2.x, the loki-sc-rules container crashes with the following error message:

{"time": "2025-12-08T14:07:21.551765+00:00", "level": "INFO", "msg": "Starting collector"}
{"time": "2025-12-08T14:07:21.644201+00:00", "level": "INFO", "msg": "No folder annotation was provided, defaulting to k8s-sidecar-target-directory"}
{"time": "2025-12-08T14:07:21.644497+00:00", "level": "INFO", "msg": "Loading incluster config..."}
{"time": "2025-12-08T14:07:21.645369+00:00", "level": "INFO", "msg": "Unique filenames will not be enforced."}
{"time": "2025-12-08T14:07:21.647169+00:00", "level": "INFO", "msg": "5xx response content will not be enabled."}
{"time": "2025-12-08T14:07:21.647701+00:00", "level": "INFO", "msg": "Performing initial list-based sync before starting watch."}
{"time": "2025-12-08T14:07:21.647922+00:00", "level": "INFO", "msg": "Loading incluster config..."}
{"time": "2025-12-08T14:07:21.649807+00:00", "level": "INFO", "msg": "Performing list-based sync on secret resources: {'namespace': 'loki'}"}
{"time": "2025-12-08T14:07:21.861818+00:00", "level": "INFO", "msg": "Loading incluster config..."}
{"time": "2025-12-08T14:07:21.862525+00:00", "level": "INFO", "msg": "Performing list-based sync on configmap resources: {'namespace': 'loki'}"}
{"time": "2025-12-08T14:07:22.150797+00:00", "level": "INFO", "msg": "Started server process [1]", "color_message": "Started server process [\u001b[36m%d\u001b[0m]"}
{"time": "2025-12-08T14:07:22.151111+00:00", "level": "INFO", "msg": "Waiting for application startup."}
{"time": "2025-12-08T14:07:22.243831+00:00", "level": "INFO", "msg": "Application startup complete."}
{"time": "2025-12-08T14:07:22.243506+00:00", "level": "INFO", "msg": "Writing /rules/fake/grafana.yaml (ascii)"}
{"time": "2025-12-08T14:07:22.245523+00:00", "level": "INFO", "msg": "Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)", "color_message": "Uvicorn running on \u001b[1m%s://%s:%d\u001b[0m (Press CTRL+C to quit)"}
{"time": "2025-12-08T14:07:22.246109+00:00", "level": "INFO", "msg": "Initial sync complete, sidecar is ready."}
{"time": "2025-12-08T14:07:22.347746+00:00", "level": "ERROR", "msg": "FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/app']", "exc_info": "Traceback (most recent call last):\n  File \"/app/sidecar.py\", line 158, in <module>\n    main()\n    ~~~~^^\n  File \"/app/sidecar.py\", line 152, in main\n    watch_for_changes(method, label, label_value, target_folder, request_url, request_method, request_payload,\n    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n                      namespace, folder_annotation, resources, unique_filenames, script, enable_5xx,\n                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n                      ignore_already_processed, resource_name)\n                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/app/resources.py\", line 442, in watch_for_changes\n    processes = _start_watcher_processes(current_namespace, folder_annotation, label,\n                                         label_value, request_method, mode, request_payload, resources,\n                                         target_folder, unique_filenames, script, request_url, enable_5xx,\n                                         ignore_already_processed, resource_name)\n  File \"/app/resources.py\", line 481, in _start_watcher_processes\n    proc.start()\n    ~~~~~~~~~~^^\n  File \"/usr/local/lib/python3.14/multiprocessing/process.py\", line 121, in start\n    self._popen = self._Popen(self)\n                  ~~~~~~~~~~~^^^^^^\n  File \"/usr/local/lib/python3.14/multiprocessing/context.py\", line 224, in _Popen\n    return _default_context.get_context().Process._Popen(process_obj)\n           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.14/multiprocessing/context.py\", line 300, in _Popen\n    return Popen(process_obj)\n  File \"/usr/local/lib/python3.14/multiprocessing/popen_forkserver.py\", line 35, in __init__\n    super().__init__(process_obj)\n    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.14/multiprocessing/popen_fork.py\", line 20, in __init__\n    self._launch(process_obj)\n    ~~~~~~~~~~~~^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.14/multiprocessing/popen_forkserver.py\", line 51, in _launch\n    self.sentinel, w = forkserver.connect_to_new_process(self._fds)\n                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^\n  File \"/usr/local/lib/python3.14/multiprocessing/forkserver.py\", line 89, in connect_to_new_process\n    self.ensure_running()\n    ~~~~~~~~~~~~~~~~~~~^^\n  File \"/usr/local/lib/python3.14/multiprocessing/forkserver.py\", line 156, in ensure_running\n    address = connection.arbitrary_address('AF_UNIX')\n  File \"/usr/local/lib/python3.14/multiprocessing/connection.py\", line 79, in arbitrary_address\n    return tempfile.mktemp(prefix='sock-', dir=util.get_temp_dir())\n                                               ~~~~~~~~~~~~~~~~~^^\n  File \"/usr/local/lib/python3.14/multiprocessing/util.py\", line 212, in get_temp_dir\n    base_tempdir = _get_base_temp_dir(tempfile)\n  File \"/usr/local/lib/python3.14/multiprocessing/util.py\", line 169, in _get_base_temp_dir\n    base_tempdir = tempfile.gettempdir()\n  File \"/usr/local/lib/python3.14/tempfile.py\", line 316, in gettempdir\n    return _os.fsdecode(_gettempdir())\n                        ~~~~~~~~~~~^^\n  File \"/usr/local/lib/python3.14/tempfile.py\", line 309, in _gettempdir\n    tempdir = _get_default_tempdir()\n  File \"/usr/local/lib/python3.14/tempfile.py\", line 224, in _get_default_tempdir\n    raise FileNotFoundError(_errno.ENOENT,\n                            \"No usable temporary directory found in %s\" %\n                            dirlist)\nFileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/app']"}

My PR fixes the helm chart: the same "tmp" emptyDir volume used in the main loki container is also mounted in the loki-sc-rules container.

Special notes for your reviewer:

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • [] Documentation added
  • [] Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • [] Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • [] If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@dcoppa dcoppa requested a review from a team as a code owner December 8, 2025 15:33
@CLAassistant
Copy link

CLAassistant commented Dec 8, 2025

CLA assistant check
All committers have signed the CLA.

@pull-request-size pull-request-size bot added size/S and removed size/XS labels Dec 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants