Background
setup-common.sh installs remote_syslog2 with a /etc/log_files.yml that tails /var/log/nginx/*.err:
https://github.com/compiler-explorer/infra/blob/main/setup-common.sh#L80-L107 (around the remote_syslog/log_files.yml/remote-syslog.service block)
setup-common.sh is sourced by seven node types:
setup-node.sh (web -- does run nginx, so valid here)setup-aarch64-node.shsetup-ci.shsetup-admin.shsetup-builder.shsetup-ce-router.shsetup-smb.sh
Only the web node runs nginx. On the other six, remote_syslog2 runs and tails /var/log/nginx/*.err, a file that never exists -- so the daemon does nothing useful there. (rsyslog/grafana-agent cover real logging.)
Why separate from #2158
The conan box has its own hand-copied duplicate of this block and was cleaned up in #2158 (conan does not source setup-common.sh, so it's self-contained). Editing setup-common.sh re-bakes all seven AMIs above, which is a much wider blast radius and deserves its own change + rebake plan.
Proposed work
- Decide per node type whether any log-file tailing is actually wanted (only the web
node plausibly needs nginx tailing).
- Either gate the
remote_syslog2 block behind a per-node flag, or remove it from setup-common.sh and add it back only where genuinely needed.
- Also bump the pinned
remote_syslog2 v0.21 (≈2020) if the daemon is kept anywhere.
- Re-bake affected AMIs.
Found while debugging the 2026-06-04 conan outage (#2158).
Background
setup-common.shinstallsremote_syslog2with a/etc/log_files.ymlthat tails/var/log/nginx/*.err:https://github.com/compiler-explorer/infra/blob/main/setup-common.sh#L80-L107 (around the
remote_syslog/log_files.yml/remote-syslog.serviceblock)setup-common.shis sourced by seven node types:setup-node.sh(web -- does run nginx, so valid here)setup-aarch64-node.shsetup-ci.shsetup-admin.shsetup-builder.shsetup-ce-router.shsetup-smb.shOnly the web
noderuns nginx. On the other six,remote_syslog2runs and tails/var/log/nginx/*.err, a file that never exists -- so the daemon does nothing useful there. (rsyslog/grafana-agent cover real logging.)Why separate from #2158
The conan box has its own hand-copied duplicate of this block and was cleaned up in #2158 (conan does not source
setup-common.sh, so it's self-contained). Editingsetup-common.shre-bakes all seven AMIs above, which is a much wider blast radius and deserves its own change + rebake plan.Proposed work
nodeplausibly needs nginx tailing).remote_syslog2block behind a per-node flag, or remove it fromsetup-common.shand add it back only where genuinely needed.remote_syslog2 v0.21(≈2020) if the daemon is kept anywhere.Found while debugging the 2026-06-04 conan outage (#2158).