Skip to content

Commit 630edad

Browse files
authored
fix(e2e/discovery): Let a crash-looping fixture fail instead of restarting forever (#54350)
### What does this PR do? Stops the fixture services restarting forever when they are broken, so a fixture that cannot start reports itself as a failed unit. The units set `StartLimitIntervalSec=0`, which disables systemd's start rate limit entirely. Combined with `Restart=always` and `RestartSec=1`, a service that fails immediately restarts forever: it stays in `activating (auto-restart)` and never reaches `failed`. This PR enables the limit (`StartLimitIntervalSec=60`, `StartLimitBurst=5`) so a service that keeps failing gives up and the unit ends in `failed`. ### Motivation DSCVR-621. There are some cases were discovery is failing but in some cases we can't clearly distinguish discovery really failing from the server just not starting up so make the service startup fail explicitly. ### Describe how you validated your changes In a container running systemd 249 (the same version as the test image), with a service that exits after 450ms to match the observed rails failure: | `[Unit]` config | State after 25s | `NRestarts` | |---|---|---| | `StartLimitIntervalSec=0` (before) | `activating (auto-restart)` | **16, still climbing** | | `StartLimitIntervalSec=60` + `StartLimitBurst=5` (after) | **`failed`** (`Result: exit-code`) | 5 | ### Additional Notes One deliberate trade-off: a fixture that currently succeeds only after more than five automatic restarts would now fail instead. That is right outcome — a fixture needing six attempts is broken and should say so. Co-authored-by: vincent.whitchurch <vincent.whitchurch@datadoghq.com>
1 parent b13ab21 commit 630edad

1 file changed

Lines changed: 10 additions & 1 deletion

File tree

test/new-e2e/tests/discovery/testdata/provision/provision.sh

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,11 +33,20 @@ install_systemd_unit() {
3333
done <<< "${extraenv} "
3434
fi
3535

36+
# The start limit is left enabled on purpose. These services are
37+
# Restart=always with RestartSec=1, so one which fails immediately restarts
38+
# forever: it stays in "activating (auto-restart)" and never reaches "failed",
39+
# which makes a broken fixture look like a service the Agent failed to
40+
# discover rather than a service that is not running. Letting the limit stop
41+
# it turns that into a failed unit instead. The counter is reset by an
42+
# explicit systemctl stop, so the start/stop cycles the tests do between
43+
# subtests never accumulate towards it.
3644
cat > "/etc/systemd/system/${name}.service" <<- EOM
3745
[Unit]
3846
Description=${name}
3947
After=network.target
40-
StartLimitIntervalSec=0
48+
StartLimitIntervalSec=60
49+
StartLimitBurst=5
4150
4251
[Service]
4352
Type=simple

0 commit comments

Comments
 (0)