improvements to initial member handling by mwrock · Pull Request #10176 · habitat-sh/habitat

mwrock · 2026-01-26T17:00:53Z

No description provided.

netlify · 2026-01-26T17:01:02Z

👷 Deploy Preview for chef-habitat processing.

Name	Link
🔨 Latest commit	`5a04c24`
🔍 Latest deploy log	https://app.netlify.com/projects/chef-habitat/deploys/6994ce50fd4ec60008b4af34

Copilot

Pull request overview

This PR modifies the Habitat Supervisor's peer discovery mechanism to continuously ping initial members from the peer watch file, rather than stopping after reaching the minimum threshold. The change enables dynamic peer discovery when new initial members are added to the watch file during runtime.

Changes:

Removed the early return check that prevented peer seeding when members existed
Added a signaling mechanism to notify the outbound thread when new initial members are added
Modified the outbound loop to re-evaluate and ping initial members when new ones are detected

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
components/sup/src/manager.rs	Removed early return check and added signal call when initial members are updated from watch file
components/butterfly/src/server/outbound.rs	Added logic to detect new initial members signal, skip pinging already-alive members, and respect min_to_start threshold
components/butterfly/src/server.rs	Added new_initial_members atomic flag and signal_new_initial_members() method; removed unused need_peer_seeding_mlr() function

Signed-off-by: Matt Wrock <matt@mattwrock.com>

… review Signed-off-by: Matt Wrock <matt@mattwrock.com>

Signed-off-by: Matt Wrock <matt@mattwrock.com>

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Copilot · 2026-02-10T22:54:19Z

+    /// Signal that new initial members have been added and should be pinged on the next outbound
+    /// cycle
+    pub fn signal_new_initial_members(&self) {
+        self.new_initial_members.store(true, Ordering::Release);


Missing import for Ordering. The atomic::Ordering type is used here but not imported in the visible portion of this file. Ensure use std::sync::atomic::Ordering; is added to the imports at the top of the file.

Copilot · 2026-02-10T22:54:20Z

@@ -16,9 +16,6 @@ services:
      - run
      - --listen-ctl=0.0.0.0:9632
      - --peer-watch-file=/hab/PEERS


The volume mount for the PEERS file has been removed, but the supervisor is still configured to watch /hab/PEERS via --peer-watch-file. Without the volume mount, the file won't exist or be accessible for dynamic updates during the test. This will cause the peer watcher test to fail.

Suggested change

- --peer-watch-file=/hab/PEERS

- --peer-watch-file=/hab/PEERS

volumes:

- ./testcases/peer_watcher/PEERS:/hab/PEERS

Copilot · 2026-02-10T22:54:20Z

+        let new_initial_members_added = server.new_initial_members.load(Ordering::Acquire);
+        if new_initial_members_added {
+            // Reset the flag and force re-evaluation of initial member pinging
+            server.new_initial_members.store(false, Ordering::Relaxed);


The flag is loaded with Ordering::Acquire but stored with Ordering::Relaxed. For proper synchronization, this should use Ordering::Release to ensure the store is visible to other threads that may be loading it.

Suggested change

server.new_initial_members.store(false, Ordering::Relaxed);

server.new_initial_members.store(false, Ordering::Release);

Signed-off-by: Matt Wrock <matt@mattwrock.com>

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Copilot · 2026-02-11T01:07:42Z

      - run
      - --listen-ctl=0.0.0.0:9632
      - --peer-watch-file=/hab/PEERS
-    volumes:
-      - ./testcases/peer_watcher/PEERS:/hab/PEERS
-

  tester:


The peer watch file path is configured on the bastion supervisor (--peer-watch-file=/hab/PEERS), but the bind mount for ./testcases/peer_watcher/PEERS is now only on the tester service. This means edits made by the test may not be visible inside the bastion container, so the supervisor won’t detect changes. Fix by mounting the same host file into bastion as well (or use a shared named volume mounted into both containers at /hab/PEERS).

Copilot · 2026-02-11T01:07:43Z

    volumes:
+      - source: ./testcases/peer_watcher/PEERS
+        target: /hab/PEERS
+        type: bind


The peer watch file path is configured on the bastion supervisor (--peer-watch-file=/hab/PEERS), but the bind mount for ./testcases/peer_watcher/PEERS is now only on the tester service. This means edits made by the test may not be visible inside the bastion container, so the supervisor won’t detect changes. Fix by mounting the same host file into bastion as well (or use a shared named volume mounted into both containers at /hab/PEERS).

Copilot · 2026-02-11T01:07:43Z

+        let new_initial_members_added = server.new_initial_members.load(Ordering::Acquire);
+        if new_initial_members_added {
+            // Reset the flag and force re-evaluation of initial member pinging
+            server.new_initial_members.store(false, Ordering::Relaxed);


Resetting new_initial_members with a separate load() then store(false, ...) can lose signals: if another thread sets the flag to true between the load and the store, this store will overwrite it back to false, and the outbound loop may miss the update. Use an atomic read-modify-write (e.g., swap(false, Ordering::AcqRel) or compare_exchange loop) to clear the flag without dropping concurrent updates, and use a non-Relaxed ordering consistent with the producer’s Release store.

Suggested change

let new_initial_members_added = server.new_initial_members.load(Ordering::Acquire);

if new_initial_members_added {

// Reset the flag and force re-evaluation of initial member pinging

server.new_initial_members.store(false, Ordering::Relaxed);

let new_initial_members_added =

server.new_initial_members.swap(false, Ordering::AcqRel);

if new_initial_members_added {

// Reset the flag and force re-evaluation of initial member pinging

Copilot · 2026-02-11T01:07:43Z

+
+    It "adds beta to the peer watch file and finds it as a peer" {
+        Add-Content -Path "/hab/PEERS" -Value "beta.habitat.dev"
+        Start-Sleep -Seconds 5 # give butterfly some time to detect the change and update the census


The fixed Start-Sleep makes the e2e test timing-dependent and more prone to flakes under load/CI variance. Prefer polling with the existing Wait-True helper until the census reflects the expected last_membership_counter (with a reasonable timeout and error message), instead of sleeping a fixed duration.

Suggested change

Start-Sleep -Seconds 5 # give butterfly some time to detect the change and update the census

$testScript = { (Invoke-WebRequest "http://bastion.habitat.dev:9631/census" | ConvertFrom-Json).last_membership_counter -eq 3 }

$timeoutScript = { Write-Error "Timed out waiting 45 seconds for beta to be detected and the census to reach last_membership_counter 3" }

Wait-True -TestScript $testScript -TimeoutScript $timeoutScript -Timeout 45

Signed-off-by: Matt Wrock <matt@mattwrock.com>

Copilot AI review requested due to automatic review settings January 26, 2026 17:00

mwrock changed the title ~~Always initial~~ improvements to initial member handling Jan 26, 2026

Copilot AI reviewed Jan 26, 2026

View reviewed changes

Comment thread components/butterfly/src/server.rs Outdated

Comment thread components/butterfly/src/server/outbound.rs

Comment thread components/butterfly/src/server/outbound.rs

chef-expeditor Bot assigned mwrock Jan 26, 2026

mwrock added 4 commits February 10, 2026 11:32

always ping any new initial members on startup

bd9bab8

Signed-off-by: Matt Wrock <matt@mattwrock.com>

always add peer watch file entries to initial members

93d1731

Signed-off-by: Matt Wrock <matt@mattwrock.com>

ping initial members when peer watch file is updated

13c145a

Signed-off-by: Matt Wrock <matt@mattwrock.com>

add e2e test for adding to the peer watch file and respond to copilot…

4d21841

… review Signed-off-by: Matt Wrock <matt@mattwrock.com>

mwrock force-pushed the always_initial branch from 2b8eb7d to 4d21841 Compare February 10, 2026 22:30

fix powershell syntax in test

ebabca7

Signed-off-by: Matt Wrock <matt@mattwrock.com>

Copilot AI review requested due to automatic review settings February 10, 2026 22:53

Copilot AI reviewed Feb 10, 2026

View reviewed changes

mwrock added 2 commits February 10, 2026 15:15

try and edit the peer file from the tester

c97dfdb

Signed-off-by: Matt Wrock <matt@mattwrock.com>

try and edit the peer file from the tester

55292c8

Signed-off-by: Matt Wrock <matt@mattwrock.com>

Copilot AI review requested due to automatic review settings February 11, 2026 01:06

Copilot AI reviewed Feb 11, 2026

View reviewed changes

fix e2e test

5a04c24

Signed-off-by: Matt Wrock <matt@mattwrock.com>

Navigation Menu

Uh oh!

improvements to initial member handling - #10176

improvements to initial member handling#10176
mwrock wants to merge 8 commits into
mainfrom
always_initial

mwrock commented Jan 26, 2026

Uh oh!

netlify Bot commented Jan 26, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 10, 2026

Uh oh!

Copilot AI Feb 10, 2026

Uh oh!

Copilot AI Feb 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 11, 2026

Uh oh!

Copilot AI Feb 11, 2026

Uh oh!

Copilot AI Feb 11, 2026

Uh oh!

Copilot AI Feb 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	server.new_initial_members.store(false, Ordering::Relaxed);
	server.new_initial_members.store(false, Ordering::Release);

-        Start-Sleep -Seconds 5 # give butterfly some time to detect the change and update the census
+        $testScript = { (Invoke-WebRequest "http://bastion.habitat.dev:9631/census" | ConvertFrom-Json).last_membership_counter -eq 3 }
+        $timeoutScript = { Write-Error "Timed out waiting 45 seconds for beta to be detected and the census to reach last_membership_counter 3" }
+        Wait-True -TestScript $testScript -TimeoutScript $timeoutScript -Timeout 45

Uh oh!

Conversation

mwrock commented Jan 26, 2026

Uh oh!

netlify Bot commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

👷 Deploy Preview for chef-habitat processing.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

netlify Bot commented Jan 26, 2026 •

edited

Loading