BB-830: fix notification config write race that silently drops events

[tcarmet]

Bucket notification events could be silently and permanently lost for a bucket whenever the one-time write of its notification configuration to Zookeeper lost a race and failed, because the delivery path treats a missing configuration as "nothing to deliver" and drops every event for that bucket without a trace. This change makes that configuration write resilient so a transient failure no longer strands a bucket's notifications, and surfaces the previously silent drop in the logs. The race was found through intermittent failures of the bucket-notification end-to-end suite in CI (BB-830).

[bert-e]

Hello tcarmet,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
/after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval	Bypass the pull request author's approval	⭐
/bypass_build_status	Bypass the build and test status	⭐
/bypass_commit_size	Bypass the check on the size of the changeset TBA	⭐
/bypass_incompatible_branch	Bypass the check on the source branch prefix	⭐
/bypass_jira_check	Bypass the Jira issue check	⭐
/bypass_peer_approval	Bypass the pull request peers' approval	⭐
/bypass_leader_approval	Bypass the pull request leaders' approval	⭐
/approve	Instruct Bert-E that the author has approved the pull request.		✍️
/create_pull_requests	Allow the creation of integration pull requests.
/create_integration_branches	Allow the creation of integration branches.
/no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity	Change review acceptance criteria from one reviewer at least to all reviewers
/wait	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
/help	Print Bert-E's manual in the pull request.
/status	Print Bert-E's current status in the pull request TBA
/clear	Remove all comments from Bert-E from the history TBA
/retry	Re-start a fresh build TBA
/build	Re-start a fresh build TBA
/force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
/reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

[bert-e]

Incorrect fix version

The Fix Version/s in issue BB-830 contains:

• None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

• 9.3.6

• 9.4.2

• 9.5.0

Please check the Fix Version/s of BB-830, or the target
branch of this pull request.

[codecov]

Codecov Report

❌ Patch coverage is 87.50000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.47%. Comparing base (595f1b4) to head (ccb6f72).

Files with missing lines	Patch %	Lines
...tification/configManager/ZookeeperConfigManager.js	87.50%	2 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
...tification/configManager/ZookeeperConfigManager.js	83.58% <87.50%> (+0.07%)	⬆️

... and 4 files with indirect coverage changes

Components	Coverage Δ
Bucket Notification	80.27% <87.50%> (+0.04%)	⬆️
Core Library	80.58% <ø> (-0.17%)	⬇️
Ingestion	70.63% <ø> (ø)
Lifecycle	78.96% <ø> (ø)
Oplog Populator	85.83% <ø> (ø)
Replication	59.70% <ø> (-0.04%)	⬇️
Bucket Scanner	85.76% <ø> (ø)

@@                 Coverage Diff                 @@
##           development/9.3    #2786      +/-   ##
===================================================
- Coverage            74.54%   74.47%   -0.07%     
===================================================
  Files                  199      199              
  Lines                13622    13629       +7     
===================================================
- Hits                 10154    10150       -4     
- Misses                3458     3469      +11     
  Partials                10       10              

Flag	Coverage Δ
api:retry	9.15% <0.00%> (-0.01%)	⬇️
api:routes	8.97% <0.00%> (-0.01%)	⬇️
bucket-scanner	85.76% <ø> (ø)
ft_test:queuepopulator	9.07% <12.50%> (-1.00%)	⬇️
ingestion	12.56% <12.50%> (+<0.01%)	⬆️
lib	7.68% <0.00%> (-0.02%)	⬇️
lifecycle	18.86% <0.00%> (+0.08%)	⬆️
notification	1.04% <12.50%> (+0.01%)	⬆️
oplogPopulator	0.14% <0.00%> (-0.01%)	⬇️
replication	18.51% <0.00%> (-0.01%)	⬇️
unit	51.14% <87.50%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[bert-e]

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.