Geyser manager read lock perf

[RadiantAeon]

Check #7857 (comment)

Every single geyser update has to obtain a readlock on the geyser manager to check the list of loaded plugins, which is very slow.

[RadiantAeon]

I've been digging into this to find a potential fix and a blocker is the fact that we don't actually directly pass in a geyser AccountsUpdateNotifier and instead pass in Arc<dyn AccountsUpdateNotifierInterface + Sync + Send>. Otherwise it would be easiest to create a method on the plugin account update notifier to return a read lock of the geyser plugin manager during index generation. during This trait has only been used for Geyser plugins since it was created 4 years ago. Can we just axe this trait and directly pass in the geyser account update notifier struct.

This would allow us to do something like grab a geyser plugin manager readlock somewhere near the top of generate_index_for_slot around

agave/accounts-db/src/accounts_db.rs

Line 6321 in 944d47a

let geyser_notifier = self

and then make notify_account_restore_from_snapshot take in a ref to plugin_manager

agave/accounts-db/src/accounts_db.rs

Lines 6390 to 6406 in 944d47a

	if let Some(geyser_notifier) = geyser_notifier {
	debug_assert!(geyser_notifier.snapshot_notifications_enabled());
	let account_for_geyser = AccountForGeyser {
	pubkey: account.pubkey(),
	lamports: account.lamports(),
	owner: account.owner(),
	executable: account.executable(),
	rent_epoch: account.rent_epoch(),
	data: account.data(),
	};
	geyser_notifier.notify_account_restore_from_snapshot(
	slot,
	write_version_for_geyser,
	&account_for_geyser,
	);
	write_version_for_geyser += 1;
	}

[RadiantAeon]

Might it be better to add an option to disable plugin manager? Geyser updates as a whole would be much faster without having to basically check if the loaded plugins are still the same for every single account, transaction, and slot update.

[RadiantAeon]

I'm looking into using ArcSwap, but if we do end up using it we end up modifying geyser plugin reload behavior. Currently, since the plugin manager is wrapped in a RwLock, when we reload we will:

1. Make the entire validator stop and wait because we've acquired a write lock for the geyser manager
2. Unload the old plugin
3. Load the new plugin
4. Drop the write lock and unpause all of replay

This is slow but notably it provides two guarantees: the old plugin will be dropped before the new plugin is loaded AND no geyser updates will be lost(since the entire validator waits for geyser plugins to be reloaded, the updates will always be sent to either the old plugin or the new plugin).

With ArcSwap, we can keep the guarantee that the old plugin will be dropped before the new plugin is loaded by waiting for all refs for the old plugin to be dropped. However, we cannot guarantee that updates will not be lost -- there will be some time where neither the old or new plugin is loaded. The only thing the code specifies for reload is that the old plugin must be unloaded before the new plugin.

agave/geyser-plugin-manager/src/geyser_plugin_manager.rs

Lines 213 to 215 in 9169c96

	// Unload and drop current plugin first in case plugin requires exclusive access to resource,
	// such as a particular port or database.
	self._drop_plugin(idx);

[RadiantAeon]

#9322

[RadiantAeon]

Stats:
Running ledger-tool verify, on the same snapshot archive
2. Current pr without geyser plugin: generate_index 45746852us, rebuild_bank 45746852us
3. Current pr with geyser plugin: generate_index 46216005us, rebuild_bank 50483453us
So it is faster. But I got some error when ledger tool completed which I will debug later.

ledger-tool-log.txt

[RadiantAeon]

Also tested reload, unload, and load with Yellowstone on a running validator without issue.