[Suggestion] Modify rabbit_plugins to honor optional_applications, enabling companion plugins to be merged into their parents

[lukebakken]

RabbitMQ series

main (4.4.x)

Operating system (distribution) used

N/A

How is RabbitMQ deployed?

N/A

What would you like to suggest for a future version of RabbitMQ?

Background

The RabbitMQ plugin system currently has no concept of optional dependencies. When a plugin lists another plugin in its applications key (via DEPS in erlang.mk), the plugin system implicitly enables that dependency whenever the declaring plugin is enabled. This forces the split-plugin pattern: rabbitmq_federation_management, rabbitmq_shovel_management, rabbitmq_stream_management, rabbitmq_federation_prometheus, and rabbitmq_shovel_prometheus all exist as separate plugins solely because there is no way for their parent plugin to depend on rabbitmq_management or rabbitmq_prometheus without unconditionally enabling them.

The gap

OTP 24 introduced optional_applications in .app files. erlang.mk already supports this via OPTIONAL_DEPS, which places apps in both applications and optional_applications. OTP's application_controller honors optional_applications at startup - it does not throw {not_started, App} for apps listed there.

However, rabbit_plugins:mkplugin/4 reads only applications when building #plugin.dependencies:

Dependencies = proplists:get_value(applications, Props, []),

It ignores optional_applications entirely, so the plugin system still treats optional deps as mandatory for implicit enablement.

Proposed change

1. Add an optional_dependencies field to #plugin{} in rabbit_common/include/rabbit.hrl
2. In rabbit_plugins:mkplugin/4, read optional_applications from the .app file and subtract those apps from dependencies:

OptionalDependencies = proplists:get_value(optional_applications, Props, []),
#plugin{...,
        dependencies          = Dependencies -- OptionalDependencies,
        optional_dependencies = OptionalDependencies,
        ...}

This is sufficient - dependencies/3 builds its graph from #plugin.dependencies only, and the rabbitmq-plugins CLI calls dependencies/3 directly, so both implicit enablement and the CLI's enable/disable/list commands automatically reflect the correct behavior with no further changes.

The interaction with extra_dependencies and is_strictly_plugin/1 is safe. ensure_dependencies/1 moves any app for which is_loadable/1 returns true from dependencies into extra_dependencies. is_loadable/1 succeeds for any app whose ebin is on the code path - this includes OTP/stdlib apps, RabbitMQ core apps like rabbit and rabbit_common, and any plugin present in the plugins directory. is_strictly_plugin/1 checks whether rabbit is in extra_dependencies, which it always will be for a real plugin.

The proposed change subtracts optional_applications from dependencies in mkplugin/4, before ensure_dependencies/1 runs. So optional deps never reach ensure_dependencies/1 at all - the interaction is fully safe regardless of whether is_loadable/1 would return true or false for them.

Opt-out configuration

Merging companion plugins into their parents raises a question: what if a user wants rabbitmq_shovel and rabbitmq_management running, but does not want the shovel management UI extension? Today that is possible by simply not enabling rabbitmq_shovel_management.

To preserve this control, each plugin would expose a cuttlefish setting that defaults to true (opt-out model). For example:

shovel.management.enabled = true
federation.management.enabled = true
stream.management.enabled = true
shovel.prometheus.enabled = true
federation.prometheus.enabled = true
stream.prometheus.enabled = true

The guard in each plugin's app module would check both whether the optional application is running and whether the setting is enabled:

case lists:keymember(rabbitmq_management, 1, application:which_applications()) andalso
     application:get_env(rabbitmq_shovel, management_enabled, true) of
    true  -> register_management_routes();
    false -> ok
end.

However, the management dispatcher also re-scans all running plugins at runtime when rabbitmq-plugins enable/disable is called on a live broker. It does this via rabbit_mgmt_reset_handler, which responds to plugins_changed events and calls rabbit_mgmt_app:reset_dispatcher/1. The dispatcher discovers extensions by finding all modules implementing the rabbit_mgmt_extension behaviour across all loaded RabbitMQ-related apps, then calling their dispatcher/0 callback. This means the opt-out guard must also live in dispatcher/0, not only at startup:

Note: rabbit_mgmt_reset_handler only triggers reset_dispatcher when a management extension changes (not when rabbitmq_management itself starts or stops - in that case the dispatcher is freshly created or destroyed). When rabbitmq_management starts, it calls start_configured_listeners([], true) which scans all already-loaded apps including rabbitmq_shovel, so the dispatcher/0 guard is respected in that path too.

%% In rabbit_shovel_mgmt_dispatcher (implements rabbit_mgmt_extension)
dispatcher() ->
    case application:get_env(rabbitmq_shovel, management_enabled, true) of
        true  -> [{"/shovels", rabbit_shovel_mgmt, []}]; %% etc.
        false -> []
    end.

This ensures the opt-out is respected both at startup and when plugins are enabled or disabled at runtime.

The default of true means existing users who upgrade get the same behaviour as today (management and/or prometheus extension active when both plugins are enabled), with no configuration change required. Users who want to opt out set the flag to false.

Impact: companion plugins that could be merged

With this change, the following thin plugins could be merged into their parents:

Plugin to eliminate	Merge target
rabbitmq_federation_prometheus	rabbitmq_federation
rabbitmq_shovel_prometheus	rabbitmq_shovel
rabbitmq_federation_management	rabbitmq_federation
rabbitmq_shovel_management	rabbitmq_shovel
rabbitmq_stream_management	rabbitmq_stream

Each merged plugin would guard its optional functionality on whether the relevant application (prometheus or rabbitmq_management) is running.

Backwards compatibility

When a companion plugin is merged and its name disappears, users with the old name in enabled_plugins will not experience a startup failure. prepare_plugins/1 builds its graph only from plugins present in the plugins directory - unknown names are silently skipped. Release notes should document the change so users can clean up their enabled_plugins files.

Test coverage

Neither existing plugin test suite (unit_plugin_directories_SUITE, unit_plugin_versioning_SUITE) covers dependencies/3, mkplugin/4, or ensure_dependencies/1. New unit tests would be required, covering: optional deps excluded from implicit enablement, missing optional deps not causing startup errors, mandatory deps still enforced, and the upgrade scenario where a stale companion plugin name is present in enabled_plugins.

If this sounds like a useful contribution I'll whip up a PR, thanks.

[michaelklishin]

@lukebakken thank you. The Broadcom side of the core team has some grievances with the current plugin system as well.

We need to try to get together on video to design something that'd hopefully give everyone what they find missing.

[lukebakken]

Great! Let's chat. I'll contact you via the Discords.

[lhoguin]

Makes sense.

I don't know how well OPTIONAL_DEPS works, I don't remember, there might be some tweaks needed to make Dialyzer/xref happy. Nothing huge I'm sure.