Bundle plugin: O(n²) compilation cost during initial multi-bundle activation

[jsteinberg]

What is the underlying problem you're trying to solve?

When OPA is configured with a large number of bundles (e.g., via discovery), initial startup is extremely slow due to quadratic (O(n²)) compilation cost. Each bundle is activated individually as its download completes, and each activation recompiles all previously-loaded modules from scratch.

In our production deployment, we have a testing configuration that loads 80 bundles (~8-10 Rego files each, ~800 modules total). Startup takes several minutes because:

1. Serial activation: Plugin.oneShot() holds p.mtx (plugin.go:479), so bundle activations are serialized even though downloads are parallel.
2. Full recompilation per activation: Each call to Plugin.activate() creates a fresh ast.NewCompiler() (plugin.go:627) and compileModules() (store.go:962-993) collects all previously-activated modules plus the new bundle's modules, then calls compiler.Compile(modules) over the full set.
3. Quadratic total work: Bundle 1 compiles 10 modules, bundle 2 compiles 20, ..., bundle 80 compiles 800. Total: ~32,400 module-compilations vs. ~800 if compiled once.

The same issue affects loadAndActivateBundlesFromDisk() (plugin.go:378), which activates persisted bundles one at a time in a retry loop.

Notably, bundle.Activate() already supports multi-bundle batch compilation — activateBundles() in store.go:530 says "Compile the modules all at once to avoid having to re-do work" and accepts multiple bundles in opts.Bundles. The bundle plugin just never takes advantage of this during initial load.

Describe the ideal solution

During initial load (before the plugin reaches ready state), the bundle plugin would buffer downloaded bundles and batch-activate them in a single bundle.Activate() call, compiling all modules once. After the plugin is ready, it would revert to per-bundle activation for fast incremental updates.

A top-level configuration option (e.g., batch_bundle_activation: true) would control this behavior, keeping it opt-in and backward-compatible.

The change would include:

• A pendingBundles buffer on the Plugin struct to hold downloaded bundles until all expected bundles arrive (or a deadline expires)
• A new activateBatch() method that passes all buffered bundles to a single bundle.Activate() call

Error handling: bundle.Activate() is all-or-nothing — one invalid bundle fails the entire batch. On failure, the bad bundle(s) can be identified and have their status set accordingly. All bundles remain in the pending buffer. The downloaders continue polling, and when a new bundle arrives, it replaces the bad one in the buffer and batch activation is retried. The service stays not-ready until all bundles are valid — matching today's behavior.

Describe a "Good Enough" solution

Not sure there is a partial fix here. We could alternatively change the way we are using OPA to reduce the number of bundles on a single deployment.

Additional Context

• OPA version: v1.18.2 (but the architecture has been the same for many versions)
• Scale: 80 bundles, ~800 Rego modules, startup takes several minutes
• Key code paths:
  • plugins/bundle/plugin.go:479 — mutex serializes activations
  • plugins/bundle/plugin.go:607-694 — activate() creates a fresh compiler per bundle
  • bundle/store.go:962-993 — compileModules() recompiles all modules each time
  • bundle/store.go:530-535 — activateBundles() already supports multi-bundle compilation
• The ast.Compiler.Compile() method runs 28 stages (ref resolution, type checking, rule indexing, etc.) over every module — the per-module cost is significant

I am happy to contribute a PR for this if the approach seems reasonable

[sspaink]

Hello @jsteinberg, thank you for the detailed issue!

I don't see any problem with adding the configuration option batch_bundle_activation to batch bundles at startup. We will happily accept a PR 👍

[ARMeeru]

@jsteinberg are you still planning to pick this up? Your note said you were happy to contribute a PR, and I did not want to duplicate anything you already have in progress.

If you would rather not, I am glad to take it. To check I would be going the right way first: ActivateOpts already carries Bundles map[string]*Bundle, so the batched path looks like buffering the bundles during the initial disk load and calling Activate once, instead of once per bundle with a fresh compiler each time. I would put that behind batch_bundle_activation as @sspaink suggested, and leave the current serial behaviour as the default.

The part I would want a steer on is error handling. Batching means one bad bundle fails the whole batch, which loses the per-bundle isolation you get today, and that also affects what readiness reports while it happens. Happy to go whichever way you both prefer.