OPAL_SCOPES=1 affects backward compatibility with non-scoped clients

[vorbidan]

When OPAL server is started with OPAL_SCOPES=1 environment variable, it exhibits the following behaviors that prevent seamless operation with both scoped and non-scoped clients:

Current Issues:

1. Creates ephemeral "default" scope - Uses OPAL_POLICY_REPO_* env vars for policy configuration
2. Ignores data configuration - The default scope is created with empty data configuration, not inheriting from OPAL_DATA_CONFIG_SOURCES
3. Prevents mixed client usage - Makes it difficult to run both scoped and non-scoped clients against the same server
4. Non-scoped endpoints may not work - When OPAL_SCOPES=1, traditional endpoints like /policy and /data/config may not function properly

Desired State - Dual Mode Support:

The same OPAL server instance with OPAL_SCOPES=1 should serve BOTH scoped and non-scoped clients:

For Non-Scoped Clients (Backward Compatibility):

These endpoints should work identically regardless of whether OPAL_SCOPES=1 is set:

1. /policy → Returns policies from OPAL_POLICY_REPO_URL (if specified)
2. /policy-data → Returns policy data
3. /data/config → Returns data config from OPAL_DATA_CONFIG_SOURCES (if specified)

For Scoped Clients (Default Scope):

When OPAL_SCOPES=1 is enabled, the "default" scope should mirror the non-scoped configuration:

1. /scopes/default/policy → Returns policies from OPAL_POLICY_REPO_URL (if specified)
2. /scopes/default/data → Returns data config entries from OPAL_DATA_CONFIG_SOURCES (if specified)

Key Requirement:

The same server configuration should support both client types simultaneously, enabling gradual migration from non-scoped to scoped deployments without disruption.

[orweis]

Hi @vorbidan - thanks for sharing.

There was never any intention for OPAL to support both scopes and non scoped behavior. migrating from a non scoped setup is rather trivial.

This doesn't seem to qualify as an issue.
is there a specific problem you are experiencing? Otherwise will likely convert to this to a discussion.

[vorbidan]

Hi @orweis
Thank you for chiming in so quickly - converting it to discussion is totally fine, maybe even preferred, since I am still learning what OPAL does.
My use case is rather trivial. We have a set of customers, which will be using the default policies/data-config. These are our MVP customers and OPAL without scopes, with the same data config is more than adequate.
Our future plans, however include isolated consumers, which may require different policies/data, so combining scopped vs. non-scopped servers/clients makes sense, IMO, with having the same server support both.
Unless, of course, there are operational limitations which I am not aware of at this point.

Thank you again!

[orweis]

If you are planning to use scopes, I'd suggest setting up all your servers with scopes by default, and just using a single scope in the simpler case.

Scopes are the recommended way to use OPAL in production, and the simpler non scope state is a product of

1. gradual development
2. desire to keep a simpler setup as an option for beginners and / or truly simple production or testing deployments.

[vorbidan]

Yes, that would probably be a viable approach, but scopes implementation has issues, which were mentioned in a bug #773 and a PR #824.
I was hoping to first make sure that the scoped vs. non-scoped server works in the same manner when it comes to respecting the Server ENV vars and creating an ephemeral "default" scope.

I can take a stab at the PR if you think this fits the OPAL direction.

[orweis]

I don't think other issues should come into play while considering the recommended strategy here.

We should simply fix other issues when needed.
I've asked RND to take a look at these issues again - I think it got sidelined due to other priorities.

As a side Note: in the background we're working on a new major version of OPAL (aka OPAL 2.0) with the pub/sub component likely to be replaced by NATS.io. I've asked our RND to publish a message and the plans for this next version, for transparency and getting the support of the community here.

[vorbidan]

Thank you @orweis,
Would love to hear more with regard to OPAL 2.0. Especially interested to see if Scopes are still going to be supported, and the rationale behind moving to NATS.io, because currently having both Scopes and the pub/sub utilizing the same backend database makes a lot of sense to me.

[orweis]

An official post with all the details would follow.
CC: @zeevmoney , @omer9564 , @danyi1212