Support for external principals#5119
Open
adutra wants to merge 2 commits into
Open
Conversation
This change introduces support for fully external principals, that is, principals that are not backed by an entity in Polaris metastore. The new "principal mode" is configurable on a per-realm basis, and by default, internal mode is used (no changes). When a realm is configured to use external principals instead: - The authentication type for the realm must NOT be `internal` (i.e., an external IDP must be used). - The authorizer type must NOT be `internal` (i.e., an external PDP must be used). - The authenticator bypasses the metastore lookup when authenticating the principal. - The authenticator sets the principal roles directly from the credential's claims, as-is (no prefix stripping, no "ALL" pseudo-role expansion). - The resolver creates synthetic principal and principal role entities if asked by the authorizer to resolve the principal and/or principal roles. Note: in theory, an external authorizer shouldn't attempt to resolve principal roles, but for now it can happen, until the Authorizer SPI is fully refactored to allow more fine-grained resolutions. See design doc: https://docs.google.com/document/d/1VSoN1-QsAJGaM40oWTxeLIoYTp-XlfQM9nG-jqfb_-E/edit Prior work: apache#3250 GitHub issue: apache#441
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request adds a configurable “principal mode” per realm to support external principals (principals not backed by Polaris metastore entities), enabling authentication purely from external IDP credentials and synthetic principal/role resolution for external authorizers (e.g., OPA).
Changes:
- Introduces
PrincipalMode(INTERNAL/EXTERNAL) and wires it into realm authentication config, readiness checks, and default auth flow. - Updates
DefaultAuthenticatorandResolverto support external principals (skip metastore lookup; synthesize principal and role entities during resolution). - Adds OPA testcontainer support plus an end-to-end Keycloak+OPA integration test, and updates related docs/config reference.
Reviewed changes
Copilot reviewed 22 out of 22 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/testcontainers/opa/src/main/resources/org/apache/polaris/test/opa/Dockerfile-opa-version | Adds Renovate-managed OPA image tag source for tests. |
| tools/testcontainers/opa/src/main/java/org/apache/polaris/test/opa/OpaTestResource.java | Quarkus test resource to start OPA and inject Polaris authz config. |
| tools/testcontainers/opa/src/main/java/org/apache/polaris/test/opa/OpaContainer.java | Testcontainers wrapper for OPA plus policy upload helper. |
| tools/testcontainers/opa/build.gradle.kts | New Gradle module for the OPA testcontainer utilities. |
| site/content/in-dev/unreleased/managing-security/external-idp/idp-dev-notes.md | Documents how external principal mode changes the auth flow. |
| site/content/in-dev/unreleased/managing-security/external-idp/_index.md | Adds user-facing documentation for enabling/configuring external principals. |
| site/content/in-dev/unreleased/configuration/config-sections/smallrye-polaris_authentication.md | Updates generated config reference for principal-mode. |
| runtime/service/src/test/java/org/apache/polaris/service/config/ProductionReadinessChecksTest.java | Adds tests for external principal readiness constraints. |
| runtime/service/src/test/java/org/apache/polaris/service/auth/DefaultAuthenticatorTest.java | Adds unit coverage for external principal authentication behavior. |
| runtime/service/src/main/java/org/apache/polaris/service/config/ProductionReadinessChecks.java | Adds readiness checks enforcing external principal configuration constraints. |
| runtime/service/src/main/java/org/apache/polaris/service/auth/PrincipalMode.java | New enum modeling internal vs external principal mode. |
| runtime/service/src/main/java/org/apache/polaris/service/auth/internal/broker/InternalPolarisToken.java | Makes internal token type public for cross-package detection. |
| runtime/service/src/main/java/org/apache/polaris/service/auth/external/mapping/PrincipalMapper.java | Clarifies mapping contract docs for external principals. |
| runtime/service/src/main/java/org/apache/polaris/service/auth/DefaultAuthenticator.java | Implements external principal authentication path and role handling changes. |
| runtime/service/src/main/java/org/apache/polaris/service/auth/AuthenticationRealmConfiguration.java | Adds principalMode() realm configuration option. |
| runtime/service/src/intTest/java/org/apache/polaris/service/it/ExternalPrincipalKeycloakOpaIT.java | New E2E integration test for Keycloak-authenticated external principals with OPA authz. |
| runtime/service/build.gradle.kts | Wires in OPA auth extension and OPA testcontainer dependency. |
| runtime/defaults/src/main/resources/application.properties | Adds default polaris.authentication.principal-mode setting and per-realm example. |
| polaris-core/src/test/java/org/apache/polaris/core/persistence/ResolverTest.java | Adds coverage for synthetic resolution of external principals/roles. |
| polaris-core/src/main/java/org/apache/polaris/core/persistence/resolver/Resolver.java | Synthesizes caller principal and principal roles for external principals. |
| gradle/projects.main.properties | Registers the new polaris-opa-testcontainer module. |
| bom/build.gradle.kts | Exposes the new OPA testcontainer module in the BOM. |
Comments suppressed due to low confidence (1)
runtime/service/src/main/java/org/apache/polaris/service/auth/external/mapping/PrincipalMapper.java:49
- Javadoc return description for mapPrincipalId() still refers to returning a "Polaris principal" even though the method returns an OptionalLong principal ID. This is confusing for implementers and readers.
*
* @param identity the {@link SecurityIdentity} of the user
* @return the Polaris principal, or an empty optional if no mapping is available
*/
OptionalLong mapPrincipalId(SecurityIdentity identity);
Comment on lines
+106
to
110
| var principalName = entity != null ? entity.getName() : credentials.getPrincipalName(); | ||
| var polarisPrincipal = PolarisPrincipal.of(principalName, attributes, roleSelection.roles()); | ||
|
|
||
| LOGGER.debug("Resolved internal principal: {}", polarisPrincipal); | ||
| return polarisPrincipal; |
Comment on lines
+70
to
+81
| URL url = getExternalUrl().resolve("v1/policies/" + name).toURL(); | ||
| HttpURLConnection conn = (HttpURLConnection) url.openConnection(); | ||
| conn.setRequestMethod("PUT"); | ||
| conn.setDoOutput(true); | ||
| conn.setRequestProperty("Content-Type", "text/plain"); | ||
| try (OutputStream os = conn.getOutputStream()) { | ||
| os.write(rego.getBytes(StandardCharsets.UTF_8)); | ||
| } | ||
| int code = conn.getResponseCode(); | ||
| if (code < 200 || code >= 300) { | ||
| throw new IllegalStateException("OPA policy upload failed, HTTP " + code); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change introduces support for fully external principals, that is, principals that are not backed by an entity in Polaris metastore.
The new "principal mode" is configurable on a per-realm basis, and by default, internal mode is used (no changes).
When a realm is configured to use external principals instead:
internal(i.e., an external IDP must be used).internal(i.e., an external PDP must be used).Note: in theory, an external authorizer shouldn't attempt to resolve principal roles, but for now it can happen, until the Authorizer SPI is fully refactored to allow more fine-grained resolutions.
See design doc:
https://docs.google.com/document/d/1VSoN1-QsAJGaM40oWTxeLIoYTp-XlfQM9nG-jqfb_-E/edit
Prior work:
#3250
GitHub issue:
#441
Checklist
CHANGELOG.md(if needed)site/content/in-dev/unreleased(if needed)