ADH-8178: add gRPC authentication between Kyuubi server and Spark engine#25
Open
werzerbb wants to merge 4 commits into
Open
ADH-8178: add gRPC authentication between Kyuubi server and Spark engine#25werzerbb wants to merge 4 commits into
werzerbb wants to merge 4 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Added token authentication to gRPC connection Kyuubi server uses to talk to Spark engine
(Spark Connect), extending the existing
kyuubi.engine.security.enabledAES-encrypted-token auththat earlier only covered Thrift connection.
What was added
KyuubiEngineTokenServerInterceptor(engine side): rejects gRPC calls missing a validx-kyuubi-engine-tokenheader. Registered on engine's Spark Connect service only whenkyuubi.engine.security.enabled=true.KyuubiEngineTokenClientInterceptor(Kyuubi server side): attaches a freshInternalSecurityAccessor-issued token to outgoing gRPC call.(
kyuubi.ha.zookeeper.engine.secure.secret.node, default/kyuubi_engine_secure_secret) is nowauto-created with a random secret on first use. Disable with new option
kyuubi.ha.zookeeper.engine.secure.secret.node.auto-create=falseto provision it manually.New config options
kyuubi.ha.zookeeper.engine.secure.secret.node.auto-createtruefalseto require manual provisioningChanged config defaults
kyuubi.ha.zookeeper.engine.secure.secret.nodenow defaults to/kyuubi_engine_secure_secret(previously had no default and required manual configuration)
Manual secret provisioning (when auto-create is disabled)
With
kyuubi.ha.zookeeper.engine.secure.secret.node.auto-create=false, the ZK secret node must becreated before starting Kyuubi. The secret should be at least
kyuubi.engine.security.crypto.keyLength / 8characters (16 for the default 128-bit key)see an example:
Alternatively, skip ZooKeeper entirely with the static
simpleprovider - set the same secret oneach Kyuubi server: