You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
sec already has a native implementation in the datafusion-spark crate, but was not yet wired into Comet, so queries using it fell back to Spark.
What changes are included in this PR?
Wires the Spark Sec expression to the datafusion-sparkSparkSec UDF (Pattern B: register the UDF in jni_api.rs + one-line passthrough in QueryPlanSerde.scala). Also updates the supported-expression docs.
welcome @comphead. Yes, the PR was created referring the wire_datafusion_function skill
thanks for trying it, it tested for simple wiring, and we need to make it also work for more complicated usecases, where datatypes can be partially supported or behavior is different, etc
welcome @comphead. Yes, the PR was created referring the wire_datafusion_function skill
thanks for trying it, it tested for simple wiring, and we need to make it also work for more complicated usecases, where datatypes can be partially supported or behavior is different, etc
Shall I try by expanding the skill file for more complicated usecases?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part #4150
Rationale for this change
secalready has a native implementation in thedatafusion-sparkcrate, but was not yet wired into Comet, so queries using it fell back to Spark.What changes are included in this PR?
Wires the Spark
Secexpression to thedatafusion-sparkSparkSecUDF (Pattern B: register the UDF injni_api.rs+ one-line passthrough inQueryPlanSerde.scala). Also updates the supported-expression docs.How are these changes tested?
Added a SQL file test (
math/sec.sql).