feat(query): 补充数据处理函数与异常诊断 - #31
Merged
Merged
Conversation
zhou-hao
changed the title
zhou-hao
reviewed
Jul 2, 2026
| .as(wrapper); | ||
| } | ||
|
|
||
| private Object evaluate(JsonLimits limits, List<Object> args, JsonPath path1, JsonPath path2, JsonPath path3, List<JsonPath> restPaths) { |
Member
There was a problem hiding this comment.
不应该每次switch吧? 在createMapper 时就可以确定行为了啊. 而且这个类膨胀太大了, 单独封装一个抽象json path, 然后每个行为单独继承?
Member
There was a problem hiding this comment.
已处理：JsonPathFunctionMapFeature 现在只负责参数校验、静态路径预编译和参数流装配；具体行为拆到 JsonPathFunctionFeatures 的各函数实现，运行期不再按函数名 switch。同时拆出 JsonFunctionSupport、JsonValueSupport、JsonCollectionOperations，降低单类膨胀。CI build / Codacy / Codecov 均已通过。
zhou-hao
force-pushed
the
codex/json-functions-plan
branch
from
July 2, 2026 01:19
97eafd1 to
bc6f050
Compare
zhou-hao
force-pushed
the
codex/json-functions-plan
branch
from
July 2, 2026 02:24
c97b241 to
c52143e
Compare
There was a problem hiding this comment.
Pull request overview
This PR extends ReactorQL’s built-in function set to better match common MySQL/PostgreSQL JSON behaviors and adds general-purpose data processing functions (string/regex/date), while tightening safety limits via metadata settings, fixing count(*) parsing compatibility, and updating build tooling for Java 21 coverage collection.
Changes:
- Add a JSONPath-based JSON function stack (path compilation, normalization, collection ops) plus MySQL/PostgreSQL-compatible JSON function names.
- Register a broader set of common numeric/string/regex/date processing functions with configurable hard-limited safety settings.
- Fix
count(*)handling for JSqlParser 4.6AllColumns, and upgrade JaCoCo to support Java 21.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| src/test/resources/logback-test.xml | Adds test Logback config and reduces JsonPath logger noise. |
| src/test/java/org/jetlinks/reactor/ql/supports/map/FunctionMapFeatureCompatibilityTest.java | Adds tests ensuring legacy FunctionMapFeature behavior remains compatible with metadata-aware mappers. |
| src/test/java/org/jetlinks/reactor/ql/ReactorQLTest.java | Expands functional coverage for count(*), common SQL-like functions, regex functions, and JSON function behaviors/limits. |
| src/test/java/org/jetlinks/reactor/ql/JsonFunctionPerformanceTest.java | Adds performance smoke tests for JSON/static-path and common function pipelines. |
| src/test/java/org/jetlinks/reactor/ql/JsonFunctionCoverageTest.java | Adds extensive JSON edge/negative/security coverage for new JSON functions. |
| src/test/java/org/jetlinks/reactor/ql/CommonFunctionCoverageTest.java | Adds boundary/safety tests for new string/regex/date functions and settings. |
| src/main/java/org/jetlinks/reactor/ql/supports/map/JsonValueSupport.java | Implements JSON value normalization, validation, depth/container/text/output limits, and scalar helpers. |
| src/main/java/org/jetlinks/reactor/ql/supports/map/JsonPathFunctionMapFeature.java | Introduces a JSONPath function base class with static path compilation and argument assembly. |
| src/main/java/org/jetlinks/reactor/ql/supports/map/JsonPathFunctionFeatures.java | Implements per-function JSON behaviors without runtime name dispatch. |
| src/main/java/org/jetlinks/reactor/ql/supports/map/JsonFunctionSupport.java | Centralizes settings parsing, JSONPath safety checks, and shared JSON function logic. |
| src/main/java/org/jetlinks/reactor/ql/supports/map/JsonFunctionContext.java | Adds a per-evaluation context carrying args, limits, and compiled static paths. |
| src/main/java/org/jetlinks/reactor/ql/supports/map/JsonCollectionOperations.java | Adds deep-equals, contains/overlaps, set ops, and merge semantics (including MySQL preserve/patch behavior). |
| src/main/java/org/jetlinks/reactor/ql/supports/map/FunctionMapFeature.java | Adds metadata-aware mapper support while preserving the legacy protected apply(record, mappers) override chain. |
| src/main/java/org/jetlinks/reactor/ql/supports/DefaultReactorQLMetadata.java | Registers common data processing functions + JSON functions and enforces configurable hard-limited safety bounds. |
| src/main/java/org/jetlinks/reactor/ql/supports/agg/CountAggFeature.java | Fixes count(*) for both isAllColumns() and parameter-list AllColumns forms. |
| pom.xml | Upgrades JaCoCo plugin and adds com.jayway.jsonpath:json-path. |
| docs/plans/json-functions-support.md | Documents implemented function set, limits, and next-step registry/policy direction. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+188
to
+191
| addGlobal(JsonPathFunctionMapFeature.jsonGet("json_get", 2, 999, false)); | ||
| addGlobal(JsonPathFunctionMapFeature.jsonExtract("json_extract", 2, 999)); | ||
| addGlobal(JsonPathFunctionMapFeature.jsonGet("json_value", 2, 999, true)); | ||
| addGlobal(JsonPathFunctionMapFeature.jsonGet("json_query", 2, 999, false)); |
| - `json_unquote(value)`：结构化值转 JSON 字符串，普通标量转字符串。 | ||
| - `json_valid(value)`：非字符串对象视为合法 JSON；字符串必须能解析为 JSON 才返回 `true`。 | ||
| - `json_type` / `json_typeof(value)`：返回 `object`、`array`、`string`、`integer`、`double`、`boolean`、`null`。 | ||
| - `json_length(json[, path])`：对象返回 key 数，数组返回元素数，标量返回 1，空值返回 0。 |
Comment on lines
+18
to
+28
| import org.junit.jupiter.api.Assertions; | ||
| import org.junit.jupiter.api.Test; | ||
| import reactor.core.publisher.Flux; | ||
| import reactor.test.StepVerifier; | ||
|
|
||
| import java.time.Duration; | ||
| import java.util.Collections; | ||
| import java.util.HashMap; | ||
| import java.util.Map; | ||
|
|
||
| class JsonFunctionPerformanceTest { |
Comment on lines
+129
to
+140
| private Flux<Object> createParameterStream(ReactorQLRecord record, | ||
| List<Function<ReactorQLRecord, Publisher<Object>>> mappers) { | ||
| return Flux.fromIterable(mappers) | ||
| .flatMap(mp -> { | ||
| if (defaultValue != null) { | ||
| return Mono | ||
| .fromDirect(mp.apply(record)) | ||
| .defaultIfEmpty(defaultValue); | ||
| } | ||
| return mp.apply(record); | ||
| }); | ||
| } |
Comment on lines
+341
to
+346
| private static String toPathSegment(String key) { | ||
| if (INTEGER.matcher(key).matches()) { | ||
| return "[" + key + "]"; | ||
| } | ||
| return "['" + key.replace("'", "\\'") + "']"; | ||
| } |
- 补充 JSON、时间、字符串等常用函数与 select 列解析场景 - 引入结构化 ReactorQLException 和中英文资源，统一用户可见错误建议 - 增强 ORDER BY、merge_by_key、窗口、聚合、JSON 等异常与安全限制测试
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
目的
count(*)在 JSqlParser 4.6 下被解析为AllColumns参数时未按全量计数处理的问题。ORDER BY的资源保护与 Top-N 排序能力，避免大量数据无界收集。ReactorQLMetadata#getSelectColumns()，在 SQL AST 释放后仍可结构化获取查询列。format_datetime(l.timestamp, 'yyyy-MM-dd HH:mm:ss')、cast(... as double)/DOUBLE PRECISION等常见函数兼容问题。核心变动
json_path作为json_get等价别名。..、filter、函数调用、通配符等恶意或高成本构造。json_merge/json_merge_preserve按 MySQL 保留重复 key 语义处理，不简化为Map.putAll；json_merge_patch按 RFC 7396 处理 null 删除。repeat、replace、regexp_replace、正则输入、JSON 文本/输出/深度/容器/路径等限制支持通过 metadata settings 调整，并通过硬上限防止 SQL hint 绕过保护。count(*)同时兼容Function#isAllColumns()和参数列表中的AllColumns形态，保持数据库常见语义。OrderBySupport：全局排序默认受orderBy.maxRows保护；ORDER BY ... LIMIT使用 Top-N 候选窗口；orderBy.windowSize仅表示显式窗口内局部排序。Column、SQLType和ReactorQLMetadata#getSelectColumns()；支持普通 select item、聚合/函数/表达式、select *、t.*、子查询、UNION首个分支列、WITH 列别名等场景。format_datetime作为date_format等价别名；增强CastFeature类型归一化，支持double precision、bigint、numeric(...)、real/float4/float8等常见类型别名。ReactorQLException，保留UnsupportedOperationException兼容性，结构化承载错误码、表达式、行列、原因、建议、示例和 i18n 参数。i18n/reactorql/messages_zh.properties与messages_en.properties，用于默认中英文错误输出。regexp_replace('abc','a','$9')这类非法替换分组异常，避免底层 JDK 异常直接泄漏为不可控错误。jacoco-maven-plugin到0.8.13，修复 Java 21 classfile instrumentation 问题。FeatureRegistry/FeaturePolicy方案：构造 ReactorQL 时绑定 registry/policy；工具类函数自控输入安全，平台内部访问类函数通过 policy 显式授权。设计与测试目标
docs/plans/json-functions-support.md、docs/plans/streaming-order-by.md、docs/plans/reactorql-exception-diagnostics.mdcount(*)、非法正则替换分组、JSON 错误结构/恶意路径、FunctionMapFeature 兼容分支、流式排序资源边界、查询列解析、format_datetime与cast类型别名、结构化异常诊断和 i18n 资源。Column、SQLType、ReactorQLMetadata#getSelectColumns()公共契约注释；SelectColumnParser、OrderBySupport、JSON 函数抽象与 helper 已说明职责、边界与数据库兼容语义；关键 merge 行为、metadata settings 硬上限、非法正则替换收敛、异常诊断边界保留代码或设计文档说明。ORDER BY通过最大行数、Top-N 和局部窗口排序避免无界内存风险。测试结果
mvn -q testReactorQLExceptionTest：覆盖结构化异常字段、兼容性、语法错误、函数参数错误、JSON 非安全路径、ORDER BY 资源限制、日期格式、窗口参数、i18n 资源键和建议文案边界。ReactorQLTest#testCount：覆盖count(*)与sum(*)星号聚合参数回归场景。ReactorQLTest#testOrderByMaxRows、testOrderByLimitUseBoundedTopN、testOrderByDynamicLimitUseBoundedTopN、testOrderByTopNBufferLimit、testOrderByWindowSize、testOrderByMultipleColumnsAndNullOrdering、testOrderByInvalidSettings：覆盖排序资源限制、Top-N、动态 limit、窗口排序、多列/null 排序与非法设置。ReactorQLTest#testFormatDatetimeAndCastWithTableAlias：覆盖format_datetime(l.timestamp, ...)、cast(l.value as double)、DOUBLE PRECISION、BIGINT。ReactorQLTest#testJsonPathFunctions、testJsonInvalidAndMaliciousInputs：覆盖json_path等价别名、错误结构数据、恶意路径和动态路径保护。MergeByKeyFeatureTest：覆盖 merge_by_key 参数、排序、重复键、资源限制和结构化异常。DefaultReactorQLMetadataTest#testSelectColumns*：覆盖普通列、聚合、函数、表达式、通配列、子查询、UNION、t.*、WITH 列别名、lateral/parenthesis/subJoin 以及 SQL AST release 后查询列缓存。JsonFunctionCoverageTest：覆盖 JSON 集合操作负向/嵌套分支、JSON 校验与参数保护分支。CommonFunctionCoverageTest：覆盖常用字符串/正则/日期边界函数与安全失败场景。FunctionMapFeatureCompatibilityTest：覆盖 legacy protected apply、metadata mapper、无参/defaultValue、参数保护、distinct/unique wrapper、异步参数顺序分支。260 passed, 0 failed, 0 errors, 0 skippedjson_get静态路径 20000 行约 392ms；通用数据函数 20000 行约 1468ms。93.57%81.43%94.52%90.97%78.94%93.31%文档同步情况
docs/plans/json-functions-support.md已从计划更新为实现状态与测试结论，补充 settings 可配置限制、JSON 函数实现拆分与下一版本 Feature Registry / Policy 设计。docs/plans/streaming-order-by.md记录流式ORDER BY的资源边界、Top-N 策略、窗口排序语义与测试计划。docs/plans/reactorql-exception-diagnostics.md记录结构化异常模型、错误码、公开输出边界、落地范围和测试要求。风险与说明
getSelectColumns()对普通表select *无外部 schema 时仍保留通配列，不凭空推断表结构。