Feat/event data search merged #1106
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1106 +/- ##
==========================================
+ Coverage 77.24% 79.13% +1.88%
==========================================
Files 625 639 +14
Lines 27912 29950 +2038
==========================================
+ Hits 21561 23701 +2140
+ Misses 6351 6249 -102 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting
| def _build_event_filter_condition(self, filter_item: Dict[str, Any], index: int) -> str: | ||
| alias = f"risk_event_{index}" | ||
| join_conditions = [f"{alias}.strategy_id = base_query.strategy_id"] | ||
|
|
||
| field_expression = self._build_event_field_expression(alias, filter_item) | ||
| if not field_expression: | ||
| return "" | ||
|
|
||
| comparison = self._build_event_filter_expression(field_expression, filter_item) | ||
| if not comparison: | ||
| return "" | ||
|
|
||
| join_clause = " AND ".join(join_conditions) | ||
| table_reference = self._get_risk_event_table_reference() |
There was a problem hiding this comment.
The BKBase event filter predicate only joins on strategy_id when determining whether a risk matches the supplied event field filters. If two risks originate from the same strategy but have different event payloads, the EXISTS clause will return true for all of them as soon as any event for that strategy satisfies the filter, because the subquery never correlates on the risk’s raw_event_id (or another risk-specific key). This means filtering by event fields can return unrelated risks. The join should include a risk-specific identifier (e.g. raw_event_id or risk_id) and the column must be selected in the base subquery so that each risk is filtered against its own event record.
Useful? React with 👍 / 👎.
1. 增加支持过滤的field的id。添加过滤operator支持。通过过滤中英文名组合限制生效策略。 2. 添加时间分区过滤条件。 3. 使用sqllot重构sql生成逻辑，确保可用性。 4. 优化测试用例。 5. 简化sql语句，修正遗漏。
1. 新增事件字段筛选及策略相关接口与常量 2. 增加事件合流入库 doris - 增加事件 doris 合流表初始化 - 审计策略-规则审计，模型审计支持 doris 合流入库（已兼容历史策略） 3. 系统初始化 - 支持策略标签资源纳入权限模型并同步 IAM 初始化 - 增加事件相关资产反向拉取初始化 - 定时同步资产快照结果表并完善插件入库任务 - 移除风险模型标签冗余字段并优化 IAM Provider 4. 优化风险列表筛选条件,增加日志打印
1. 新增资源反向拉取 2. 修复 github 单测异常 3. 资产表反向拉取支持 JSON 字段
JetDrag
force-pushed
the
feat/event_data_search_merged
branch
2 times, most recently
from
bec2dee to
13ae5a5
Compare
1. 修正兼容性差的*写法。 2. 添加doris后缀。 3. 新增工单权限表资源反向拉取的初始化 4. 兼容query某些情况下提前为空的情况。 5. 移除表标识符格式化中的反引号包裹。 6. 新增事件时间关联约束
1. 优化模型审计策略更新逻辑，兼容 doris 存储节点变更
1. 新增重复字段处理逻辑及测试，优化类型与格式
Signed-off-by: raja <[email protected]> (cherry picked from commit 6606b32) (cherry picked from commit 8fd6fda)
Signed-off-by: raja <[email protected]> (cherry picked from commit d8514fa) (cherry picked from commit cd31c28) (cherry picked from commit 37b6c8a)
1. 新增 BKBase SQL 记录与票权限表替换
1. 调整标签，工单权限，策略的资产反向拉取 schema
1. 优化时区问题 2. 事件结束时间字段序列化后自动向上对齐到秒 3. 资产清洗字段非必要标准字段可以被业务字段覆盖。补充单测。
1. 事件去重字段关联检索优化
1. 风险信息事件结束时间按秒向上取整 (cherry picked from commit ac625dc)
1. 风险信息事件结束时间按秒向上取整 (cherry picked from commit 23fa976)
1. 去掉 DISABLE_RISK_PERMISSION_FILTER 测试环境变量
1. 去除mysql冗余 cas
JetDrag
force-pushed
the
feat/event_data_search_merged
branch
from
13ae5a5 to
976ba81
Compare
1. 时间字段值做时区转换
1. 添加排序功能 2. 重构关联事件检索BKBase链路代码结构，抽象重组代码。
JetDrag
force-pushed
the
feat/event_data_search_merged
branch
from
824abd6 to
e720bac
Compare
Signed-off-by: raja <[email protected]>
Signed-off-by: raja <[email protected]>
Signed-off-by: raja <[email protected]>
Signed-off-by: raja <[email protected]>
No description provided.