Skip to content

feat: 日志查询支持doris --story=121933504 #712

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 162 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
162 commits
Select commit Hold shift + click to select a range
e2c2f91
fix: 修复共享集群过滤无效问题 --bug=129763639 (#505)
EASYGOING45 Aug 28, 2024
2cc6884
feat: BCS共享集群-添加更多日志 --story=119412655 (#507)
EASYGOING45 Aug 28, 2024
c2a64e3
sync: master branch (#509)
chenjiandongx Aug 29, 2024
d776e8c
feat: 解决 bksql 鉴权参数为空的问题 --story=119438132 (#511)
shamcleren Aug 29, 2024
fd5e030
Merge branch 'release'
shamcleren Aug 29, 2024
56a128c
Merge branch 'release'
shamcleren Aug 29, 2024
73673e9
Merge branch 'master' of github.com:TencentBlueKing/bkmonitor-datalink
shamcleren Aug 29, 2024
facbd04
Merge branch 'master' of github.com:TencentBlueKing/bkmonitor-datalink
shamcleren Aug 29, 2024
fc55a67
Merge branch 'master' of github.com:TencentBlueKing/bkmonitor-datalink
shamcleren Sep 4, 2024
c46dabc
Merge branch 'master' of github.com:TencentBlueKing/bkmonitor-datalink
shamcleren Sep 9, 2024
f211ddb
Merge branch 'master' of github.com:TencentBlueKing/bkmonitor-datalink
shamcleren Sep 13, 2024
7d4b16b
Merge branches 'master' and 'master' of github.com:TencentBlueKing/bk…
shamcleren Sep 27, 2024
cdbbad1
Merge branches 'master' and 'master' of github.com:TencentBlueKing/bk…
shamcleren Oct 30, 2024
0eb964d
Merge branch 'master' of github.com:TencentBlueKing/bkmonitor-datalink
shamcleren Nov 4, 2024
743e424
Merge branches 'master' and 'master' of github.com:TencentBlueKing/bk…
shamcleren Nov 14, 2024
89cb6a0
Merge branches 'master' and 'master' of github.com:TencentBlueKing/bk…
shamcleren Dec 2, 2024
9e34c4b
Merge branch 'master' of github.com:TencentBlueKing/bkmonitor-datalink
shamcleren Dec 6, 2024
1674b73
Merge branch 'master' of github.com:TencentBlueKing/bkmonitor-datalink
shamcleren Dec 9, 2024
121d75b
feat: 支持 es query string 语法解析 #1010158081121859811
shamcleren Feb 8, 2025
df195b6
feat: 支持 es query string 语法解析 #1010158081121859811
shamcleren Feb 8, 2025
34f457a
feat: 支持 es query string 语法解析 #1010158081121859811
shamcleren Feb 8, 2025
bc2bb49
feat: 支持 es query string 语法解析 #1010158081121859811
shamcleren Feb 8, 2025
9e67702
feat: 支持 es query string 语法解析 #1010158081121859811
shamcleren Feb 10, 2025
9c8bf14
feat: 支持 es query string 语法解析 #1010158081121859811
shamcleren Feb 10, 2025
d5fbee6
feat: 支持 es query string 语法解析 #1010158081121859811
shamcleren Feb 11, 2025
9ca61bf
feat: 支持 es query string 语法解析 #1010158081121859811
shamcleren Feb 11, 2025
8c88d58
Merge remote-tracking branch 'refs/remotes/upstream/master' into feat…
shamcleren Feb 13, 2025
f645cc8
Merge branch 'master' into feat/es_query_string_parser_yacc/#10101580…
shamcleren Feb 13, 2025
6093924
feat: 日志查询支持doris #1010158081121933504
shamcleren Feb 13, 2025
bd07b11
feat: 日志查询支持doris #1010158081121933504
shamcleren Feb 13, 2025
b84789b
feat: 日志查询支持doris #1010158081121933504
shamcleren Feb 18, 2025
de7cbcf
Merge branch 'feat/es_query_string_parser_yacc/#1010158081121859811' …
shamcleren Feb 19, 2025
6dff927
feat: 日志查询支持doris #1010158081121933504
shamcleren Feb 20, 2025
019a91f
Merge remote-tracking branch 'upstream/master' into feat/log_query_fo…
shamcleren Feb 20, 2025
73302fc
feat: 日志查询支持doris #1010158081121933504
shamcleren Feb 21, 2025
d837422
feat: 日志查询支持doris #1010158081121933504
shamcleren Feb 26, 2025
c74f7e5
feat: 日志查询支持doris #1010158081121933504
shamcleren Feb 26, 2025
b5c8b06
feat: 修改单测
shamcleren Feb 26, 2025
8970c5d
feat: 日志查询支持doris #1010158081121933504
shamcleren Feb 26, 2025
ebcd361
Merge branch 'feat/log_query_for_doris/#1010158081121933504' of githu…
shamcleren Feb 26, 2025
4301113
feat: 日志查询支持doris #1010158081121933504
shamcleren Feb 26, 2025
25c6006
feat: 日志查询支持doris #1010158081121933504
shamcleren Feb 26, 2025
d16ae66
Merge branch 'master' of github.com:TencentBlueKing/bkmonitor-datalink
shamcleren Mar 5, 2025
f47ee1b
Merge branches 'master' and 'master' of github.com:TencentBlueKing/bk…
shamcleren Mar 6, 2025
008af22
Merge branch 'master' into feat/log_query_for_doris/#1010158081121933504
shamcleren Mar 6, 2025
c0afb65
Merge remote-tracking branch 'origin/feat/log_query_for_doris/#101015…
shamcleren Mar 6, 2025
760011a
feat: 日志查询支持doris #1010158081121933504
shamcleren Mar 7, 2025
80fd646
feat: 日志查询支持doris #1010158081121933504
shamcleren Mar 7, 2025
ffd4d87
feat: 日志查询支持doris #1010158081121933504
shamcleren Mar 7, 2025
8734f85
feat: 日志查询支持doris #1010158081121933504
shamcleren Mar 12, 2025
4a5d575
feat: 优化 json 函数 #1010158081122355416
shamcleren Mar 13, 2025
0dc75f7
Merge branch 'feat/optimization_json/#1010158081122355416' into feat/…
shamcleren Mar 13, 2025
70c8b37
feat: 日志查询支持doris #1010158081121933504
shamcleren Mar 13, 2025
bd56f0e
Merge branch 'master' of github.com:TencentBlueKing/bkmonitor-datalin…
shamcleren Mar 13, 2025
03d1a87
feat: 日志查询支持doris #1010158081121933504
shamcleren Mar 13, 2025
1bb0ada
feat: 日志查询支持doris #1010158081121933504
shamcleren Mar 13, 2025
16b09b8
feat: 日志查询支持doris #1010158081121933504
shamcleren Mar 13, 2025
f025797
feat: 日志查询支持doris #1010158081121933504
shamcleren Mar 17, 2025
40e53d2
Merge branch 'master' of github.com:TencentBlueKing/bkmonitor-datalink
shamcleren Mar 18, 2025
54edeb9
Merge branch 'master' into feat/log_query_for_doris/#1010158081121933504
shamcleren Mar 18, 2025
c269606
feat: 日志查询支持doris #1010158081121933504
shamcleren Mar 19, 2025
4c2a2dc
Merge remote-tracking branch 'upstream/master' into feat/log_query_fo…
shamcleren Mar 20, 2025
94da045
feat: 日志查询支持doris #1010158081121933504
shamcleren Mar 20, 2025
5ac4b18
feat: 日志查询支持doris #1010158081121933504
shamcleren Mar 20, 2025
f16dea3
feat: 日志查询支持doris #1010158081121933504
shamcleren Mar 21, 2025
1f294d8
feat: 日志查询支持doris #1010158081121933504
shamcleren Mar 26, 2025
1836f3d
Merge branch 'master' of github.com:TencentBlueKing/bkmonitor-datalin…
shamcleren Mar 26, 2025
c248e02
feat: 日志查询支持doris #1010158081121933504
shamcleren Mar 26, 2025
074130d
Merge branch 'master' into feat/log_query_for_doris/#1010158081121933504
shamcleren Mar 27, 2025
b8fc582
feat: 日志查询支持doris #1010158081121933504
shamcleren Mar 27, 2025
6573b81
feat: es查询支持 scroll 和 searchafter #1010158081122617468
shamcleren Mar 27, 2025
4585959
feat: es查询支持 scroll 和 searchafter #1010158081122617468
shamcleren Mar 31, 2025
01ebf4d
feat: es 聚合 missing 兼容 int 类型的字段 #1010158081122704186
shamcleren Apr 1, 2025
5555dba
Merge branch 'feat/es_aggregate_missing_with_int/#1010158081122704186…
shamcleren Apr 1, 2025
2328bf8
feat: es 聚合 missing 兼容 int 类型的字段 #1010158081122704186
shamcleren Apr 1, 2025
00d8965
Merge branch 'feat/es_aggregate_missing_with_int/#1010158081122704186…
shamcleren Apr 1, 2025
bc085c8
feat: check_influxdb_group_by_metric_name #1010158081122720875
shamcleren Apr 1, 2025
132cce2
Merge branch 'feat/check_influxdb_group_by_metric_name/#1010158081122…
shamcleren Apr 1, 2025
455d0a6
Merge branch 'master' into feat/log_query_for_doris/#1010158081121933504
shamcleren Apr 1, 2025
e79482f
feat: es查询支持 scroll 和 searchafter #1010158081122617468
shamcleren Apr 2, 2025
2c24337
feat: es 全字段查询兼容 #1010158081122750810
shamcleren Apr 2, 2025
a05bd02
Merge branch 'feat/es_query_string_with_fields/#1010158081122750810' …
shamcleren Apr 2, 2025
ac483fc
Merge branch 'master' of github.com:TencentBlueKing/bkmonitor-datalin…
shamcleren Apr 2, 2025
9bc072d
feat: es查询支持 scroll 和 searchafter #1010158081122617468
shamcleren Apr 3, 2025
58da710
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 7, 2025
b477c7a
feat: es查询支持 scroll 和 searchafter #1010158081122617468
shamcleren Apr 7, 2025
94b3745
feat: es查询支持 scroll 和 searchafter #1010158081122617468
shamcleren Apr 8, 2025
1dc09a1
feat: es查询支持 scroll 和 searchafter #1010158081122617468
shamcleren Apr 8, 2025
1da26ba
feat: es查询支持 scroll 和 searchafter #1010158081122617468
shamcleren Apr 8, 2025
4bcb683
feat: es查询支持 scroll 和 searchafter #1010158081122617468
shamcleren Apr 8, 2025
af38d63
feat: es查询支持 scroll 和 searchafter #1010158081122617468
shamcleren Apr 8, 2025
c12ea40
feat: es查询支持 scroll 和 searchafter #1010158081122617468
shamcleren Apr 8, 2025
ac38f26
feat: es查询支持 scroll 和 searchafter #1010158081122617468
shamcleren Apr 8, 2025
70f20b4
feat: es查询支持 scroll 和 searchafter #1010158081122617468
shamcleren Apr 9, 2025
44cc07c
feat: es查询支持 scroll 和 searchafter #1010158081122617468
shamcleren Apr 9, 2025
2d5053e
feat: relation api 支持并发查询 #1010158081122852024
shamcleren Apr 9, 2025
af7ceff
feat: relation api 支持并发查询 #1010158081122852024
shamcleren Apr 9, 2025
25225e9
Merge branch 'feat/es_query_with_scroll_and_search_after/#10101580811…
shamcleren Apr 9, 2025
04548cc
feat: relation api 支持并发查询 #1010158081122852024
shamcleren Apr 9, 2025
51e24dc
feat: relation api 支持并发查询 #1010158081122852024
shamcleren Apr 9, 2025
ac6da6b
feat: relation api 支持并发查询 #1010158081122852024
shamcleren Apr 9, 2025
6b6d520
Merge branch 'master' into feat/relation_api_with_sync_pool/#10101580…
shamcleren Apr 9, 2025
21bc121
Merge remote-tracking branch 'upstream/master' into feat/log_query_fo…
shamcleren Apr 9, 2025
6cc7211
Merge branch 'feat/relation_api_with_sync_pool/#1010158081122852024' …
shamcleren Apr 9, 2025
d7733ce
Merge remote-tracking branch 'upstream/master' into feat/log_query_fo…
shamcleren Apr 9, 2025
a78152f
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 10, 2025
7a2b93a
Merge remote-tracking branch 'upstream/master' into feat/log_query_fo…
shamcleren Apr 10, 2025
1e9a098
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 10, 2025
46dbe4c
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 10, 2025
c37ff6d
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 10, 2025
e909db2
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 10, 2025
976c536
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 10, 2025
0698752
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 10, 2025
1dafbbc
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 11, 2025
4150e4c
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 11, 2025
5fcd0d0
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 11, 2025
6f73ae7
feat: 优化 tagvalues 查询接口,超过一天的使用 labelvalues 查询 #1010158081123432529
shamcleren Apr 11, 2025
6d92f2f
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 14, 2025
5a38f32
feat: 查询原始数据移除多余字段 #1010158081123461236
shamcleren Apr 14, 2025
2c41ce3
feat: 查询原始数据移除多余字段 #1010158081123461236
shamcleren Apr 14, 2025
4e5bcd0
Merge branch 'feat/query_raw_without_dimension/#1010158081123461236' …
shamcleren Apr 14, 2025
0a92a14
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 14, 2025
9a4c4da
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 14, 2025
5e1f095
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 15, 2025
4aa7758
Merge branch 'feat/tag_values_with_label_value/#1010158081123432529' …
shamcleren Apr 15, 2025
7dff4cc
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 15, 2025
bcd6740
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 15, 2025
9d76961
Merge branch 'master' of github.com:TencentBlueKing/bkmonitor-datalin…
shamcleren Apr 16, 2025
7995135
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 16, 2025
ef1aa28
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 16, 2025
910e7a9
feat: 优化 tagvalues 查询接口,超过一天的使用 labelvalues 查询 #1010158081123432529
shamcleren Apr 16, 2025
7551812
Merge branch 'feat/tag_values_with_label_value/#1010158081123432529' …
shamcleren Apr 16, 2025
a708cd9
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 16, 2025
47d1ab3
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 17, 2025
2cc8db1
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 17, 2025
bd43799
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 18, 2025
ba43ca1
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 18, 2025
062ed0e
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 18, 2025
62e35e8
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 18, 2025
91e5bad
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 18, 2025
fe314e1
Merge remote-tracking branch 'upstream/master' into feat/log_query_fo…
shamcleren Apr 18, 2025
780c481
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 21, 2025
fc219a5
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 21, 2025
129868c
Merge branch 'master' of github.com:TencentBlueKing/bkmonitor-datalin…
shamcleren Apr 22, 2025
16c57ce
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 22, 2025
53590fa
feat: 非时间聚合接口排序问题修改 #1010158081123667928
shamcleren Apr 23, 2025
92306e7
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 23, 2025
37d5566
Merge branch 'master' of github.com:TencentBlueKing/bkmonitor-datalin…
shamcleren Apr 23, 2025
0f698b9
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 24, 2025
b6124da
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 24, 2025
241a302
feat: 日志查询支持doris #1010158081121933504
shamcleren Apr 25, 2025
c2100e7
fix: 修复 es 查询 qr 为空的问题 #1010158081141941944
shamcleren Apr 30, 2025
5e7f5f1
fix: 修复 es 查询 qr 为空的问题 #1010158081141941944
shamcleren Apr 30, 2025
e107d0a
Merge branch 'fix/query_result_is_nil/#1010158081141941944' into feat…
shamcleren Apr 30, 2025
478281e
feat: es 查询支持前缀模式 #1010158081123906702
shamcleren May 8, 2025
42e997c
feat: 日志查询支持doris #1010158081121933504
shamcleren May 8, 2025
0ccce9e
feat: es 查询聚合时间适配非毫秒单位 #1010158081123913773
shamcleren May 9, 2025
e210175
Merge branch 'master' into feat/es_date_histogram_time_fixed/#1010158…
shamcleren May 9, 2025
e799941
Merge remote-tracking branch 'upstream/master' into feat/log_query_fo…
shamcleren May 9, 2025
46ad3c0
Merge branch 'feat/es_date_histogram_time_fixed/#1010158081123913773'…
shamcleren May 9, 2025
ae66f3b
Merge remote-tracking branch 'upstream/master' into feat/log_query_fo…
shamcleren May 9, 2025
3f63de0
Merge remote-tracking branch 'upstream/master' into feat/log_query_fo…
shamcleren May 9, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions pkg/unify-query/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -50,3 +50,7 @@ addlicense:
.PHONY: imports
imports: addlicense
goimports-reviser -rm-unused -set-alias -format -project-name "github.com/TencentBlueKing/bkmonitor-datalink/pkg" ./...

.PHONY: race
race:
go test -race -timeout 1m -parallel 8 ./...
44 changes: 33 additions & 11 deletions pkg/unify-query/influxdb/router_mock.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ const (
ResultTableEs = "result_table.es"
ResultTableBkBaseEs = "result_table.bk_base_es"
ResultTableBkSQL = "result_table.bk_sql"
ResultTableDoris = "result_table.doris"
)

var (
Expand Down Expand Up @@ -123,7 +124,7 @@ func MockSpaceRouter(ctx context.Context) {
)
tsdb.SetStorage("2", &tsdb.Storage{Type: consul.InfluxDBStorageType})
tsdb.SetStorage("3", &tsdb.Storage{Type: consul.ElasticsearchStorageType, Address: mock.EsUrl})
tsdb.SetStorage("4", &tsdb.Storage{Type: consul.BkSqlStorageType, Address: mock.BkSQLUrl})
tsdb.SetStorage("4", &tsdb.Storage{Type: consul.BkSqlStorageType, Address: mock.BkBaseUrl})

r := GetInfluxDBRouter()
r.clusterInfo = ir.ClusterInfo{
Expand Down Expand Up @@ -183,6 +184,9 @@ func MockSpaceRouter(ctx context.Context) {
ResultTableBkBaseEs: &ir.SpaceResultTable{
TableId: ResultTableBkBaseEs,
},
ResultTableDoris: &ir.SpaceResultTable{
TableId: ResultTableDoris,
},
},
},
ir.ResultTableDetailInfo{
Expand All @@ -195,6 +199,7 @@ func MockSpaceRouter(ctx context.Context) {
Measurement: "kubelet_info",
BcsClusterID: "BCS-K8S-00000",
MeasurementType: redis.BkSplitMeasurement,
StorageType: consul.VictoriaMetricsStorageType,
DataLabel: "kubelet_info",
},
"system.cpu_summary": &ir.ResultTableDetail{
Expand All @@ -206,6 +211,7 @@ func MockSpaceRouter(ctx context.Context) {
VmRt: "",
Fields: []string{"usage", "free"},
MeasurementType: redis.BKTraditionalMeasurement,
StorageType: consul.InfluxDBStorageType,
DataLabel: "cpu_summary",
},
"system.cpu_detail": &ir.ResultTableDetail{
Expand All @@ -214,6 +220,7 @@ func MockSpaceRouter(ctx context.Context) {
VmRt: "100147_ieod_system_cpu_detail_raw",
Fields: []string{"usage", "free"},
MeasurementType: redis.BKTraditionalMeasurement,
StorageType: consul.InfluxDBStorageType,
DataLabel: "cpu_detail",
},
"system.disk": &ir.ResultTableDetail{
Expand All @@ -222,6 +229,7 @@ func MockSpaceRouter(ctx context.Context) {
VmRt: "100147_ieod_system_disk_raw",
Fields: []string{"usage", "free"},
MeasurementType: redis.BKTraditionalMeasurement,
StorageType: consul.InfluxDBStorageType,
DataLabel: "disk",
},
ResultTableVM: &ir.ResultTableDetail{
Expand All @@ -231,6 +239,7 @@ func MockSpaceRouter(ctx context.Context) {
Fields: vmFiedls,
BcsClusterID: "BCS-K8S-00000",
MeasurementType: redis.BkSplitMeasurement,
StorageType: consul.VictoriaMetricsStorageType,
DataLabel: "vm",
},
ResultTableInfluxDB: &ir.ResultTableDetail{
Expand All @@ -243,12 +252,14 @@ func MockSpaceRouter(ctx context.Context) {
MeasurementType: redis.BkSplitMeasurement,
ClusterName: "default",
DataLabel: "influxdb",
StorageType: consul.InfluxDBStorageType,
},
ResultTableEs: &ir.ResultTableDetail{
StorageId: 3,
TableId: ResultTableEs,
DB: "es_index",
SourceType: "bkdata",
StorageId: 3,
TableId: ResultTableEs,
DB: "es_index",
SourceType: "",
StorageType: consul.ElasticsearchStorageType,
StorageClusterRecords: []ir.Record{
{
StorageID: 3,
Expand All @@ -264,14 +275,25 @@ func MockSpaceRouter(ctx context.Context) {
DataLabel: "es",
},
ResultTableBkSQL: &ir.ResultTableDetail{
StorageId: 4,
TableId: ResultTableBkSQL,
DataLabel: "bksql",
StorageId: 4,
TableId: ResultTableBkSQL,
DataLabel: "bksql",
DB: "2_bklog_bkunify_query_doris",
StorageType: consul.BkSqlStorageType,
},
ResultTableDoris: &ir.ResultTableDetail{
StorageId: 4,
TableId: ResultTableDoris,
DB: "2_bklog_bkunify_query_doris",
Measurement: "doris",
DataLabel: "bksql",
StorageType: consul.BkSqlStorageType,
},
ResultTableBkBaseEs: &ir.ResultTableDetail{
SourceType: "bkdata",
DB: "es_index",
DataLabel: "bkbase_es",
SourceType: "bkdata",
DB: "es_index",
DataLabel: "bkbase_es",
StorageType: consul.ElasticsearchStorageType,
},
}, nil,
ir.DataLabelToResultTable{
Expand Down
Loading