ilogtail 和 loongconnector 都无法上传日志到 SLS #2088
-
|
求助，已无力吐槽， flushers:
- Type: flusher_stdout
# FileName: test.log
OnlyStdout: true
- Type: flusher_sls
Region: cn-beijing
Endpoint: cn-beijing.log.aliyuncs.com
Project: beijingtest
Logstore: test其中 flusher_stdout 有内容， ram开了sls full access，用 java sdk 测试了 ak 是能往logstore写入数据的 minikube环境，配置个简单的日志上传到sls就卡住了😓 |
Beta Was this translation helpful? Give feedback.
Answered by
Takuka0311
Feb 12, 2025
Replies: 1 comment 6 replies
-
检查网络和cr都没问题。ilogtail和loongconnector 源码也都扫了一遍，没看出为啥写不了sls。 日志ilogtail 2.0.7 日志: 明文ak，日志可读性极差，显示 push 到 reader 了实际压根没写入 /src/core/event_handler/EventHandler.cpp:480 log reader creation succeed:pushed into the corresponding reader queue project:beijingtest logstore:testconfig:plain_text_log log reader queue name:/root/log/app.log log reader queue length:1 file device:65025 file inode:11796676 file size:336 reading start position:0loongconnector 0.2.0 日志: 用的 example_config 中的 k8s daemonset 配置，读nginx日志 ed
[2025-02-11 15:32:02.299554] [info] [30] /src/core/monitor/AlarmManager.cpp:133 alarm gathering:started
[2025-02-11 15:32:02.299610] [info] [31] /src/core/monitor/SelfMonitorServer.cpp:41 self-monitor:started
[2025-02-11 15:32:02.299634] [info] [11] /src/core/monitor/Monitor.cpp:589 machine cpu cores:12
[2025-02-11 15:32:02.299698] [info] [32] /src/core/monitor/Monitor.cpp:145 profiling:started
[2025-02-11 15:32:02.299824] [info] [11] /src/core/go_pipeline/LogtailPlugin.cpp:332 valid plugin adapter version, version:300
[2025-02-11 15:32:02.351208] [info] [11] /src/core/go_pipeline/LogtailPlugin.cpp:463 Go plugin system init:succeeded
[2025-02-11 15:32:02.351327] [info] [46] /src/core/runner/ProcessorRunner.cpp:90 processor runner:started thread no:0
[2025-02-11 15:32:02.351491] [info] [11] /src/core/config/watcher/PipelineConfigWatcher.cpp:294 new config found and passed topology check:prepare to build pipeline config:inner-self-monitor-metric-pipeline
[2025-02-11 15:32:02.351769] [info] [11] /src/core/config/watcher/PipelineConfigWatcher.cpp:294 new config found and passed topology check:prepare to build pipeline config:nginx_stdout
[2025-02-11 15:32:02.351807] [info] [11] /src/core/config/watcher/PipelineConfigWatcher.cpp:69 config files scan done:got updates, begin to update pipelines added:2 modified:0 removed:0
[2025-02-11 15:32:02.351925] [info] [11] /src/core/pipeline/PipelineManager.cpp:128 pipeline building for new config succeeded:begin to start pipeline config:inner-self-monitor-metric-pipeline
[2025-02-11 15:32:02.351934] [info] [11] /src/core/monitor/SelfMonitorServer.cpp:79 self-monitor metrics pipeline:updated
[2025-02-11 15:32:02.351937] [info] [11] /src/core/pipeline/Pipeline.cpp:360 pipeline start:succeeded config:inner-self-monitor-metric-pipeline
[2025-02-11 15:32:02.351986] [info] [11] /src/core/plugin/flusher/sls/SLSClientManager.cpp:318 add data server endpoint, region:cn-beijing endpoint:http://cn-beijing.log.aliyuncs.com isDefault:yes isProxy:false #endpoint:1
[2025-02-11 15:32:02.354559] [info] [11] /src/core/pipeline/PipelineManager.cpp:128 pipeline building for new config succeeded:begin to start pipeline config:nginx_stdout
[2025-02-11 15:32:02.354588] [info] [11] /src/core/sls_control/SLSControl.cpp:103 user agent:ilogtail/0.2.0 (Linux; 6.10.14; aarch64) ip/192.168.49.2 env/K8S-Daemonset
[2025-02-11 15:32:02.354679] [info] [47] /src/core/plugin/flusher/sls/SLSClientManager.cpp:388 sls endpoint probe:started
[2025-02-11 15:32:02.354726] [info] [48] /src/core/plugin/flusher/sls/DiskBufferWriter.cpp:144 disk buffer sender:started
[2025-02-11 15:32:02.354733] [info] [11] /src/core/go_pipeline/LogtailPlugin.cpp:174 Go pipelines start:starts config name:nginx_stdout/1
[2025-02-11 15:32:02.354745] [info] [49] /src/core/plugin/flusher/sls/DiskBufferWriter.cpp:119 disk buffer writer:started
[2025-02-11 15:32:02.354758] [info] [11] /src/core/go_pipeline/LogtailPlugin.cpp:179 Go pipelines start:succeeded config name:nginx_stdout/1
[2025-02-11 15:32:02.354763] [info] [11] /src/core/pipeline/Pipeline.cpp:360 pipeline start:succeeded config:nginx_stdout
[2025-02-11 15:33:02.392291] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:22 writeMetricsDeleteTotal:0 metricsSnapshotTotal:22
[2025-02-11 15:34:02.485856] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:22 writeMetricsDeleteTotal:0 metricsSnapshotTotal:22
[2025-02-11 15:35:02.571056] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:22 writeMetricsDeleteTotal:0 metricsSnapshotTotal:22
[2025-02-11 15:36:02.669346] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:22 writeMetricsDeleteTotal:0 metricsSnapshotTotal:22
[2025-02-11 15:37:02.754859] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:22 writeMetricsDeleteTotal:0 metricsSnapshotTotal:22
[2025-02-11 15:38:02.858897] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:22 writeMetricsDeleteTotal:0 metricsSnapshotTotal:22
[2025-02-11 15:39:02.970432] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:22 writeMetricsDeleteTotal:0 metricsSnapshotTotal:22
[2025-02-11 15:40:02.111379] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:22 writeMetricsDeleteTotal:0 metricsSnapshotTotal:22
[2025-02-11 15:41:02.214495] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:22 writeMetricsDeleteTotal:0 metricsSnapshotTotal:22
[2025-02-11 15:42:02.301484] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:22 writeMetricsDeleteTotal:0 metricsSnapshotTotal:22
[2025-02-11 15:43:02.402654] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:22 writeMetricsDeleteTotal:0 metricsSnapshotTotal:22
[2025-02-11 15:44:02.484371] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:22 writeMetricsDeleteTotal:0 metricsSnapshotTotal:22
[2025-02-11 15:45:02.574514] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:22 writeMetricsDeleteTotal:0 metricsSnapshotTotal:22
[2025-02-11 15:46:02.685189] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:22 writeMetricsDeleteTotal:0 metricsSnapshotTotal:22
[2025-02-11 15:47:02.768733] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:22 writeMetricsDeleteTotal:0 metricsSnapshotTotal:22
[2025-02-11 15:47:07.793542] [info] [11] /src/core/file_server/EventDispatcher.cpp:983 checkpoint dump:starts
[2025-02-11 15:47:07.793605] [info] [11] /src/core/file_server/FileServer.cpp:81 file server pause:starts
[2025-02-11 15:47:07.793730] [info] [11] /src/core/file_server/polling/PollingDirFile.cpp:93 polling discovery pause:starts
[2025-02-11 15:47:07.793743] [info] [11] /src/core/file_server/polling/PollingDirFile.cpp:96 polling discovery pause:succeeded
[2025-02-11 15:47:07.793799] [info] [11] /src/core/file_server/polling/PollingModify.cpp:79 polling modify pause:starts
[2025-02-11 15:47:07.793808] [info] [11] /src/core/file_server/polling/PollingModify.cpp:82 polling modify pause:succeeded
[2025-02-11 15:47:07.793824] [info] [11] /src/core/file_server/event_handler/LogInput.cpp:119 input event handle daemon pause:starts
[2025-02-11 15:47:07.793832] [info] [11] /src/core/file_server/event_handler/LogInput.cpp:122 input event handle daemon pause:succeeded
[2025-02-11 15:47:07.793840] [info] [11] /src/core/file_server/FileServer.cpp:94 file server pause:succeededcost:0ms
[2025-02-11 15:47:07.793858] [info] [11] /src/core/file_server/EventDispatcher.cpp:966 save log reader status:succeeded
[2025-02-11 15:47:07.794319] [info] [11] /src/core/file_server/FileServer.cpp:105 file server resume:starts
[2025-02-11 15:47:07.794343] [info] [11] /src/core/file_server/FileServer.cpp:107 watch dirs:succeeded
[2025-02-11 15:47:07.794353] [info] [11] /src/core/file_server/event_handler/LogInput.cpp:103 event handle daemon resume:starts
[2025-02-11 15:47:07.794356] [info] [11] /src/core/file_server/event_handler/LogInput.cpp:106 event handle daemon resume:succeeded
[2025-02-11 15:47:07.794358] [info] [11] /src/core/file_server/polling/PollingModify.cpp:72 polling modify resume:starts
[2025-02-11 15:47:07.794371] [info] [11] /src/core/file_server/polling/PollingModify.cpp:75 polling modify resume:succeeded
[2025-02-11 15:47:07.794373] [info] [11] /src/core/file_server/polling/PollingDirFile.cpp:100 polling discovery resume:starts
[2025-02-11 15:47:07.794375] [info] [11] /src/core/file_server/polling/PollingDirFile.cpp:103 polling discovery resume:succeeded
[2025-02-11 15:47:07.794376] [info] [11] /src/core/file_server/FileServer.cpp:116 file server resume:succeeded
[2025-02-11 15:47:07.794378] [info] [11] /src/core/file_server/EventDispatcher.cpp:994 checkpoint dump:succeeded
[2025-02-11 15:48:02.875596] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:23 writeMetricsDeleteTotal:0 metricsSnapshotTotal:23
[2025-02-11 15:49:02.969533] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:23 writeMetricsDeleteTotal:0 metricsSnapshotTotal:23
[2025-02-11 15:50:02.057727] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:23 writeMetricsDeleteTotal:0 metricsSnapshotTotal:23
[2025-02-11 15:51:02.142518] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:23 writeMetricsDeleteTotal:0 metricsSnapshotTotal:23
[2025-02-11 15:52:02.228556] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:23 writeMetricsDeleteTotal:0 metricsSnapshotTotal:23
[2025-02-11 15:53:02.279924] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:23 writeMetricsDeleteTotal:0 metricsSnapshotTotal:23
[2025-02-11 15:54:02.361991] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:23 writeMetricsDeleteTotal:0 metricsSnapshotTotal:23
[2025-02-11 15:55:02.420219] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:23 writeMetricsDeleteTotal:0 metricsSnapshotTotal:23
[2025-02-11 15:56:02.480319] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:23 writeMetricsDeleteTotal:0 metricsSnapshotTotal:23
[2025-02-11 15:57:02.569091] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:23 writeMetricsDeleteTotal:0 metricsSnapshotTotal:23
[2025-02-11 15:58:02.642088] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:23 writeMetricsDeleteTotal:0 metricsSnapshotTotal:23
[2025-02-11 15:59:02.721212] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:23 writeMetricsDeleteTotal:0 metricsSnapshotTotal:23
[2025-02-11 16:00:02.814998] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:23 writeMetricsDeleteTotal:0 metricsSnapshotTotal:23
[2025-02-11 16:01:02.900207] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:23 writeMetricsDeleteTotal:0 metricsSnapshotTotal:23
[2025-02-11 16:02:02.983095] [info] [31] /src/core/monitor/MetricManager.cpp:310 writeMetricsTotal:23 writeMetricsDeleteTotal:0 metricsSnapshotTotal:23
[2025-02-11 16:02:21.973786] [info] [11] /src/core/file_server/EventDispatcher.cpp:983 checkpoint dump:starts
[2025-02-11 16:02:21.973915] [info] [11] /src/core/file_server/FileServer.cpp:81 file server pause:starts
[2025-02-11 16:02:21.974007] [info] [11] /src/core/file_server/polling/PollingDirFile.cpp:93 polling discovery pause:starts
[2025-02-11 16:02:21.974011] [info] [11] /src/core/file_server/polling/PollingDirFile.cpp:96 polling discovery pause:succeeded
[2025-02-11 16:02:21.974040] [info] [11] /src/core/file_server/polling/PollingModify.cpp:79 polling modify pause:starts
[2025-02-11 16:02:21.974057] [info] [11] /src/core/file_server/polling/PollingModify.cpp:82 polling modify pause:succeeded
[2025-02-11 16:02:21.974104] [info] [11] /src/core/file_server/event_handler/LogInput.cpp:119 input event handle daemon pause:starts
[2025-02-11 16:02:21.974117] [info] [11] /src/core/file_server/event_handler/LogInput.cpp:122 input event handle daemon pause:succeeded
[2025-02-11 16:02:21.974127] [info] [11] /src/core/file_server/FileServer.cpp:94 file server pause:succeededcost:1ms
[2025-02-11 16:02:21.974137] [info] [11] /src/core/file_server/EventDispatcher.cpp:966 save log reader status:succeeded
[2025-02-11 16:02:21.974936] [info] [11] /src/core/file_server/FileServer.cpp:105 file server resume:starts
[2025-02-11 16:02:21.974974] [info] [11] /src/core/file_server/FileServer.cpp:107 watch dirs:succeeded
[2025-02-11 16:02:21.974979] [info] [11] /src/core/file_server/event_handler/LogInput.cpp:103 event handle daemon resume:starts
[2025-02-11 16:02:21.974994] [info] [11] /src/core/file_server/event_handler/LogInput.cpp:106 event handle daemon resume:succeeded
[2025-02-11 16:02:21.974998] [info] [11] /src/core/file_server/polling/PollingModify.cpp:72 polling modify resume:starts
[2025-02-11 16:02:21.975002] [info] [11] /src/core/file_server/polling/PollingModify.cpp:75 polling modify resume:succeeded
[2025-02-11 16:02:21.975010] [info] [11] /src/core/file_server/polling/PollingDirFile.cpp:100 polling discovery resume:starts
[2025-02-11 16:02:21.975013] [info] [11] /src/core/file_server/polling/PollingDirFile.cpp:103 polling discovery resume:succeeded
[2025-02-11 16:02:21.975015] [info] [11] /src/core/file_server/FileServer.cpp:116 file server resume:succeeded
[2025-02-11 16:02:21.975018] [info] [11] /src/core/file_server/EventDispatcher.cpp:994 checkpoint dump:succeeded之前无ai一个多小时配置了fluentd + opensearch，以为已经够难用了... |
Beta Was this translation helpful? Give feedback.
6 replies
-
|
关于ilogtail 2.0.7：logtail只采集增量数据，这条日志是采集配置加载成功后，成功创建了reader（用于读取文件的对象）并push到内部的reader queue，并不是文件的内容被读取了。 关于发送sls，您提到“在 env 里配置的 ak id/secret，严格遵循示例”，请问是参考的哪个文档呢？可以提供下配置的方式、样例吗 |
Beta Was this translation helpful? Give feedback.
-
你好，ilogtail 参考的 ilogtail-sidecar-file-to-sls.yaml 这个配置。明白 push 到 reader queue 这个不代表读取，但没有和读取相关的日志，所以只截取了 reader queue 的日志。 |
Beta Was this translation helpful? Give feedback.
-
|
测试发现把 stdout flusher 给注释掉，就可以投递sls了，配置 stdout 后一直无法投递 flushers:
- Type: flusher_stdout
# FileName: test.log
OnlyStdout: true
- Type: flusher_sls
Region: cn-beijing
Endpoint: cn-beijing.log.aliyuncs.com
Project: a
Logstore: b
|
Beta Was this translation helpful? Give feedback.
-
|
这个是一个已知bug，已经修复了，尚未出版。 修复PR：#2015 |
Beta Was this translation helpful? Give feedback.
Answer selected by
qidian99
-
|
好的，loongcollector 0.2.0 无法写入 sls 的问题，和这个有关么，测试 flusher_sls + 开启/关闭 stdout，无法写入 logstore: 参考的 loongcollector-daemonset.yaml 中的nginx + daemonset 配置 # Copyright 2022 loongcollector Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: v1
kind: ConfigMap
metadata:
name: loongcollector-user-cm
namespace: loongcollector
data:
nginx_stdout.yaml: |
enable: true
inputs:
- Type: service_docker_stdout
Stderr: false
Stdout: true # only collect stdout
IncludeK8sLabel:
# kubectl get pods --selector=app=nginx
app: nginx # choose containers with this label
processors:
- Type: processor_regex # structure log
SourceKey: content
Regex: '([\d\.:]+) - (\S+) \[(\S+) \S+\] \"(\S+) (\S+) ([^\\"]+)\" (\d+) (\d+) \"([^\\"]*)\" \"([^\\"]*)\" \"([^\\"]*)\"'
Keys:
- remote_addr
- remote_user
- time_local
- method
- url
- protocol
- status
- body_bytes_sent
- http_referer
- http_user_agent
- http_x_forwarded_for
flushers:
- Type: flusher_sls
Region: cn-beijing
Endpoint: cn-beijing.log.aliyuncs.com
Project: a
Logstore: bnginx 日志，通过开启 stdout 获取: |
Beta Was this translation helpful? Give feedback.
-
|
loongcollector在配置了多个flusher的情况下无法写入sls确实与这个bug有关。 |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
这个是一个已知bug，已经修复了，尚未出版。
修复PR：#2015