Skip to content

fix(sls): add structured flush failure diagnostics #130

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Monsooooon wants to merge 2 commits into
base: main
Choose a base branch
from
Open

fix(sls): add structured flush failure diagnostics #130

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d9530e6
fix(sls): add structured flush failure diagnostics
Monsooooon Jul 10, 2026
544c50b
fix(sls): classify nested fetch network failures
Monsooooon Jul 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 39 additions & 2 deletions assets/skills/references/sls-diagnostics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ SlsFlusher 按 endpoint 分别发送
↓
发送失败（3 次重试后仍失败）
→ 写入 ~/.loongsuite-pilot/sls-failed-logs/{endpoint-name}.jsonl
→ 记录 FLUSH_SEND_ALARM（含 failure_class / status_code / endpoint_host 等诊断字段）
```

关键事实：
Expand All @@ -27,6 +28,7 @@ SlsFlusher 按 endpoint 分别发送
- 失败自动重试 3 次（指数退避：1s → 2s → 4s），可重试的状态码：408、429、500、502、503、504
- 最终失败的日志持久化到 `sls-failed-logs/` 目录（不会丢失）
- 支持多 endpoint 同时发送（互不阻塞）
- `FLUSH_SEND_ALARM` 保持原 alarm_type，同时附加结构化诊断字段；`endpoint_host` 只包含安全解析后的 hostname，不包含 path/query/userinfo；`reason` 已限长并去敏

---

Expand Down Expand Up @@ -156,18 +158,52 @@ tail -5 ~/.loongsuite-pilot/sls-failed-logs/*.jsonl 2>/dev/null
{
"ts": 1716000000000, // 失败时间戳
"endpoint": "internal-sls", // endpoint 名称
"endpoint_host": "...", // 仅 hostname，不含完整 endpoint URL
"mode": "webtracking", // webtracking 或 ak
"kind": "agentActivity", // 数据类型
"project": "...", // SLS project
"logstore": "...", // SLS logstore
"failure_class": "permission_denied",
"status_code": 403,
"retryable": false,
"reason": "HTTP 403 ...", // 限长、去敏后的错误摘要
"logGroup": { ... }, // 完整的日志内容（可用于手动重放）
"error": "..." // 错误详情
"error": "..." // 与 reason 一致的安全错误摘要
}
```

文件名按 endpoint 名称分隔：`{endpoint-name}.jsonl`，不同 endpoint 的失败互不干扰。

### 4.1 常见失败原因分析

`FLUSH_SEND_ALARM` 和 failed-log 都包含以下诊断字段：

| 字段 | 含义 |
|------|------|
| `endpoint_name` / `endpoint` | SLS endpoint 配置名，也是 failed-log 文件名来源 |
| `endpoint_host` | 从 endpoint 安全解析出的 hostname，不透传完整 URL |
| `mode` | `webtracking` 或 `ak` |
| `project` / `logstore` | 目标 SLS project / logstore |
| `failure_class` | 归一化失败类型 |
| `status_code` | HTTP 或 SDK 返回的状态码；无状态码时为空 |
| `retryable` | 该错误是否属于重试类错误 |
| `reason` | 限长、去敏后的错误摘要 |

`failure_class` 常见取值：

| failure_class | 常见原因 | 建议处理 |
|---|---|---|
| `auth_failed` | 401、AK/SK 无效、签名错误 | 检查 AK/SK 是否正确、是否过期 |
| `permission_denied` | 403、RAM 权限不足 | 确认具备 SLS 写入权限 |
| `not_found` | 404、project/logstore 不存在 | 核对 project/logstore 拼写和地域 |
| `quota_throttle` | 429、quota/throttle/server busy | 检查写入限额、QPS、SLS 侧限流 |
| `network_timeout` | 408、AbortError、TimeoutError、ETIMEDOUT | 检查网络连通性、代理、防火墙 |
| `network_refused` | ECONNREFUSED、ECONNRESET、socket hang up | 检查 endpoint 可达性和网络链路 |
| `server_error` | 5xx、InternalServerError | 关注 SLS 服务状态，必要时重试观察 |
| `payload_too_large` | 413、请求体过大 | 检查单批大小和单条日志体积 |
| `bad_request` | 其他 4xx | 检查 endpoint/project/logstore 和请求格式 |
| `unknown` | 未识别错误 | 结合服务日志和 failed-log 的 `reason` 排查 |

```bash
# 提取失败原因统计
python3 -c "
Expand All @@ -178,7 +214,8 @@ for f in glob.glob('$HOME/.loongsuite-pilot/sls-failed-logs/*.jsonl'):
for line in open(f):
try:
r = json.loads(line)
errors[r.get('error','unknown')[:80]] += 1
key = f"{r.get('failure_class','unknown')} status={r.get('status_code','')}"
errors[key] += 1
except: pass
for err, cnt in errors.most_common(10):
print(f'{cnt:5d} {err}')
Expand Down
17 changes: 17 additions & 0 deletions docs/sls-output.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,23 @@ Local JSONL output can help confirm whether collection itself is working before
tail -f ~/.loongsuite-pilot/logs/output/*.jsonl
```

## Failure Diagnostics

When an SLS endpoint still fails after the built-in retries, Pilot keeps the existing `FLUSH_SEND_ALARM` alarm type and adds structured diagnostic fields:

| Field | Description |
|-------|-------------|
| `endpoint_name` | Configured endpoint name. |
| `endpoint_host` | Hostname safely parsed from the endpoint URL. It does not include path, query, userinfo, or the full endpoint string. |
| `mode` | `webtracking` or `ak`. |
| `project` / `logstore` | Target SLS project and logstore. |
| `failure_class` | Normalized class such as `auth_failed`, `permission_denied`, `not_found`, `quota_throttle`, `network_timeout`, `network_refused`, `server_error`, `bad_request`, `payload_too_large`, or `unknown`. |
| `status_code` | HTTP/SDK status code when available. |
| `retryable` | Whether the failure matches Pilot's retryable set. |
| `reason` | Redacted, length-limited failure summary. |

The same diagnostic fields are written to `~/.loongsuite-pilot/sls-failed-logs/<endpoint-name>.jsonl` together with the failed `logGroup`. `FLUSH_QUOTA_ALARM` is still emitted for 429 throttling; retry count, batch size, failed-log path, and multi-endpoint isolation are unchanged.

## Privacy Notes

SLS is a remote destination. Review [Agent Configuration](agents.md) for content capture controls and [Data Masking](masking.md) for secret masking before enabling SLS in sensitive environments.
17 changes: 17 additions & 0 deletions docs/zh-CN/sls-output.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,23 @@ ls ~/.loongsuite-pilot/sls-failed-logs/
tail -f ~/.loongsuite-pilot/logs/output/*.jsonl
```

## 失败诊断

SLS endpoint 在内置重试后仍失败时，Pilot 会保留既有 `FLUSH_SEND_ALARM` alarm_type，并附加结构化诊断字段：

| 字段 | 说明 |
|------|------|
| `endpoint_name` | 配置中的 endpoint 名称。 |
| `endpoint_host` | 从 endpoint URL 安全解析出的 hostname，不包含 path、query、userinfo 或完整 endpoint 字符串。 |
| `mode` | `webtracking` 或 `ak`。 |
| `project` / `logstore` | 目标 SLS project 和 logstore。 |
| `failure_class` | 归一化失败类型，例如 `auth_failed`、`permission_denied`、`not_found`、`quota_throttle`、`network_timeout`、`network_refused`、`server_error`、`bad_request`、`payload_too_large`、`unknown`。 |
| `status_code` | 可获得时记录 HTTP/SDK 状态码。 |
| `retryable` | 是否属于 Pilot 的可重试错误集合。 |
| `reason` | 已去敏、限长的失败摘要。 |

同样的诊断字段会写入 `~/.loongsuite-pilot/sls-failed-logs/<endpoint-name>.jsonl`，并保留失败的 `logGroup` 便于排查。429 限流仍会同时产生 `FLUSH_QUOTA_ALARM`；重试次数、batch 大小、失败落盘路径和多 endpoint 隔离语义均不变。

## 隐私说明

SLS 是远端输出目标。敏感环境中开启前，请先查看 [Agent 配置](agents.md) 的内容采集控制和 [数据脱敏](masking.md)。
Loading
Loading