Skip to content

feat(resource): 相同图片字节复用一个 handle,不再重复写字节与重复注册 - #4419

Merged
lyingbug merged 3 commits into
Tencent:mainfrom
linus-liu-web:feat/resource-content-hash-dedup
Oct 17, 2026
Merged

lyingbug merged 3 commits into
Tencent:mainfrom
linus-liu-web:feat/resource-content-hash-dedup

Conversation

@linus-liu-web

Copy link
Copy Markdown
Contributor

Fixes #4418

问题

Register 只按物理路径去重(locationHash),而抽取出来的图片每次都是新文件名,于是同一份字节被抽取几次就注册几个 handle:字节存多份、VLM 对同一张图重复 caption(描述还不一致)、画廊出现多张相同卡片。

resources.content_hash 从资源目录引入起就在写,但全仓没有一处按它查询;SaveBytes 又是"先写字节、后算 hash",想在写之前查也没有机会。上游 #4121 的正文已经把这个缺口写成边界("this was not changed to content-addressed deduplication in this PR")。

改动

  • 迁移 migrations/versioned/000123_resource_content_hash_index.up/down.sql:(tenant_id, content_hash) 部分索引,WHERE deleted_at IS NULL AND content_hash <> ''。故意非唯一:存量行里已经有重复 payload,唯一索引建不上;部分谓词把已删行与无 hash 的行(历史注册、临时/未哈希写入)挡在索引外。
  • 仓储:ResourceRepository 新增 GetActiveByContentHash,实现按 tenant_id + content_hash + state=active + deleted_at IS NULL 取最旧一条(重复哈希时结果稳定)。
  • catalog:新增 ReuseByContentHash,命中返回 types.BuildResourcePath(existing.Handle);只复用 lifecycle=persistent 的行 —— 临时行是 scratch,把长期引用别名到它上面,scratch 清理会删掉仍在服务的字节。
  • SaveBytes:先算 sha256;仅 !temp && isImagePayload(name, data) 时查复用;命中不写字节直接返回已有 ref;未命中或查询报错 fail-open 走原来的写入+注册路径。
    • isImagePayload 以字节为主(http.DetectContentType 前缀 image/),扩展名只作兜底(复用同一个 resourceKind 规则,覆盖 SVG 这类没有嗅探签名的矢量资源 —— 恰好是跨页重复的那类)。

测试

  • internal/application/service/file/resource_content_dedup_test.go:同字节存两次 → inner 只写一次、两次同 ref;不同字节 → 两次写入;temp=true 不去重;非图片同字节不去重(守住名字边界);catalog 查询报错仍写入成功(fail-open);isImagePayload 的名字边界。
  • internal/application/repository/resource_content_hash_test.go:命中 / 未命中 / 软删行不算命中 / 重复哈希取最旧。
  • internal/application/service/resource_test.go:catalog 级复用、租户隔离、临时不复用。
  • 接口新增方法导致 4 个既有 fake 需要补 no-hit stub(纯编译修复,不改变既有测试行为)。

反向验证(复用逻辑未接 → 必红 → 接上 → 绿 → 改回恒不命中 → 再红):

--- FAIL: TestResourceCatalogFileServiceReusesIdenticalImageBytes (0.00s)
    resource_content_dedup_test.go:119: the second store minted a new handle instead of reusing the first payload's
FAIL	github.com/Tencent/WeKnora/internal/application/service/file	0.399s

接上后新用例全 PASS;go vet 全仓通过;go test ./internal/application/service/file/ ./internal/application/repository/ ./internal/database/ -count=1 三包 ok,另跑了 ./internal/application/service/ 与 ./internal/router/(改过它们的测试 stub)也无回归;以 CI 同版本 golangci-lint --new-from-merge-base=e07a19021 跑过 0 issues;gofmt -l / gofumpt -l 对改动文件为空。

边界

  • 只影响新写入:存量库里已有的多 handle 重复不做清理/合并(本迁移也明确允许非唯一),需要后续单独收敛。
  • caption 层未动:相同字节共用一个 handle 后,画廊按 image_key 去重的既有逻辑自然只剩一张卡;同一图片收到多条 caption 的进一步收敛留作后续。
  • temp=true 不查索引;查询失败被吞掉直接走原路径(不写日志、绝不影响写入)。
  • 用户上传/产物中的图片若字节完全相同,仍会共享 handle 并显示先注册的 OriginalName —— 这是"只限图片"射程内的已知取舍。
  • 跨租户隔离靠 tenant_id:仓储查询与 catalog 入口都以 tenant 为条件。
  • 迁移深度:migrations/versioned 是 PostgreSQL 专用;本机未起 PG,因此该 up/down 未在真实 PG 上执行过(已确认目录可被 golang-migrate 自动拾取、最新版本号自动纳入,sqlite 基线不受影响)。

linus-liu-web and others added 3 commits October 17, 2026 00:43
Register only deduplicates by physical location, so the same image
extracted onto new paths — a logo or mascot repeated across pages — was
written and registered once per occurrence. That duplicated storage, made
the VLM caption the same picture several times, and put identical cards in
the gallery.

Give the content hash a usable identity:

- 000123 adds a non-unique partial index on
  resources(tenant_id, content_hash) for live, hashed rows. It must not be
  unique: existing rows already hold duplicate payloads.
- ResourceRepository.GetActiveByContentHash and
  ResourceCatalog.ReuseByContentHash resolve a live persistent resource by
  payload hash, scoped to the tenant.
- SaveBytes hashes first and reuses a matching handle before writing;
  a miss or a lookup error falls through to the previous write-and-register
  path, so the lookup can never make a save fail.

Sharing is limited to images outside temporary scratch space: for an
upload or an artifact the original name is part of the result, while
extracted document images are named from a generated UUID. The image test
is byte-first (http.DetectContentType) with an extension fallback for
formats such as SVG that have no sniffable signature.

Temporary rows are never reused, so scratch cleanup cannot delete bytes a
long-lived reference still serves.

@lyingbug lyingbug left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已复核此前评审结论及最新提交,相关 CI 检查全部通过,同意合入。

@lyingbug
lyingbug merged commit a7d915a into Tencent:main Oct 17, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: 同一份图片字节被重复抽取成多个资源 —— 重复存储、重复 caption、图库多张相同卡片

2 participants