Repository navigation
feat(resource): 相同图片字节复用一个 handle,不再重复写字节与重复注册 - #4419
Merged
lyingbug merged 3 commits intoOct 17, 2026
Merged
Conversation
Register only deduplicates by physical location, so the same image extracted onto new paths — a logo or mascot repeated across pages — was written and registered once per occurrence. That duplicated storage, made the VLM caption the same picture several times, and put identical cards in the gallery. Give the content hash a usable identity: - 000123 adds a non-unique partial index on resources(tenant_id, content_hash) for live, hashed rows. It must not be unique: existing rows already hold duplicate payloads. - ResourceRepository.GetActiveByContentHash and ResourceCatalog.ReuseByContentHash resolve a live persistent resource by payload hash, scoped to the tenant. - SaveBytes hashes first and reuses a matching handle before writing; a miss or a lookup error falls through to the previous write-and-register path, so the lookup can never make a save fail. Sharing is limited to images outside temporary scratch space: for an upload or an artifact the original name is part of the result, while extracted document images are named from a generated UUID. The image test is byte-first (http.DetectContentType) with an extension fallback for formats such as SVG that have no sniffable signature. Temporary rows are never reused, so scratch cleanup cannot delete bytes a long-lived reference still serves.
lyingbug
approved these changes
Oct 17, 2026
lyingbug
left a comment
Collaborator
There was a problem hiding this comment.
已复核此前评审结论及最新提交,相关 CI 检查全部通过,同意合入。
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4418
问题
Register只按物理路径去重(locationHash),而抽取出来的图片每次都是新文件名,于是同一份字节被抽取几次就注册几个 handle:字节存多份、VLM 对同一张图重复 caption(描述还不一致)、画廊出现多张相同卡片。resources.content_hash从资源目录引入起就在写,但全仓没有一处按它查询;SaveBytes又是"先写字节、后算 hash",想在写之前查也没有机会。上游 #4121 的正文已经把这个缺口写成边界("this was not changed to content-addressed deduplication in this PR")。改动
migrations/versioned/000123_resource_content_hash_index.up/down.sql:(tenant_id, content_hash)部分索引,WHERE deleted_at IS NULL AND content_hash <> ''。故意非唯一:存量行里已经有重复 payload,唯一索引建不上;部分谓词把已删行与无 hash 的行(历史注册、临时/未哈希写入)挡在索引外。ResourceRepository新增GetActiveByContentHash,实现按tenant_id + content_hash + state=active + deleted_at IS NULL取最旧一条(重复哈希时结果稳定)。ReuseByContentHash,命中返回types.BuildResourcePath(existing.Handle);只复用lifecycle=persistent的行 —— 临时行是 scratch,把长期引用别名到它上面,scratch 清理会删掉仍在服务的字节。SaveBytes:先算 sha256;仅!temp && isImagePayload(name, data)时查复用;命中不写字节直接返回已有 ref;未命中或查询报错 fail-open 走原来的写入+注册路径。isImagePayload以字节为主(http.DetectContentType前缀image/),扩展名只作兜底(复用同一个resourceKind规则,覆盖 SVG 这类没有嗅探签名的矢量资源 —— 恰好是跨页重复的那类)。测试
internal/application/service/file/resource_content_dedup_test.go:同字节存两次 → inner 只写一次、两次同 ref;不同字节 → 两次写入;temp=true不去重;非图片同字节不去重(守住名字边界);catalog 查询报错仍写入成功(fail-open);isImagePayload的名字边界。internal/application/repository/resource_content_hash_test.go:命中 / 未命中 / 软删行不算命中 / 重复哈希取最旧。internal/application/service/resource_test.go:catalog 级复用、租户隔离、临时不复用。反向验证(复用逻辑未接 → 必红 → 接上 → 绿 → 改回恒不命中 → 再红):
接上后新用例全 PASS;
go vet全仓通过;go test ./internal/application/service/file/ ./internal/application/repository/ ./internal/database/ -count=1三包ok,另跑了./internal/application/service/与./internal/router/(改过它们的测试 stub)也无回归;以 CI 同版本golangci-lint --new-from-merge-base=e07a19021跑过 0 issues;gofmt -l/gofumpt -l对改动文件为空。边界
image_key去重的既有逻辑自然只剩一张卡;同一图片收到多条 caption 的进一步收敛留作后续。temp=true不查索引;查询失败被吞掉直接走原路径(不写日志、绝不影响写入)。OriginalName—— 这是"只限图片"射程内的已知取舍。tenant_id:仓储查询与 catalog 入口都以 tenant 为条件。migrations/versioned是 PostgreSQL 专用;本机未起 PG,因此该 up/down 未在真实 PG 上执行过(已确认目录可被 golang-migrate 自动拾取、最新版本号自动纳入,sqlite 基线不受影响)。