Skip to content

feat: 增加事务化文件系统复制能力 - #4185

Open
KCHENPENGFEI wants to merge 20 commits into
volcengine:mainfrom
KCHENPENGFEI:codex/feature-ov-cp
Open

feat: 增加事务化文件系统复制能力#4185
KCHENPENGFEI wants to merge 20 commits into
volcengine:mainfrom
KCHENPENGFEI:codex/feature-ov-cp

Conversation

@KCHENPENGFEI

@KCHENPENGFEI KCHENPENGFEI commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

变更说明

新增文件与目录复制能力,覆盖 VikingFS、HTTP API 和 ov CLI。复制时同步迁移已有向量记录,避免重复解析、文件级 VLM 和 embedding;提交后异步重建受影响父目录的语义信息。

人工参与

  • 有人工参与实现或评审过程
  • 本 PR 完全由 AI Agent 生成,没有人工参与

关联事项

接口提案已经在项目 Discussion 中批准。

变更类型

  • Bug 修复(不破坏现有功能)
  • 新功能(不破坏现有功能)
  • 破坏性变更
  • 文档更新
  • 重构(无独立功能变化)
  • 性能优化
  • 测试更新

主要改动

cp 能力

新增 POST /api/v1/fs/cpov cp

  • ov cp 支持单文件复制,ov cp -r 支持目录递归复制。
  • 目标父目录必须存在,目标路径不能已存在。
  • 同步复制文件内容及对应向量,保留源数据。
  • 递归复制逐项校验权限,并排除 watch 控制文件。

RAGFS 原子发布与失败补偿

AGFS 复制已下沉到 RAGFS:

  • RAGFS 统一持有源路径、目标路径和 staging 路径锁。
  • 数据先完整复制到目标目录下的隐藏 staging 路径,成功后通过一次 rename 发布到最终目标;失败时清理 staging,避免暴露半成品目标。对象存储后端仍遵循其原生 best-effort rename 语义。
  • VikingFS 在 AGFS 发布成功后复制已有向量并改写 URI,不重新生成 embedding。
  • 向量复制失败时清理已写入的目标向量和目标 AGFS 数据;补偿失败会返回失败阶段及残留 URI,便于人工修复。
  • mv 继续采用“复制目标 → 迁移向量 → 删除源”的补偿式流程。

这里不是跨 AGFS 与 VectorDB 的强一致两阶段提交:AGFS 内部通过 staging 原子发布,跨存储一致性通过严格校验和补偿回滚保证。

父目录语义刷新

cp 完成后异步刷新目标父目录;mv 完成后异步刷新源、目标父目录。刷新消息使用 generation_trigger="content_copy"

  • 根据变更后的目录独立采样,默认最多 32 项。
  • 只读取已有文件 L2 摘要和子目录 abstract,不重新解析文件,也不生成缺失的 L2 摘要。
  • 调用目录级 VLM 生成 overview,并从新 overview 提取 abstract。
  • 只刷新直接父目录,不继续向祖先冒泡。

接口只等待消息入队,不等待语义处理完成。

文档

补充中英文 HTTP 接口、端点总览和 CLI 使用文档。

局限性

当前没有提供 AGFS 与 VectorDB 的强一致快照。如果 add-resource 已完成 AGFS 落盘,但向量仍在生成,cp 可能只复制当时已经写入的向量,导致目标向量不完整。

正常情况下,add-resource 在语义和 embedding 阶段持有目录 Tree Lock,重叠的 cp 通常会返回 409 path_busy,实际竞态窗口较短,本期暂不增加索引完成屏障或增量补齐机制。

父目录语义刷新是异步的,cp/mv 返回成功时,新的 overview 和 abstract 可能尚未生成。

测试

  • 已增加测试验证功能行为
  • 新增及相关测试在本地通过
  • 已在 macOS 验证

本次 rebase 后的定向验证:

  • 80 个相关 Python 测试通过。
  • 11 个 RAGFS copy 测试通过。
  • Rust ov_clicp 请求测试通过。
  • cargo check -p ragfs-python 通过。
  • 改动文件通过 Ruff lint、format、Python compile 和 diff 检查。
  • 已通过本地服务手工验证 HTTP 接口和 ov cp 命令。

全量 filesystem router 测试仍有一个 upstream/main 已存在的 attrs Filter DSL 用例失败,与本 PR 无关;本机 ragfs-python 单元测试链接阶段缺少 Python 符号,但 cargo check 已通过。

检查清单

  • 代码符合项目风格
  • 已完成自查
  • 已为难以理解的逻辑补充注释
  • 已补充对应文档
  • 未引入新的依赖

@KCHENPENGFEI KCHENPENGFEI changed the title feat: add transactional filesystem copy support feat: 增加事务化文件系统复制能力 Aug 21, 2026
@baojun-zhang

baojun-zhang commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

@KCHENPENGFEI 可以直接在 ragfs 的 filesystem.rs 中实现 原子化 cp 呢?

Comment thread openviking/storage/viking_fs/_ops.py
@ZaynJarvis

Copy link
Copy Markdown
Collaborator

Thanks for the substantial work on transactional copy! This branch currently has merge conflicts with main (mergeable_state: dirty). Could you rebase/resolve so review can proceed? The CLI-side design aligns well with existing conventions — main review focus will be the rollback semantics in _ops.py / vector transfer.

@baojun-zhang baojun-zhang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个 PR 的方向是给 VikingFS / API / CLI 增加事务化 cp,并在 cp/mv 后迁移或刷新语义数据。当前实现还有几处会导致数据损坏、锁保护失效或取消后绕过补偿的问题,需要先修完再合并。

典型场景:S3 后端复制 20 MiB 文件时,RAGFS 进入 raw copy 分块写目标,但 S3 write 是整对象 put,offset 不生效,最终对象只剩最后一块。另一个场景是目录 cp 发布目标后只持有目标 Exact 锁,向量迁移失败回滚时可能把其他请求并发写入目标子目录的数据一起删除。

结论:REQUEST_CHANGES。

Comment thread crates/ragfs/src/core/mountable.rs Outdated
Comment thread crates/ragfs/src/core/mountable.rs Outdated
Comment thread openviking/storage/viking_fs/_ops.py
Comment thread openviking/storage/viking_fs/_ops.py
Comment thread openviking/storage/viking_fs/_ops.py
Comment thread openviking/server/routers/filesystem.py Outdated
Comment thread openviking/storage/queuefs/semantic_dag.py Outdated
Comment thread openviking/service/fs_service.py Outdated
Comment thread openviking/storage/queuefs/semantic_dag.py
Comment thread openviking/storage/queuefs/semantic_dag.py
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 24, 2026
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 24, 2026
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 24, 2026
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 24, 2026
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 24, 2026
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 24, 2026
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 24, 2026
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 24, 2026
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 24, 2026
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 24, 2026
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 24, 2026
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 24, 2026
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 24, 2026
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 24, 2026

@baojun-zhang baojun-zhang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

亮点:本 PR 已打通 RAGFS、VikingFS、VectorDB、HTTP API 与 CLI 的事务化复制链路,并修复了 S3 大文件复制和 staging 锁清理问题。

仍存在的问题:目录 mv 回滚与取消语义、PathLock 后台 downgrade 一致性及语义刷新边界仍有 blocking 缺陷,详见 inline comments。

Comment thread openviking/storage/viking_fs/_ops.py
Comment thread openviking/storage/viking_fs/_ops.py
Comment thread crates/ragfs/src/lock/manager.rs Outdated
Comment thread crates/ragfs/src/lock/manager.rs Outdated
Comment thread openviking/storage/queuefs/semantic_dag.py Outdated
Comment thread openviking/service/fs_service.py
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 25, 2026
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 25, 2026
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 25, 2026
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 25, 2026
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 25, 2026
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 25, 2026
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 25, 2026
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 25, 2026
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 25, 2026
KCHENPENGFEI pushed a commit to KCHENPENGFEI/OpenViking that referenced this pull request Aug 25, 2026
@KCHENPENGFEI
KCHENPENGFEI force-pushed the codex/feature-ov-cp branch 2 times, most recently from d8b02a9 to c0ce194 Compare August 25, 2026 14:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

3 participants