feat(ragfs): unify CacheRuntime and Redis-backed CacheFS/QueueFS

[txfang]

Description

This PR consolidates the RAGFS cache infrastructure into a shared CacheRuntime inside the ragfs crate. It adds a built-in Fred-based RedisProvider, routes CacheFS and QueueFS cache operations through the same runtime, and makes ragfs-python the single Python binding and wheel build entry point.

The default OpenViking behavior remains unchanged. CacheRuntime is initialized only when CacheFS or QueueFS selects backend=cache. DynamicProvider and distributed PathLock support remain explicitly deferred.

Human Involvement

• A human participated in the implementation or review loop
• This PR was generated entirely by AI agents without human participation in the loop

Related Issue

4352

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Refactoring (no functional changes)
• Performance improvement
• Test update

Changes Made

• Added a shared asynchronous CacheRuntime and synchronous SyncCacheRuntimeFacade inside the ragfs crate. The runtime exposes the Redis-style operations required by CacheFS and QueueFS, including String, batch String, counters, Set queries, non-blocking List commands, registered scripts, health checks, and lifecycle operations.
• Added a Fred-based RedisProvider with Standalone, Cluster, and Sentinel support, including ACL authentication, Sentinel authentication, TLS options, command/connect timeouts, reconnect handling, topology updates, and Redis Cluster slot validation.
• Kept one asynchronous Redis client path. Synchronous callers use SyncCacheRuntimeFacade and RuntimeExecutor to call the same provider instead of maintaining a second synchronous connection pool.
• Updated CachedFileSystem to use the shared CacheRuntime while preserving read-through, write-through, invalidation, subtree generation, inflight miss coalescing, backend fallback, policy, and metrics behavior.
• Replaced the independent QueueFS Redis runtime path with CacheQueueStorage -> CacheRuntime -> RedisProvider. QueueFS key layout, Lua scripts, heartbeat, stale-message recovery, and result parsing remain owned by QueueFS in cache_protocol.rs.
• Preserved legacy QueueFS Redis configuration by normalizing backend=redis to backend=cache with the global RedisProvider. Redis failures in QueueFS return an error and do not switch to SQLite at runtime.
• Added a top-level Provider-neutral cache configuration using cache.provider and cache.params. CacheFS and QueueFS select only backend=local or backend=cache; no global cache.enabled flag is required.
• Added compatibility migration for the legacy storage.agfs.cache configuration, legacy QueueFS Redis settings, singleton mode, and QueueFS Redis key prefixes. Conflicting legacy and canonical settings fail during configuration validation.
• Consolidated Python bindings into ragfs-python and removed the separate ragfs-python-native crate. CacheRuntime configuration is passed through the existing RAGFS binding and wheel path.
• Removed bundled Mooncake, Yuanrong, Yuanrong sys, and standalone Redis Provider crates. This release includes only the built-in RedisProvider; configuring DynamicProvider returns an explicit unsupported-provider error.
• Updated English and Chinese configuration/cache guides and added unit, integration, compatibility, topology, and smoke-test coverage.

Testing

• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have tested this on the following platforms:
  • Linux
  • macOS
  • Windows

Automated verification:

• cargo test --locked -p ragfs --features cache
  • 458 RAGFS unit tests passed.
  • 13 CacheRuntime integration tests passed.
  • 37 CachedFileSystem integration tests passed.
  • 7 RedisRuntime integration tests passed.
  • DynamicProvider unsupported-path and Rust documentation tests passed.
• cargo check --locked -p ragfs-python passed.
• python3 -m pytest -q -o addopts='' tests/misc/test_config_validation.py
  • 74 configuration and compatibility tests passed.
• Python compile checks and git diff --check passed.

Docker and Redis validation:

Redis 场景	CacheFS	QueueFS	结果
Standalone	miss 回源、命中读、read_dir、写后读、rename、delete、remove_all、指标	创建、stat/read_dir、enqueue、peek、dequeue、ack、clear、remove、shutdown	通过
Sentinel	同上，并确认数据写入 Sentinel 解析出的 master	同上	通过
Cluster	同上，generation key 分布在不同节点且未发生 CROSSSLOT	Lua 和 QueueFS key 使用同一 hash slot，未发生 CROSSSLOT	通过

• Redis Standalone:
  • Verified top-level cache configuration and RAGFS binding initialization.
  • Verified CacheFS write, cache hit read, update, read-after-write, delete, backend fallback, and invalidation behavior.
  • Verified QueueFS create, enqueue, size, peek, dequeue, ack, clear, nested queue, and remove operations.
• CacheFS workload:
  • Exercised OpenViking read, ls, tree, and grep operations with CacheFS using RedisProvider.
• Redis Sentinel:
  • Tested one primary, one replica, and three Sentinel nodes.
  • Verified master discovery, full QueueFS operations, primary failover, reconnect, and FIFO visibility before and after failover.
• Redis Cluster:
  • Tested three primary and three replica nodes with all 16,384 slots covered.
  • Verified QueueFS scripts stay in one hash slot, primary failover, reconnect, and FIFO ordering.
• Multiple OpenViking instances:
  • Verified cross-instance QueueFS visibility and FIFO ordering.
  • Verified processing-message recovery after the owning instance exited and its heartbeat expired.

Checklist

• My code follows the project's coding style
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• Any dependent changes have been merged and published

Screenshots (if applicable)

Not applicable.

Additional Notes

• CacheFS continues to treat the backend filesystem as the source of truth. Cache misses and Redis failures fall back to the backend filesystem.
• QueueFS stores authoritative queue state in the selected backend. When configured with backend=cache, Redis failures are returned to the caller instead of silently switching to SQLite.
• PathLock code and behavior are not changed by this PR. Filesystem and memory PathLock implementations remain the default.
• DynamicProvider, external .so loading, and capability negotiation are not implemented in this PR. They are planned as follow-up work.
• Default configurations do not initialize CacheRuntime or create Redis connections.
• Two pre-existing QueueFS issues remain outside this PR:
  1. NamedQueue.enqueue() returns the binding write result instead of the generated message ID. This affects all QueueFS backends.
  2. If an enqueue operation times out after Redis commits it, the current protocol cannot determine the final outcome because it has no idempotency marker or outcome query.

[txfang]

主要问题
[P1] QueueFS 将 Redis 故障伪装成“队列不存在/空列表”
queue_exists 捕获错误后返回 false，list_queues 返回空列表。上层 stat/read_dir 因而返回正常但错误的数据，违背“QueueFS 不可用时直接报错”的要求。
建议将两者改为 Result，让 stat/read_dir 传播缓存错误。

[P1] read_from_replica=true 会破坏 QueueFS 强一致语义
RedisClient 会把 SISMEMBER/SMEMBERS/LLEN/LRANGE/MGET 发往副本；而 QueueFS 使用这些操作判断队列状态。复制延迟可能让刚创建或写入的队列被报告为不存在。
建议 QueueFS 强制读主节点，或在 QueueFS 使用缓存时拒绝 read_from_replica=true。

[P1] 已发布文档中的旧 CacheFS 配置升级后无法启动
旧文档推荐 key_prefix: "ragfs-cache"。迁移代码在 open_viking_config.py 原样复制该字段，但新 binding 在 lib.rs 要求它为空。
应在迁移阶段消费或转换旧 key_prefix，而不是将其传给新 Provider 后拒绝。

[P1] CacheFS 在 Redis 启动时不可用的情况下不能 fail-open
build_configured_stack 必须先成功连接 Redis，否则整个 RAGFS 初始化失败。这不满足 issue 中“CacheFS 在 Redis 不可用时继续使用 Backend FileSystem”的要求。
应区分 CacheFS-only 与 QueueFS 场景：前者允许禁用缓存继续启动，后者仍应失败。

[P1] rediss:// 的 TLS 语义被丢弃
parse_endpoint 接受 rediss://，但只返回 host/port；Fred 配置 仅依据默认关闭的 tls_enabled 决定是否启用 TLS。
应保留 URL scheme，并要求 rediss:// 必须启用 TLS，同时拒绝混合 scheme。

另有一个兼容性风险：旧 QueueFS 默认连接/命令超时均为 3000ms；迁移仅复制显式字段，省略时会采用新 Provider 的 1000ms/20ms 默认值，可能导致升级后大量超时。

[baojun-zhang]

建议 ov.example.conf 上面增加 cache 的示例配置

[txfang]

建议 ov.example.conf 上面增加 cache 的示例配置

已增加

[baojun-zhang]

感谢这次较完整的 CacheRuntime 重构，整体方向清晰，CacheFS 与 QueueFS 共用运行时也减少了重复实现。我重点检查了运行时生命周期、Redis 拓扑能力、QueueFS 消息恢复和新旧配置兼容，目前有 5 处希望合并前再确认并完善的问题：Redis 初始化失败后的任务清理、replica reads 的实现与描述一致性、config_path 对 canonical cache 配置的支持、QueueFS 关闭时的 heartbeat 清理，以及一组旧配置迁移兼容场景。具体触发路径和建议已分别标在对应代码处；辛苦修复并补充回归测试后再合并。

[txfang]

主要问题 [P1] QueueFS 将 Redis 故障伪装成“队列不存在/空列表” queue_exists 捕获错误后返回 false，list_queues 返回空列表。上层 stat/read_dir 因而返回正常但错误的数据，违背“QueueFS 不可用时直接报错”的要求。 建议将两者改为 Result，让 stat/read_dir 传播缓存错误。

[P1] read_from_replica=true 会破坏 QueueFS 强一致语义 RedisClient 会把 SISMEMBER/SMEMBERS/LLEN/LRANGE/MGET 发往副本；而 QueueFS 使用这些操作判断队列状态。复制延迟可能让刚创建或写入的队列被报告为不存在。 建议 QueueFS 强制读主节点，或在 QueueFS 使用缓存时拒绝 read_from_replica=true。

[P1] 已发布文档中的旧 CacheFS 配置升级后无法启动 旧文档推荐 key_prefix: "ragfs-cache"。迁移代码在 open_viking_config.py 原样复制该字段，但新 binding 在 lib.rs 要求它为空。 应在迁移阶段消费或转换旧 key_prefix，而不是将其传给新 Provider 后拒绝。

[P1] CacheFS 在 Redis 启动时不可用的情况下不能 fail-open build_configured_stack 必须先成功连接 Redis，否则整个 RAGFS 初始化失败。这不满足 issue 中“CacheFS 在 Redis 不可用时继续使用 Backend FileSystem”的要求。 应区分 CacheFS-only 与 QueueFS 场景：前者允许禁用缓存继续启动，后者仍应失败。

[P1] rediss:// 的 TLS 语义被丢弃 parse_endpoint 接受 rediss://，但只返回 host/port；Fred 配置 仅依据默认关闭的 tls_enabled 决定是否启用 TLS。 应保留 URL scheme，并要求 rediss:// 必须启用 TLS，同时拒绝混合 scheme。

另有一个兼容性风险：旧 QueueFS 默认连接/命令超时均为 3000ms；迁移仅复制显式字段，省略时会采用新 Provider 的 1000ms/20ms 默认值，可能导致升级后大量超时。

已修复