build(dbgpt-ext): relax spacy pin to >=3.8 for Python 3.13 support (#3006) - #3185
build(dbgpt-ext): relax spacy pin to >=3.8 for Python 3.13 support (#3006)#3185yyyCode wants to merge 1 commit into
Conversation
…osphoros-ai#3006) spacy 3.7 predates Python 3.13 and ships no compatible wheels, so installing the `rag` extra fails on 3.13. spacy 3.8+ is compiled with Cython 3 and provides prebuilt wheels for 3.13. - Relax the `rag` extra pin from `spacy==3.7` to `spacy>=3.8,<4.0`; the upper bound guards against a future major release with API changes. - Add a SpacyTextSplitter smoke test that skips gracefully when the optional `rag` extra (spacy) is not installed, guarding the public API used by the splitter (spacy.load / nlp(text).sents). Closes eosphoros-ai#3006 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
📝 WalkthroughPurpose and implementationThis change removes the Python 3.13 compatibility bottleneck caused by Affected packages and configuration
RisksThe broader spaCy version range may expose behavior differences between spaCy 3.8 minor releases. The No security or performance risks are introduced by the described changes. Tests and verificationExisting coverage includes the new Recommended targeted verification: pytest packages/dbgpt-core/src/dbgpt/rag/text_splitter/tests/test_splitters.pyInstall the WalkthroughThe RAG extra now supports spaCy 3.8 through 3.x. A conditional smoke test validates ChangesRAG spaCy support
Estimated code review effort: 2 (Simple) | ~10 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 201be4dd-0207-406b-9274-d9c125153a05
📒 Files selected for processing (2)
packages/dbgpt-core/src/dbgpt/rag/text_splitter/tests/test_splitters.pypackages/dbgpt-ext/pyproject.toml
📜 Review details
🧰 Additional context used
📓 Path-based instructions (4)
**/pyproject.toml
⚙️ CodeRabbit configuration file
**/pyproject.toml: - 本仓库使用 uv workspace 和 Hatch 进行源码开发、依赖维护、锁定和构建。不要使用
pip 替代这些仓库工作流。pip 和 uv pip 仍可用于从 PyPI 安装已发布的软件包。
- 检查 Python >=3.10 支持、package version、workspace source、entry point、
wheel 和 sdist 内容以及 package 依赖方向。- 将依赖添加到实际使用它的最底层 package。将数据库、模型和 GPU 重型依赖放入
命名清晰的 optional extra。- 依赖变更必须更新 uv.lock，并保持基础安装及相关 extra 的 import 和 build 行为。
Files:
packages/dbgpt-ext/pyproject.toml
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Use Python 3.10 or newer for project development.
Files:
packages/dbgpt-core/src/dbgpt/rag/text_splitter/tests/test_splitters.py
packages/dbgpt-core/src/dbgpt/**/*.py
⚙️ CodeRabbit configuration file
packages/dbgpt-core/src/dbgpt/**/*.py: 这是以 dbgpt 名义发布的核心库。
- 严格审查公共 API、构造函数参数、返回类型、Pydantic 字段、序列化格式和异常语义的向后兼容性。
- 不得引入或扩大 core 对 dbgpt_ext、dbgpt_serve、dbgpt_app、dbgpt_client 或
dbgpt_sandbox 的反向依赖。只报告当前 diff 新增或扩大的依赖违规问题。- 对于 AWEL 变更，应同时审查同步、异步和流式执行路径。检查 DAG 关系、
ContextVar 传播、背压、顺序、结束信号、异常传播和取消处理。- 不得在 async 函数中直接执行阻塞式数据库、文件、网络或模型操作。检查连接、
task、thread、generator 和临时资源的清理。- 公共行为变更必须提供邻近的回归测试，并覆盖相关的正常、错误、空输入、并发、
取消或流式场景。
Files:
packages/dbgpt-core/src/dbgpt/rag/text_splitter/tests/test_splitters.py
**/{tests/**/*.py,test_*.py,*_test.py}
⚙️ CodeRabbit configuration file
**/{tests/**/*.py,test_*.py,*_test.py}: 检查 Python 测试、fixture 和测试辅助代码是否提供了有意义的回归覆盖。
- 应断言可观察行为和对外相关契约，而不是只断言 mock 调用次数或实现细节。将 mock
保持在真实 I/O 或进程边界，使被测行为本身仍会执行。- 对于 bug 修复，聚焦的回归测试应在旧行为上失败。对于安全修复，在能够安全测试时，
应包含一个具体的绕过方式或恶意输入场景。- 当这些场景与变更的生产路径相关时，覆盖错误、空输入、边界、清理，以及异步取消
或 timeout 行为；不要要求与变更无关的穷尽式覆盖。- 保持单元测试确定且隔离，不依赖真实网络、外部模型、数据库、GPU、Docker、
wall-clock time 或开发者机器状态。确实需要这些资源的测试应位于 integration test
中或明确标记为 integration test，说明其前置条件，并清理创建的资源。- 如果可以使用 event、mock clock、同步原语或有界不变量验证行为，应避免使用 sleep
和脆弱的性能阈值。- 对于聚焦验证，建议运行 uv run pytest 并指定相关测试路径。不要假设 make test 会运行
dbgpt_app、dbgpt_serve、dbgpt_ext、dbgpt_client 或 dbgpt_sandbox 的测试；其当前
target 运行的是 dbgpt package 测试。不要将 make fmt-check 描述为只读命令，因为
当前 target 会调用 ruff check --fix。
Files:
packages/dbgpt-core/src/dbgpt/rag/text_splitter/tests/test_splitters.py
🪛 Ruff (0.16.1)
packages/dbgpt-core/src/dbgpt/rag/text_splitter/tests/test_splitters.py
[warning] 84-84: Do not catch blind exception: Exception
(BLE001)
🔇 Additional comments (1)
packages/dbgpt-core/src/dbgpt/rag/text_splitter/tests/test_splitters.py (1)
1-2: LGTM!
| try: | ||
| splitter = SpacyTextSplitter(pipeline="en_core_web_sm", chunk_size=1000) | ||
| except Exception as e: # pragma: no cover - depends on model download | ||
| pytest.skip(f"spacy pipeline unavailable: {e}") |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== files =="
git ls-files | rg 'test_splitters\.py|text_splitter\.py|requirements|pyproject|setup\.py' || true
echo "== locate relevant snippets =="
rg -n "SpacyTextSplitter|spacy|download|en_core_web_sm|pytest.skip" packages/dbgpt-core/src/dbgpt/rag packages/dbgpt-core -g '*.py' | head -200
echo "== inspect test section =="
cat -n packages/dbgpt-core/src/dbgpt/rag/text_splitter/tests/test_splitters.py | sed -n '60,95p'
echo "== inspect splitter constructor =="
cat -n packages/dbgpt-core/src/dbgpt/rag/text_splitter/text_splitter.py | sed -n '330,390p'
echo "== dependency declarations =="
for f in pyproject.toml packages/dbgpt-core/pyproject.toml; do
if [ -f "$f" ]; then
echo "--- $f ---"
rg -n "spacy|model|spaCy|packages" "$f" | head -100 || true
fi
doneRepository: eosphoros-ai/DB-GPT
Length of output: 14876
Keep the Spacy smoke test offline before construction.
SpacyTextSplitter.__init__ calls spacy.cli.download(pipeline) before the constructor returns when the model is missing, so SpacyTextSplitter(pipeline="en_core_web_sm", ...) can start a real network download before pytest.skip. Check spacy.util.is_package("en_core_web_sm") or otherwise confirm model availability before constructing the splitter, or move this test to an explicit model-dependent integration test.
🧰 Tools
🪛 Ruff (0.16.1)
[warning] 84-84: Do not catch blind exception: Exception
(BLE001)
Source: Path instructions
| except Exception as e: # pragma: no cover - depends on model download | ||
| pytest.skip(f"spacy pipeline unavailable: {e}") |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Files matching test_splitters.py:"
fd -a 'test_splitters\.py$' . || true
echo "Relevant lines:"
for f in $(fd 'test_splitters\.py$' .); do
echo "--- $f"
sed -n '70,100p' "$f" | cat -n
done
echo "Search SpacyTextSplitter tests:"
rg -n "SpacyTextSplitter|spacy|BLE001|skip\(f" packages/dbgpt-core/src/dbgpt packages -g '*.py' | head -n 120 || true
echo "Ruff config references:"
(fd 'pyproject\.toml|ruff\.toml|\.ruff\.toml$' . | xargs -r sed -n '1,220p' | rg -n "ruff|select|ignore|BLE001|flake8-blind-except|flake8-blind-except" -C 3 || true)Repository: eosphoros-ai/DB-GPT
Length of output: 7028
🌐 Web query:
Ruff BLE001 blanket exception handling except Exception flake8-blind-except
💡 Result:
The Ruff rule BLE001 (blind-except) is designed to identify and flag overly broad exception handling, specifically except BaseException and except Exception clauses [1][2]. This rule is inspired by the flake8-blind-except plugin [1][3]. Why this is flagged: Catching these broad exception types is discouraged because it can unintentionally trap system-level exceptions like KeyboardInterrupt or SystemExit, which are often necessary to allow a program to exit gracefully [1][2]. Exceptions to the rule: Ruff will not flag these clauses in certain scenarios where the exception is handled safely or intentionally: - If the exception is re-raised within the except block [1][2]. - If the exception is logged using common logging methods (e.g., logging.exception or logging.critical with exc_info=True) [1][4][2]. Distinction from bare except: BLE001 specifically targets except BaseException and except Exception [1][5]. It is distinct from E722 (bare-except), which flags except: without any specified exception type [6][7]. For more information, see the official Ruff documentation for rule BLE001 [1].
Citations:
- 1: https://docs.astral.sh/ruff/rules/blind-except/
- 2: https://github.com/astral-sh/ruff/blob/main/crates%2Fruff_linter%2Fsrc%2Frules%2Fflake8_blind_except%2Frules%2Fblind_except.rs
- 3: https://docs.astral.sh/ruff/rules/
- 4: [
flake8-blind-except] ChangeBLE001to permitlogging.critical(..., exc_info=True). astral-sh/ruff#19520 - 5: Document that BLE001 supports both BaseException and Exception astral-sh/ruff#12788
- 6: https://pypi.org/project/flake8-blind-except/
- 7: Merge rules checking the same violation astral-sh/ruff#2714
Let unknown SpacyTextSplitter construction failures fail the test.
The current except Exception wraps any missing model, spacy.load incompatibility, and SpacyTextSplitter regression into pytest.skip, which can hide real failures from CI. Narrow the skip to the missing-model condition and let unexpected exceptions propagate instead.
🧰 Tools
🪛 Ruff (0.16.1)
[warning] 84-84: Do not catch blind exception: Exception
(BLE001)
Sources: Path instructions, Linters/SAST tools
| text = "This is the first sentence. Here is the second one. And a third." | ||
| output = splitter.split_text(text) | ||
| assert output, "SpacyTextSplitter returned no chunks" | ||
| assert "first sentence" in " ".join(output) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate file and tests =="
git ls-files | rg 'test_splitters\.py|splitter|text_splitter' | head -200
echo
echo "== relevant test section =="
sed -n '1,150p' packages/dbgpt-core/src/dbgpt/rag/text_splitter/tests/test_splitters.py
echo
echo "== locate SpacyTextSplitter implementation =="
rg -n "class SpacyTextSplitter|def _merge_splits|chunk_size|merge_splits" packages/dbgpt-core/src/dbgpt/rag text 2>/dev/null | head -200Repository: eosphoros-ai/DB-GPT
Length of output: 11735
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== SpacyTextSplitter implementation relevant lines =="
sed -n '351,388p' packages/dbgpt-core/src/dbgpt/rag/text_splitter/text_splitter.py
sed -n '452,590p' packages/dbgpt-core/src/dbgpt/rag/text_splitter/text_splitter.py
sed -n '690,715p' packages/dbgpt-core/src/dbgpt/rag/text_splitter/text_splitter.py
echo
echo "== merge_splits implementation for SpacyTextSplitter (around 697-747) =="
sed -n '697,748p' packages/dbgpt-core/src/dbgpt/rag/text_splitter/text_splitter.py
echo
echo "== deterministic behavior for sample text under SpacyTextSplitter semantics =="
python3 - <<'PY'
samples = [
"This is the first sentence. Here is the second one. And a third.",
"A. B. C. " + "x" * 200 + ". B. C.",
]
separator = "\n\n"
overlap = 0
chunk_size = 1000
for text in samples:
sent_text = text
merged_text = []
buffer = []
current = []
total = 0
for s in sent_text.splitlines(keepends=False):
s = s.strip()
# actual sent splitting path from code:
nlp = type('Fake', (), {"sentences": sent_text.split(". ")})()
# emulate relevant TextSplitter.merge_chunks_1 implementation with len(s) and separator joining.
nlp = type('Fake', (), {})()
nlp.sentences = iter((s for s in sent_text.split(". ") if s.strip()))
current = []
buffer = []
total = 0
for s in nlp.sentences:
s = s.strip()
if s:
current.append(s)
print("len(sentences):", len(current))
print("sentences:", current)
print("len(separator.join(current)):", len(separator.join(current)))
print("single_chunk_with_current_chunk_size:", len(separator.join(current)) <= chunk_size)
PYRepository: eosphoros-ai/DB-GPT
Length of output: 10699
Assert sentence boundaries, not only non-empty output.
With chunk_size=1000, the sample text is merged into one chunk, so this test cannot detect missing sentence splitting. Add a smaller chunk_size and assert multiple chunks with expected sentence content.
Source: Path instructions
| # spacy 3.8+ is compiled with Cython 3 and ships wheels for Python 3.13. | ||
| # Pin below 4.0 to avoid picking up a future major release with API changes. | ||
| "spacy>=3.8,<4.0", |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate files =="
git ls-files | rg '(^|/)(pyproject\.toml|uv\.lock)$|test_splitters\.py$' | sed -n '1,120p'
echo
echo "== target pyproject =="
sed -n '1,120p' packages/dbgpt-ext/pyproject.toml 2>/dev/null || true
echo
echo "== related test references =="
fd -a test_splitters.py . | while read -r f; do
echo "--- $f"
sed -n '60,85p' "$f"
done
echo
echo "== search spaCy references =="
rg -n 'spacy|spaCy|3\.8\.7|3\.8\.6|3\.8\.' pyproject.toml packages -S | sed -n '1,220p'Repository: eosphoros-ai/DB-GPT
Length of output: 50376
🌐 Web query:
spaCy release 3.8.7 Python 3.13 support release 3.8.6 removed
💡 Result:
spaCy version 3.8.7 was released to provide official support for Python 3.13 [1][2]. This support required compiling spaCy with Cython 3 and restructuring parts of the codebase, specifically moving factory function implementations to spacy.pipeline.factories and catalog registrations to spacy.registrations to manage import-time behavior changes associated with Python 3.13 [1][2]. Regarding spaCy version 3.8.6, it was not removed from existence but rather marked as a release that explicitly removed Python 3.13 compatibility [2][3]. The release notes for v3.8.6 specifically describe its purpose as to "Restore wheels, remove Python 3.13 compatibility" and to restore wheel support for ARM platforms [3]. Version 3.8.7 subsequently reintroduced and finalized the support for Python 3.13 [1][2].
Citations:
- 1: https://github.com/explosion/spaCy/releases/tag/release-v3.8.7
- 2: https://github.com/explosion/spaCy/releases
- 3: https://github.com/explosion/spaCy/releases/tag/release-v3.8.6
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== git status short =="
git status --short
echo
echo "== diff stat =="
git diff --stat || true
git diff --cached --stat || true
echo
echo "== spaCy references (focused) =="
rg -n 'spacy|spaCy|3\.8\.7|3\.8\.6|3\.8\.' packages pyproject.toml uv.lock --glob '!**/package-lock.json' --glob '!**/*.js' --glob '!**/*.jsx' --glob '!**/*.ts' --glob '!**/*.tsx' --glob '!**/*.md' --glob '!**/*.json' -SRepository: eosphoros-ai/DB-GPT
Length of output: 21140
Raise the spaCy lower bound to 3.8.7.
spacy>=3.8 still permits releases before Python 3.13 support, such as 3.8.6. Pin the floor at 3.8.7 so the constraint matches the compatibility objective, and update the version note in packages/dbgpt-core/src/dbgpt/rag/text_splitter/tests/test_splitters.py accordingly. Commit the regenerated uv.lock.
Source: Path instructions
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
lockfile="$(fd --hidden --type f '^uv\.lock$' . | head -n1)"
test -n "$lockfile"
uv lock --check
rg -n -C 2 'name = "spacy"|version = ' "$lockfile"Repository: eosphoros-ai/DB-GPT
Length of output: 197
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf 'Repository files of interest:\n'
git ls-files | rg '(^|/)(uv\.lock|pyproject\.toml|uv\.)$' || true
printf '\nWorkspace pyproject relevant dependency snippets:\n'
rg -n -C 3 'spacy|extra|extras' pyproject.toml packages/dbgpt-ext/pyproject.toml packages/dbgpt-ext 2>/dev/null || true
printf '\nTop-level lockfile package entries around spacy:\n'
if [ -f uv.lock ]; then
rg -n -C 8 'name = "spacy"|package name = "spacy"|spacy>=' uv.lock
else
echo "uv.lock not found at repository root"
fi
printf '\nPackage-level lockfile package entries around spacy:\n'
if [ -f packages/dbgpt-ext/uv.lock ]; then
rg -n -C 8 'name = "spacy"|package name = "spacy"|spacy' packages/dbgpt-ext/uv.lock
else
echo "packages/dbgpt-ext/uv.lock not found"
fiRepository: eosphoros-ai/DB-GPT
Length of output: 50377
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
import re
lock = Path("uv.lock").read_text(encoding="utf-8")
# Extract spaCy package block and the optional extra graph blocks.
blocks = []
current = None
for i, line in enumerate(lock.splitlines(), 1):
if re.match(r'^\[\[package\]\]$', line):
if current:
blocks.append(current)
current = {"name": "UNKNOWN", "version": "", "lines": [], "start": i}
if current:
if line.startswith("name = "):
current["name"] = line.split("=", 1)[1].strip().strip('"')
elif line.startswith("version = "):
current["version"] = line.split("=", 1)[1].strip().strip('"')
if current and line.startswith("[[package]]"):
pass
if current:
current["lines"].append((i, line))
spa = next((b for b in blocks if b["name"] == "spacy"), None)
print("spaCy package found:", bool(spa))
if spa:
print("spaCy version:", spa["version"])
print("spaCy line range:", f"{spa['start']}-" + str(spa['lines'][-1][0]))
for ln, text in spa["lines"]:
if ln <= spa["start"] + 30 or ln >= spa["lines"][-1][0] - 20:
print(f"{ln}: {text}")
for section in ("rag", "extras"):
print(f"\n[{section}] sections:")
found = False
for start, line in enumerate(lock.splitlines(), 1):
if line == f"[options.{section}]":
found = True
print(start, line)
for ln, text in enumerate(lock.splitlines()[start:start+70], start):
if text.strip().startswith("packages") or "spacy" in text.lower():
print(f"{ln}: {text}")
if text.strip() and text.strip().startswith("[") and not text.strip().startswith("[["):
assert ln > start+1
break
break
if not found:
print(f"missing {section}")
# Validate that spaCy version in lock satisfies ">=3.8,<4.0" using semantic ints.
if spa:
v = re.split(r"[._-]", spa["version"].split("+")[0].split("-")[0])
try:
v = tuple(int(part) for part in v)
except Exception as e:
v = None
print("\nParsed spaCy version tuple:", v)
if v:
print("satisfies >=3.8,<4.0:", ((3, 8) <= v < (4, 0)))
PYRepository: eosphoros-ai/DB-GPT
Length of output: 6445
Update uv.lock for the spaCy constraint.
packages/dbgpt-ext/pyproject.toml now requires spacy>=3.8,<4.0, but uv.lock still pins spacy==3.7.0, which does not satisfy the new rag extra constraint. Run the uv workspace lock update workflow and update uv.lock accordingly; do not use pip.
Source: Path instructions
|
Thanks for the contribution! The Python 3.13 direction is right, but there's a compatibility concern worth confirming: Starting with 3.8.2, spaCy is built against the numpy 2.0 ABI (their release notes state numpy 2 is binary-incompatible with numpy 1), while dbgpt-core still pins For Python 3.11/3.12 users who already depend on dbgpt-core and thus have numpy pinned to 1.x, this combination may cause Could you verify |
|
Thanks for the careful review, @chen-alan — that's a valid concern to check. I verified the numpy 1.x + spaCy 3.8 combination on a real Python 3.11 environment: Result: on Python 3.11 with numpy pinned to 1.26.4, One thing worth flagging that surfaced during testing: To keep this PR focused and low-risk, I've scoped it to the spaCy bump only — it fixes the |
2 participants
Description
`packages/dbgpt-ext/pyproject.toml` pins `spacy==3.7` in the `rag` extra. spaCy 3.7 predates Python 3.13 and ships no compatible prebuilt wheels, so `pip install "dbgpt-ext[rag]"` fails on Python 3.13 (falls back to a source build that typically fails without a C toolchain).
spaCy 3.8+ is compiled with Cython 3 and provides wheels for Python 3.13 (installed 3.8.15 in verification), so relaxing the pin fixes the install without any code change.
Fixes #3006
Changes
The only in-repo spaCy usage is `SpacyTextSplitter`, whose public API is stable across 3.7 → 3.8 (the bump is compilation-level), so no functional code change is required.
How Has This Been Tested?
Verified on Python 3.13.14, Windows 11:
Note: Python 3.14 wheel support for spaCy's dependencies (blis etc.) is still in progress upstream (explosion/spaCy#13885); this PR targets Python 3.13 as requested in the issue.
Snapshots
N/A (dependency + test change).
Checklist