Summary
Files larger than 1 MB (_MAX_FILE_BYTES in src/semble/index/files.py) are skipped during indexing without any warning, and there is no way to raise or lower the limit. The result is a silent coverage gap: queries against content that only exists in large files return plausible-looking results from other files, with no indication anything is missing.
Version
semble 0.5.5 (also present on current main).
Reproduction (synthetic)
mkdir demo && cd demo
python -c "
from pathlib import Path
body = '\n'.join(f'Paragraph {i} with unrelated filler text.' for i in range(30000))
Path('bigdoc.md').write_text('# UNIQUE.HEAD.42 widget alignment protocols\n\n' + body)
Path('small.md').write_text('# Small doc\n\nText about procedural compliance and unrelated filler.\n')
"
# bigdoc.md is ~2 MB; small.md is small
semble search "widget alignment protocols" . --content allObserved: only small.md is returned. Nothing on stderr or stdout mentions that bigdoc.md was skipped — the search looks successful, just wrong.
The existing test suite even documents this as intended behavior ("Files exceeding _MAX_FILE_BYTES are silently skipped during indexing"), so the skip itself is by design — the silence is the problem.
Root cause
src/semble/index/files.py:8 — _MAX_FILE_BYTES = 1_000_000, a hard constant; no env var, config file, or CLI flag overrides it.src/semble/index/create.py — create_index_from_path skips any file whose status is not VALID with a bare continue; TOO_LARGE is never surfaced.src/semble/cache.py — cache validation skips the same files just as silently.
Why it matters
For agents (the tool's primary audience), plausible-but-incomplete search results are worse than an error: there is no signal that trust in the results should be discounted. The failure mode is especially costly because large files are often generated or ingested documents — precisely the material an agent is asked to search.
Suggested remedies (in preference order)
- Make the cap configurable — e.g. a
SEMBLE_MAX_FILE_BYTES env var and/or a CLI flag, following the existing SEMBLE_CACHE_LOCATION / SEMBLE_CLONE_TIMEOUT / SEMBLE_MODEL_NAME idiom.
- Warn at index time — log a
WARNING listing skipped TOO_LARGE files so the gap is discoverable.
- Optionally, chunk-stream large files instead of skipping them whole.
Separately — and purely as a soft suggestion, not something this report depends on — it may be worth reconsidering whether 1 MB is the right default: embeddings are a model2vec StaticModel lookup (not transformer inference) and indexes are disk-cached with incremental rebuild, so a larger default cap looks cheap in practice. Happy to leave the default exactly where it is if the lean-build posture is preferred; configurability alone resolves the practical problem.
Summary
Files larger than 1 MB (
_MAX_FILE_BYTESinsrc/semble/index/files.py) are skipped during indexing without any warning, and there is no way to raise or lower the limit. The result is a silent coverage gap: queries against content that only exists in large files return plausible-looking results from other files, with no indication anything is missing.Version
semble 0.5.5 (also present on current
main).Reproduction (synthetic)
Observed: only
small.mdis returned. Nothing on stderr or stdout mentions thatbigdoc.mdwas skipped — the search looks successful, just wrong.The existing test suite even documents this as intended behavior ("Files exceeding _MAX_FILE_BYTES are silently skipped during indexing"), so the skip itself is by design — the silence is the problem.
Root cause
src/semble/index/files.py:8—_MAX_FILE_BYTES = 1_000_000, a hard constant; no env var, config file, or CLI flag overrides it.src/semble/index/create.py—create_index_from_pathskips any file whose status is notVALIDwith a barecontinue;TOO_LARGEis never surfaced.src/semble/cache.py— cache validation skips the same files just as silently.Why it matters
For agents (the tool's primary audience), plausible-but-incomplete search results are worse than an error: there is no signal that trust in the results should be discounted. The failure mode is especially costly because large files are often generated or ingested documents — precisely the material an agent is asked to search.
Suggested remedies (in preference order)
SEMBLE_MAX_FILE_BYTESenv var and/or a CLI flag, following the existingSEMBLE_CACHE_LOCATION/SEMBLE_CLONE_TIMEOUT/SEMBLE_MODEL_NAMEidiom.WARNINGlisting skippedTOO_LARGEfiles so the gap is discoverable.Separately — and purely as a soft suggestion, not something this report depends on — it may be worth reconsidering whether 1 MB is the right default: embeddings are a model2vec
StaticModellookup (not transformer inference) and indexes are disk-cached with incremental rebuild, so a larger default cap looks cheap in practice. Happy to leave the default exactly where it is if the lean-build posture is preferred; configurability alone resolves the practical problem.