Skip to content

[Bug] JSONReader default chunk=False is ignored and large JSON objects are split into invalid fragments #8679

Description

@bogdancherniy11-sudo

Summary

JSONReader says its default is chunk=False, but a plain JSONReader() instance actually gets chunk=True from the base Reader.

That changes the default behavior for large JSON values: each top-level JSON object can be split as plain text chunks, so the returned document content is no longer valid JSON.

Affected path

libs/agno/agno/knowledge/reader/json_reader.py

JSONReader declares:

chunk: bool = False

but its constructor does not pass that default into Reader.__init__(), whose default is chunk=True.

Repro

import json
from pathlib import Path
from tempfile import TemporaryDirectory

from agno.knowledge.reader.json_reader import JSONReader

with TemporaryDirectory() as td:
    path = Path(td) / "large.json"
    path.write_text(json.dumps([
        {"id": 1, "text": "alpha " * 1200},
        {"id": 2, "text": "beta " * 1200},
    ]), encoding="utf-8")

    reader = JSONReader()
    docs = reader.read(path)

    print("instance_chunk", reader.chunk)
    print("doc_count", len(docs))
    print("chunk_metadata", [doc.meta_data.get("chunk") for doc in docs])

    try:
        json.loads(docs[0].content)
        print("first_doc_json_valid", True)
    except json.JSONDecodeError:
        print("first_doc_json_valid", False)

Current result:

instance_chunk True
doc_count 4
chunk_metadata [1, 2, 1, 2]
first_doc_json_valid False

Expected result:

instance_chunk False
doc_count 2
chunk_metadata [None, None]

The two returned documents should still contain the two original top-level JSON objects.

Why this matters

For JSON files, each top-level object is already a natural document. Splitting a large object with a text chunker can produce invalid JSON fragments and makes downstream parsing or metadata-sensitive ingestion unreliable.

This also affects Knowledge.insert(path="...json"), because the non-URL path flow picks JSONReader from the file suffix and calls reader.read(path).

Suggested fix

Pass the intended default into the base reader constructor:

class JSONReader(Reader):
    chunk: bool = False

    def __init__(self, chunking_strategy: Optional[ChunkingStrategy] = None, chunk: bool = False, **kwargs):
        if chunking_strategy is None:
            chunk_size = kwargs.get("chunk_size", 5000)
            chunking_strategy = FixedSizeChunking(chunk_size=chunk_size)
        super().__init__(chunk=chunk, chunking_strategy=chunking_strategy, **kwargs)

This keeps explicit JSONReader(chunk=True) working, while making the default match the class declaration.

Regression test

def test_json_reader_default_does_not_chunk_large_objects(tmp_path):
    test_data = [{"id": 1, "text": "alpha " * 1200}, {"id": 2, "text": "beta " * 1200}]
    json_path = tmp_path / "large.json"
    json_path.write_text(json.dumps(test_data), encoding="utf-8")

    reader = JSONReader()
    documents = reader.read(json_path)

    assert reader.chunk is False
    assert len(documents) == 2
    assert [json.loads(doc.content)["id"] for doc in documents] == [1, 2]
    assert all("chunk" not in doc.meta_data for doc in documents)

Checks run locally

Against current main at 695ff9f9ca28b8985636acd815d64c13655e064f:

/mnt/ntfs-p2/codex-agno-test/venv/bin/python -m pytest -q tests/unit/reader/test_json_reader.py

Result:

23 passed in 0.48s

Also checked:

/mnt/ntfs-p2/codex-agno-test/venv/bin/python -m py_compile agno/knowledge/reader/json_reader.py
git diff --check -- libs/agno/agno/knowledge/reader/json_reader.py libs/agno/tests/unit/reader/test_json_reader.py

Both passed.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions