|
| 1 | +--- |
| 2 | +title: "PythonCodeSplitter" |
| 3 | +id: pythoncodesplitter |
| 4 | +slug: "/pythoncodesplitter" |
| 5 | +description: "Split Python source documents into syntax-aware chunks using Python's AST, with metadata for line ranges, classes, decorators, and docstrings." |
| 6 | +--- |
| 7 | + |
| 8 | +# PythonCodeSplitter |
| 9 | + |
| 10 | +`PythonCodeSplitter` splits Python source code documents into syntax-aware chunks. It is designed for Python files and keeps code units such as imports, functions, classes, and methods together where possible. |
| 11 | + |
| 12 | +<div className="key-value-table"> |
| 13 | + |
| 14 | +| | | |
| 15 | +| --- | --- | |
| 16 | +| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx), before [Embedders](../embedders.mdx) or [`DocumentWriter`](../writers/documentwriter.mdx) | |
| 17 | +| **Mandatory run variables** | `documents`: A list of Python source code documents | |
| 18 | +| **Output variables** | `documents`: A list of Python source code documents split into syntax-aware chunks | |
| 19 | +| **API reference** | [PreProcessors](/reference/preprocessors-api) | |
| 20 | +| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/python_code_splitter.py | |
| 21 | +| **Package name** | `haystack-ai` | |
| 22 | + |
| 23 | +</div> |
| 24 | + |
| 25 | +## Overview |
| 26 | + |
| 27 | +`PythonCodeSplitter` expects each input document's `content` to be valid Python source code. It parses the source with Python's `ast` module and creates ordered split units for: |
| 28 | + |
| 29 | +- Module docstrings |
| 30 | +- Consecutive import blocks |
| 31 | +- Top-level functions |
| 32 | +- Class headers |
| 33 | +- Methods and nested classes |
| 34 | +- Remaining top-level statements |
| 35 | + |
| 36 | +The splitter merges these units in source order toward `max_effective_lines`. Effective lines are calculated from character length with `ceil(len(source) / expected_chars_per_line)`, so long lines count as more than one line. |
| 37 | + |
| 38 | +Functions and methods are kept whole by the primary AST split. If one syntactic unit is larger than `oversized_factor * max_effective_lines`, the splitter falls back to a line-based secondary split using [`DocumentSplitter`](documentsplitter.mdx). This oversized fallback is the only case where chunks can overlap; the primary AST split does not add overlap. |
| 39 | + |
| 40 | +By default, `preserve_class_definition=True`. When a chunk contains class members without the original class header, the splitter prefixes the bare class signature so the chunk still carries the class context. |
| 41 | + |
| 42 | +If `strip_docstrings=True`, function, method, and class docstrings are removed from chunk content and stored in `meta["docstrings"]`. Module docstrings stay in the chunk content because they are their own top-level unit. |
| 43 | + |
| 44 | +### Per-chunk metadata |
| 45 | + |
| 46 | +Each output document carries the metadata below. All fields from the parent document's `meta` (except `split_id`) are also propagated. |
| 47 | + |
| 48 | +| Field | Description | |
| 49 | +| --- | --- | |
| 50 | +| `source_id` | ID of the originating document | |
| 51 | +| `split_id` | Sequential index of this chunk within its source document | |
| 52 | +| `start_line` | First line of the chunk in the original source (1-indexed). Oversized secondary chunks keep the originating unit's range. | |
| 53 | +| `end_line` | Last line of the chunk in the original source (1-indexed). Oversized secondary chunks keep the originating unit's range. | |
| 54 | +| `unit_kinds` | List of syntactic unit kinds included in this chunk, such as `imports`, `function`, `class_header`, or `method` | |
| 55 | +| `include_classes` | *(when applicable)* Ordered list of class names whose members appear in this chunk | |
| 56 | +| `decorators` | *(when applicable)* Ordered list of decorator strings found on included functions, methods, or classes | |
| 57 | +| `docstrings` | *(when `strip_docstrings=True`)* List of stripped docstring strings in source order | |
| 58 | +| `secondary_split` | `True` if this chunk was produced by the oversized fallback splitter | |
| 59 | +| `secondary_split_index` | Index of this piece within the secondary split sequence | |
| 60 | +| `secondary_split_total` | Total number of pieces produced by the secondary split | |
| 61 | + |
| 62 | +Documents with `None` content raise `ValueError`, documents with non-string content raise `TypeError`, and invalid Python source raises `SyntaxError`. Empty documents are skipped. |
| 63 | + |
| 64 | +## Configuration |
| 65 | + |
| 66 | +| Parameter | Type | Default | Description | |
| 67 | +| --- | --- | --- | --- | |
| 68 | +| `min_effective_lines` | `int` | `20` | Minimum effective lines per chunk. While a chunk is below this value, the splitter keeps merging in the next unit. | |
| 69 | +| `max_effective_lines` | `int` | `100` | Target effective lines per chunk. Units are merged greedily toward this value. | |
| 70 | +| `expected_chars_per_line` | `int` | `45` | Character count used to estimate effective lines via `ceil(len(source) / expected_chars_per_line)`. | |
| 71 | +| `oversized_factor` | `int` | `3` | Multiplier that triggers secondary line-based splitting for oversized syntactic units. | |
| 72 | +| `strip_docstrings` | `bool` | `False` | Moves function, method, and class docstrings from content into `meta["docstrings"]`. | |
| 73 | +| `preserve_class_definition` | `bool` | `True` | Prefixes class signatures on chunks that contain class members without the class header. | |
| 74 | +| `secondary_split_overlap` | `int` | `5` | Line overlap used only by the oversized secondary split. | |
| 75 | +| `secondary_split_length` | `int \| None` | `None` | Line length for the oversized secondary split. Defaults to `max_effective_lines` when `None`. | |
| 76 | + |
| 77 | +## Usage |
| 78 | + |
| 79 | +### On its own |
| 80 | + |
| 81 | +```python |
| 82 | +import textwrap |
| 83 | + |
| 84 | +from haystack import Document |
| 85 | +from haystack.components.preprocessors import PythonCodeSplitter |
| 86 | + |
| 87 | +source = textwrap.dedent( |
| 88 | + ''' |
| 89 | + """Math utilities.""" |
| 90 | + from math import pi |
| 91 | +
|
| 92 | +
|
| 93 | + class Circle: |
| 94 | + """A circle.""" |
| 95 | +
|
| 96 | + def __init__(self, radius: float) -> None: |
| 97 | + self.radius = radius |
| 98 | +
|
| 99 | + def area(self) -> float: |
| 100 | + return pi * self.radius * self.radius |
| 101 | + ''' |
| 102 | +).lstrip() |
| 103 | + |
| 104 | +splitter = PythonCodeSplitter( |
| 105 | + min_effective_lines=4, |
| 106 | + max_effective_lines=12, |
| 107 | + strip_docstrings=True, |
| 108 | +) |
| 109 | + |
| 110 | +result = splitter.run( |
| 111 | + documents=[Document(content=source, meta={"file_name": "geometry.py"})], |
| 112 | +) |
| 113 | + |
| 114 | +for chunk in result["documents"]: |
| 115 | + print(chunk.meta["start_line"], chunk.meta["end_line"], chunk.meta.get("include_classes")) |
| 116 | +``` |
| 117 | + |
| 118 | +### With docstring stripping for RAG |
| 119 | + |
| 120 | +Set `strip_docstrings=True` when docstrings are verbose. The docstring text is moved out of the chunk content into `meta["docstrings"]`, keeping the stored chunk compact. Pass `meta_fields_to_embed=["docstrings"]` to your embedder so the docstring text still influences retrieval even though it is no longer in the chunk content. |
| 121 | + |
| 122 | +```python |
| 123 | +from haystack import Document |
| 124 | +from haystack.components.preprocessors import PythonCodeSplitter |
| 125 | + |
| 126 | +source = ''' |
| 127 | +"""Example module.""" |
| 128 | +from math import pi |
| 129 | +
|
| 130 | +
|
| 131 | +class Circle: |
| 132 | + """A circle defined by its radius.""" |
| 133 | +
|
| 134 | + def __init__(self, r: float) -> None: |
| 135 | + """Store the radius.""" |
| 136 | + self.r = r |
| 137 | +
|
| 138 | + def area(self) -> float: |
| 139 | + """Return the area of the circle.""" |
| 140 | + return pi * self.r * self.r |
| 141 | +''' |
| 142 | + |
| 143 | +splitter = PythonCodeSplitter( |
| 144 | + min_effective_lines=20, |
| 145 | + max_effective_lines=100, |
| 146 | + strip_docstrings=True, |
| 147 | +) |
| 148 | +result = splitter.run(documents=[Document(content=source, meta={"file_name": "my_module.py"})]) |
| 149 | +for chunk in result["documents"]: |
| 150 | + print(chunk.content) |
| 151 | + print(chunk.meta.get("docstrings")) |
| 152 | +``` |
| 153 | + |
| 154 | +### In a pipeline |
| 155 | + |
| 156 | +This pipeline converts Python files to documents, splits them with `PythonCodeSplitter`, and writes the chunks to an in-memory document store. |
| 157 | + |
| 158 | +```python |
| 159 | +from pathlib import Path |
| 160 | + |
| 161 | +from haystack import Pipeline |
| 162 | +from haystack.components.converters.txt import TextFileToDocument |
| 163 | +from haystack.components.preprocessors import PythonCodeSplitter |
| 164 | +from haystack.components.writers import DocumentWriter |
| 165 | +from haystack.document_stores.in_memory import InMemoryDocumentStore |
| 166 | + |
| 167 | +document_store = InMemoryDocumentStore() |
| 168 | + |
| 169 | +p = Pipeline() |
| 170 | +p.add_component("converter", TextFileToDocument()) |
| 171 | +p.add_component("splitter", PythonCodeSplitter(max_effective_lines=80)) |
| 172 | +p.add_component("writer", DocumentWriter(document_store=document_store)) |
| 173 | + |
| 174 | +p.connect("converter.documents", "splitter.documents") |
| 175 | +p.connect("splitter.documents", "writer.documents") |
| 176 | + |
| 177 | +files = list(Path("path/to/your/project").glob("**/*.py")) |
| 178 | +p.run({"converter": {"sources": files}}) |
| 179 | +``` |
0 commit comments