Use encoding auto-detection for mcp tools

Author: peteretelej

docs/plans/encoding.md

@@ -3,121 +3,145 @@
 ## Current Encoding Usage
 
 ### APIs (15+ functions with encoding parameter)
+
 - `get_overview(absolute_file_path, encoding="utf-8")`
 - `search_content(..., encoding="utf-8", ...)`
 - `read_content(..., encoding="utf-8", ...)`
 - `edit_content(..., encoding="utf-8", ...)`
 
 ### Internal Functions
+
 - `src/file_access.py`: 6 functions with encoding
-- `src/search_engine.py`: `search_file()` 
+- `src/search_engine.py`: `search_file()`
 - `src/editor.py`: 2 functions with encoding
 - `src/data_models.py`: `FileOverview.encoding`
 
 ## Implementation Plan
 
 #### Phase 1: Add chardet and detection
-- [x] Add chardet dependency  
+
+- [x] Add chardet dependency
 - [x] Create `detect_file_encoding(file_path: str) -> str` in `file_access.py`
 - [x] Fallback to utf-8 on detection failure
 
 **Implementation Notes:**
+
 - Added 0.7 confidence threshold to prevent poor chardet guesses
 - Detection function handles empty files, exceptions, and low confidence cases
 - Test coverage for UTF-8, Latin-1, UTF-16, empty files, and non-existent files
 
 #### Phase 2: Update file access layer
+
 - [x] Remove encoding parameter from `read_file_content()`
-- [x] Remove encoding parameter from `read_file_lines()`  
+- [x] Remove encoding parameter from `read_file_lines()`
 - [x] Remove encoding parameter from `write_file_content()`
 - [x] Update internal strategy functions
 - [x] Add encoding cache per file session
 
 **Implementation Notes:**
+
 - `read_file_content()` and `read_file_lines()` now call `detect_file_encoding()` internally
 - `write_file_content()` preserves existing file encoding or defaults to utf-8 for new files
 - Internal strategy functions (`_read_file_memory`, etc.) still accept encoding parameter
 - **Skipped caching**: Detection is fast (~1-5ms), avoiding session management complexity
 - All existing tests pass with auto-detection
 
 #### Phase 3: Update search and edit layers
+
 - [x] Remove encoding parameter from `search_file()`
 - [x] Remove encoding parameter from `atomic_edit_file()`
 - [x] Remove encoding parameter from `_preview_edit()`
 
 **Implementation Notes:**
+
 - `search_file()` now uses `read_file_lines()` without encoding parameter
 - `atomic_edit_file()` and `replace_content()` updated to use auto-detection
 - **No `_preview_edit()` function**: Preview functionality is built into `replace_content()`
 - All edit operations now preserve original file encoding automatically
 
 #### Phase 4: Update public APIs
-- [ ] Remove encoding parameter from all 4 MCP tools
-- [ ] Update MCP schema definitions
-- [ ] Update `FileOverview.encoding` to show detected value
+
+- [x] Remove encoding parameter from all 4 MCP tools
+- [x] Update MCP schema definitions
+- [x] Update `FileOverview.encoding` to show detected value
+
+**Implementation Notes:**
+- All 4 MCP tools (`get_overview`, `search_content`, `read_content`, `edit_content`) no longer accept encoding
+- MCP schema definitions updated to remove encoding parameter from all tools
+- `FileOverview.encoding` now shows the actual detected encoding value
+- **Breaking change**: All tool signatures simplified, auto-detection is now transparent to users
 
 #### Phase 5: Documentation and testing
+
 - [ ] Update API.md, design.md, examples
 - [ ] Add encoding detection tests
 - [ ] Test various encodings and edge cases
 
 #### Phase 6: Quality assurance
+
 - [ ] Run pre-push script
 - [ ] Verify all tests pass
 - [ ] Update plan as complete
 
 ## Technical Implementation
 
 ### Detection Function
+
 ```python
 def detect_file_encoding(file_path: str) -> str:
     """Detect file encoding using chardet with utf-8 fallback."""
     try:
         with open(file_path, 'rb') as f:
             sample = f.read(65536)  # 64KB sample
-            
+
         if not sample:
             return 'utf-8'
-            
+
         result = chardet.detect(sample)
-        
+
         # Use detection only if confidence is reasonable (>= 0.7)
         if result and result.get('confidence', 0) >= 0.7:
             return result['encoding'] or 'utf-8'
         else:
             return 'utf-8'
-            
+
     except Exception:
         return 'utf-8'
 ```
 
 ## Testing
 
 ### Encodings to Test
+
 - UTF-8, UTF-16 LE/BE, Latin-1, ASCII, Windows-1252
 
-### Edge Cases  
+### Edge Cases
+
 - Empty files, binary files, small files, BOM markers
 
 ## Code Quality
 
 ### Requirements
+
 - Simple, clear, concise code
 - Avoid complexity and premature optimization
 - Self-documenting functions
 - Minimal abstraction layers
 - Direct error handling
 
 ### Implementation Guidance
-- Document scope changes and decisions made during each phase
-- Note what was skipped and why (e.g., complexity vs benefit trade-offs)
+
+- Document scope changes and decisions made during each phase as Implementation Notes below the phase
 - Keep implementation notes concise but informative for future reference
+- Note what was skipped and why (e.g., complexity vs benefit trade-offs)
+- ALWAYS use a todo when implementing a phase, the last step of the todo being to mark checklist items as complete when done.
 
 ### API After Implementation
+
 ```python
 # Simplified APIs
 get_overview("/path/to/file.py")
 search_content("/path/to/file.py", "pattern")
 read_content("/path/to/file.py", target)
 edit_content("/path/to/file.py", search_text, replace_text)
-```
+```

src/editor.py

@@ -248,6 +248,4 @@ def atomic_edit_file(
         raise EditError(f"File is not writable: {file_path}")
 
     # Perform the edit operation
-    return replace_content(
-        canonical_path, search_text, replace_text, fuzzy, preview
-    )
+    return replace_content(canonical_path, search_text, replace_text, fuzzy, preview)

src/file_access.py

@@ -27,22 +27,28 @@ def choose_file_strategy(file_size: int) -> str:
 def detect_file_encoding(file_path: str) -> str:
     """Detect file encoding using chardet with utf-8 fallback."""
     try:
-        with open(file_path, 'rb') as f:
+        with open(file_path, "rb") as f:
             sample = f.read(65536)  # 64KB sample
-            
+
         if not sample:
-            return 'utf-8'
-            
+            return "utf-8"
+
         result = chardet.detect(sample)
-        
+
+        # If chardet detects ASCII, try UTF-8 first since ASCII is subset of UTF-8
+        # and UTF-8 is more robust for files that might have non-ASCII later
+        encoding = result.get("encoding") if result else None
+        if encoding and encoding.lower() == "ascii":
+            return "utf-8"
+
         # Use detection only if confidence is reasonable (>= 0.7)
-        if result and result.get('confidence', 0) >= 0.7:
-            return result['encoding'] or 'utf-8'
+        if result and result.get("confidence", 0) >= 0.7:
+            return result["encoding"] or "utf-8"
         else:
-            return 'utf-8'
-            
+            return "utf-8"
+
     except Exception:
-        return 'utf-8'
+        return "utf-8"
 
 
 def get_file_info(file_path: str) -> dict:
@@ -208,12 +214,12 @@ def write_file_content(file_path: str, content: str) -> None:
     """Write content to file atomically using temp file + rename with auto-detected encoding."""
     canonical_path = normalize_path(file_path)
     temp_path = f"{canonical_path}.tmp"
-    
+
     # Detect encoding from existing file, or default to utf-8 for new files
     if os.path.exists(canonical_path):
         encoding = detect_file_encoding(canonical_path)
     else:
-        encoding = 'utf-8'
+        encoding = "utf-8"
 
     try:
         # Clean up any existing temp file first

src/mcp_schemas.py

@@ -32,11 +32,6 @@ def get_tool_schemas() -> list[types.Tool]:
                         "type": "string",
                         "description": "Absolute path to the file",
                     },
-                    "encoding": {
-                        "type": "string",
-                        "description": "File encoding",
-                        "default": "utf-8",
-                    },
                 },
                 "required": ["absolute_file_path"],
             },
@@ -52,11 +47,6 @@ def get_tool_schemas() -> list[types.Tool]:
                         "description": "Absolute path to the file",
                     },
                     "pattern": {"type": "string", "description": "Search pattern"},
-                    "encoding": {
-                        "type": "string",
-                        "description": "File encoding",
-                        "default": "utf-8",
-                    },
                     "max_results": {
                         "type": "integer",
                         "description": "Maximum number of results",
@@ -90,11 +80,6 @@ def get_tool_schemas() -> list[types.Tool]:
                         "oneOf": [{"type": "integer"}, {"type": "string"}],
                         "description": "Line number or search pattern",
                     },
-                    "encoding": {
-                        "type": "string",
-                        "description": "File encoding",
-                        "default": "utf-8",
-                    },
                     "mode": {
                         "type": "string",
                         "description": "Reading mode",
@@ -123,11 +108,6 @@ def get_tool_schemas() -> list[types.Tool]:
                         "type": "string",
                         "description": "Replacement text",
                     },
-                    "encoding": {
-                        "type": "string",
-                        "description": "File encoding",
-                        "default": "utf-8",
-                    },
                     "fuzzy": {
                         "type": "boolean",
                         "description": "Enable fuzzy matching",

src/search_engine.py

@@ -71,9 +71,7 @@ def combine_results(
     return sorted(combined, key=lambda x: (x.line_number, -x.similarity_score))
 
 
-def search_file(
-    file_path: str, pattern: str, fuzzy: bool = True
-) -> list[SearchMatch]:
+def search_file(file_path: str, pattern: str, fuzzy: bool = True) -> list[SearchMatch]:
     """Search file content using auto-detected encoding. Returns clear results or clear errors."""
 
     try: