AI support

[ebowman]

PR: AI Tools Integration for DEVONthink MCP Server

Summary

• ~42k LOC across 84 files
• 16 new AI tools + infrastructure
• 206 tests added; all currently passing
• Backward compatible; no API changes

AI Status Tool

• AppleScript reports tools as “available” if configured, even without valid keys
• MCP server now runs actual API tests to confirm which engines work
• Output is direct and accurate, e.g.:

✅ Working: ChatGPT | ❌ Need setup: Claude, Gemini, Mistral

• Simple usage: {} for all engines, {"engine":"X"} for one

New Tools

• check_ai_status — verifies engines with live API calls
• chat_with_knowledge_base — natural-language document queries
• extract_keywords — keyword extraction with tagging/output options
• analyze_document_themes — theme detection with confidence scores + citations
• find_similar_documents — semantic/textual similarity search
• summarize_contents — configurable document summaries

Architecture Highlights

• JXA script builder: structured generation, validation, templates, debugger
• AI abstraction layer: availability checks, diagnostics, fallback handling
• Tool framework: base classes, Zod validation, standardized error/result handling

Testing

• 206 tests (unit + workflow with mocks)
• All passing

Code Changes (8 files modified)

• Core: src/devonthink.ts, src/applescript/execute.ts
• Performance: src/tools/compare.ts
• Development: .gitignore, test infra updates, package.json deps

Compatibility

• Requires DEVONthink Pro (not Personal), Node.js/npm, and an AI service
• Supported services: OpenAI (GPT-3.5/4), Anthropic (Claude), Google (Gemini), local (GPT4All, Ollama)

Documentation

• Added CLAUDE.md with examples, config, and troubleshooting

Status

npm test      # 206 tests passing
npm run build # clean build

[dvcrn]

Thanks for the PR!

This is almost 20,000 LoC and will take a while to review 😅

on first look I see a lot of boilerplatey and hard to follow code, can we simplify this a bit?

[Copilot]

Pull Request Overview

This PR introduces comprehensive AI support for the DEVONthink MCP Server, adding 16 new AI-powered tools with robust infrastructure and extensive testing capabilities.

Key changes include:

• Implementation of AI tools for document analysis, chat, summarization, keyword extraction, and theme analysis
• Comprehensive testing framework with unit, integration, and performance tests
• Enhanced JXA script generation with validation, debugging, and error handling

Reviewed Changes

Copilot reviewed 68 out of 70 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
vitest.config.ts	Extensive test configuration with coverage thresholds, setup files, and path aliases
tests/utils/test-helpers.ts	Comprehensive testing utilities for AI tool validation, XSS prevention, and performance testing
tests/tools/ai/utils/*.test.ts	Unit tests for core AI infrastructure components
tests/tools/ai/*.test.ts	Unit tests for AI tools (extractKeywords, findSimilarDocuments, chatWithKnowledgeBase, etc.)
tests/integration/ai-tool-integration.test.ts	End-to-end integration tests for complete AI workflows
tests/mocks/devonthink.ts	Mock utilities for DEVONthink interactions in tests
src/utils/scriptDebugger.ts	Development tools for debugging and analyzing generated JXA scripts

Comments suppressed due to low confidence (1)

tests/tools/ai/utils/aiErrorHandler.test.ts:1

• [nitpick] The comment suggests error categorization is based on string matching ('Contains "ai"'), which indicates fragile error classification logic. Consider using more robust error categorization methods like error codes or structured error objects instead of substring matching.

/**

[deadmanoz]

Although this is not my repo, I would assume that the following would be pretty standard feedback you'll get from most projects if they were faced with changes: "~42k LOC across 84 files"

I think there are far too many changes here to validly review?! Even with automation/LLM

@ebowman can you break it into multiple, more atomic, changes?

[dvcrn]

Yes I'm sorry but I think this is just too huge to review, even on my bigger display. Could you split this into smaller chunks? From the looks of it Claude wrote all of it, and I don't fully trust Claude not sneaking in some stuff that shouldn't be there, or messing something up 🙏

On a quick skim:

• The robust JXA execution part could be it's own thing
• The refactor that adds the type import to each tool sounds like a separate thing as well
• baseAITool can be separate and need more documentation what it actually does, and how to use it
• analyzeDocumentThemes / chatWithKnowledgeBase look to be very specific usecases to a specific workflow and not fully relevant to the generic MCP implementation. Maybe a good point to discuss how much 'batteries included' this MCP should be, or add a way to include external tools into the MCP
• findSimilarDocuments sounds similar to devonthink:compare (built-in devonthink), this will also find similar documents and assign them a similarity score and is exposed as compare tool

[zsbenke]

Also, adding more tools will eat more into the context window. These DEVONthink MCP tools already using a healthy chunk of the context window in Claude Code.

     └ mcp__DEVONthink__is_running (DEVONthink): 431 tokens
     └ mcp__DEVONthink__create_record (DEVONthink): 816 tokens
     └ mcp__DEVONthink__delete_record (DEVONthink): 702 tokens
     └ mcp__DEVONthink__move_record (DEVONthink): 759 tokens
     └ mcp__DEVONthink__get_record_properties (DEVONthink): 762 tokens
     └ mcp__DEVONthink__get_record_by_identifier (DEVONthink): 678 tokens
     └ mcp__DEVONthink__search (DEVONthink): 1.5k tokens
     └ mcp__DEVONthink__lookup_record (DEVONthink): 871 tokens
     └ mcp__DEVONthink__create_from_url (DEVONthink): 962 tokens
     └ mcp__DEVONthink__get_open_databases (DEVONthink): 592 tokens
     └ mcp__DEVONthink__current_database (DEVONthink): 446 tokens
     └ mcp__DEVONthink__selected_records (DEVONthink): 463 tokens
     └ mcp__DEVONthink__list_group_content (DEVONthink): 501 tokens
     └ mcp__DEVONthink__get_record_content (DEVONthink): 465 tokens
     └ mcp__DEVONthink__rename_record (DEVONthink): 486 tokens
     └ mcp__DEVONthink__add_tags (DEVONthink): 468 tokens
     └ mcp__DEVONthink__remove_tags (DEVONthink): 499 tokens
     └ mcp__DEVONthink__classify (DEVONthink): 551 tokens
     └ mcp__DEVONthink__compare (DEVONthink): 568 tokens
     └ mcp__DEVONthink__replicate_record (DEVONthink): 833 tokens
     └ mcp__DEVONthink__duplicate_record (DEVONthink): 830 tokens
     └ mcp__DEVONthink__convert_record (DEVONthink): 984 tokens
     └ mcp__DEVONthink__update_record_content (DEVONthink): 525 tokens

[ebowman]

Yeah, fair enough. It's working great for me, but I am struggling to find the time to keep working on it. Happy to let this languish for now - sorry for the hassle. I'll likely pick it up again at some point.

[ebowman]

Closing to refine and consolidate changes before resubmission. Will create a more focused PR after further testing and cleanup.