⚠️ This is beta software currently under heavy development. It is at this point a research project, and there will likely be false positives in the results.Note: Plothole is currently only tested on Linux systems.
AI-powered coding tools have changed, and will continue to change, the way we write software. They have dramatically accelerated how we write code. But this speed comes with a trade-off: AI tools routinely cut corners, skip steps, and hallucinate solutions to meet completion goals. As developers increasingly rely on AI assistants, we're discovering that these tools will sometimes cause code to look "finished", even if it means leaving critical gaps in the implementation.
AI coding assistants are often optimized for completion, not correctness. When faced with complex requirements or tight context windows, they might systematically take shortcuts. Sometimes they might do the right thing, they write the code that we need, but at other times they simply might not.
Examples of missing code or "cut corners":
- Skipping complex logic - Replacing intricate business rules with simplified placeholder implementations
- Hallucinating APIs - Confidently calling functions or methods that don't exist, assuming they'll be implemented later
- Deferring difficult parts - Converting challenging problems into TODO comments rather than solving them
- Oversimplifying error handling - Using generic catch-all exceptions instead of proper error management
- Ignoring edge cases - Focusing only on the "happy path" while leaving corner cases unhandled
- Fabricating configurations - Generating plausible-looking config values that may not work in practice
Traditional static analysis tools are not equipped to detect these sophisticated, AI-generated plot holes. Rule-based linters cannot understand context or intent, nor can they recognise the subtle ways in which AI tools mask incomplete implementations. A possible, at least partial, solution? Use AI to detect AI.
Plothole uses an 'LLM-as-judge' approach, leveraging the contextual understanding that makes AI coding tools powerful to systematically audit their output. One AI-assisted tool can be used to write the code, while another, such as Mistral's Devstral, can be used to audit it.
Plothole is an AI-powered code analysis tool that uses Ollama's OpenAI-compatible API and local models to systematically detect the shortcuts, hallucinations, and incomplete implementations that AI-assisted development inevitably leaves behind.
Plothole works to identify where AI tools have cut corners by finding:
- TODO/FIXME comments masking unfinished logic
- Stub implementations with empty or placeholder bodies
- Magic numbers and hardcoded values substituting for proper configuration
- Incomplete error handling and missing validation
By catching these AI-generated plotholes early, you can harness the speed of AI-assisted development while ensuring your codebase remains robust and production-ready.
- Quickstart Guide - Installation, configuration, and usage
- PDF Version - PDF version of the quickstart guide
- Security Overview - Security features and practices
| Feature | Description |
|---|---|
| AI-Powered Analysis | Uses any LLM that Ollama supports for intelligent code analysis |
| Git Integration | Analyze staged changes, commits, or commit ranges |
| Multiple Output Formats | Text, JSON, and XML output for different use cases |
| Configurable Detection | Adjustable severity levels and file pattern filtering |
| Smart File Filtering | Automatic detection of code files with customizable include/exclude patterns |
| In-Code Ignore Directives | Suppress specific findings using comments in your code |
| Configuration Management | YAML configuration files with environment variable support |
| Git Hooks | Install pre-commit and pre-push hooks for automatic analysis |
| Intelligent Caching | Content-based caching for improved performance and reduced API calls |
| Database Integration | Store and track analysis results over time with unique finding IDs |
| Finding Management | Mark findings as ignored, filter by file/status, and track changes |
# Get latest version and download the binary
VERSION=$(basename $(curl -Ls -o /dev/null -w %{url_effective} https://github.com/ccollicutt/plothole/releases/latest))
curl -Ls -o plothole "https://github.com/ccollicutt/plothole/releases/download/${VERSION}/plothole-${VERSION}-linux-amd64"
# Make it executable
chmod +x plothole
# Move to a directory in your PATH
sudo mv plothole /usr/local/bin/One thing to keep in mind, is that there is no easy solution to all of this. Plothole is just another potential tool in the tool belt.
As an LLM-based analysis tool, Plothole has several known limitations that are being actively worked on:
- File-level analysis limitation - Cannot see methods or functions defined in other files, which can lead to false positives about missing implementations
- String literal confusion - Difficulty distinguishing between test data and actual code, particularly TODO comments in test fixtures
- Pattern matching errors - May occasionally hallucinate findings that don't actually exist in the code
- Context misunderstanding - Can mistake valid defaults and intentional values for placeholders or incomplete implementations
These limitations result in false positives that require manual review. We recommend treating Plothole's findings as suggestions requiring human verification rather than absolute truths. As the tool evolves, we're working to reduce these false positive rates while maintaining high detection accuracy for genuine issues.
Note: Plothole is also serving as a testing ground for in-code metadata systems.
You'll notice that each non-test Go file in this codebase begins with a large comment block containing detailed metadata about the file's purpose, implementation details, algorithms, and technologies used. This is an experimental approach to embedding contextual information directly in source code.
These metadata comments serve multiple purposes:
- AI Context: Provide detailed context for AI-powered tools and code analysis
- Documentation: Self-documenting code that explains not just "what" but "why" and "how"
- Onboarding: Help new developers quickly understand complex files
- Architecture: Capture design decisions and implementation rationale
This experiment explores whether structured metadata comments can improve code comprehension for both humans and AI systems. The format and content of these comments may evolve as we learn what works best.
See the Security Overview for more information.
MIT License