Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .aidigestignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
test/test-ignore.ts
dist
.claude
src/__snapshots__/*
src/__snapshots__/*
scripts
3 changes: 3 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,5 +26,8 @@ jobs:
- name: Install dependencies
run: npm ci

- name: Run linting
run: npm run lint

- name: Run tests
run: npm test
1 change: 1 addition & 0 deletions .husky/pre-commit
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
npx lint-staged
39 changes: 39 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,42 @@
### 1.5.0

- Added `.aidigestminify` file support for including files with placeholder content instead of full content
- Files matching minify patterns are included but show only file type and a message that content is excluded
- Added `--minify-file` CLI option to specify custom minify file name
- Watch mode now monitors `.aidigestminify` file for changes
- All library functions now support `minifyFile` option

### 1.4.1

- Added `additionalDefaultIgnores` option to file processing functions
- Allows users to specify extra glob patterns to ignore programmatically
- Available in `processFiles`, `generateDigestFiles`, `generateDigestContent`, and `getFileStats`

### 1.4.0

- Switch from `js-tiktoken` to `tiktoken` package for better performance and compatibility
- Performance improvement: ~35% faster token counting
- Add token analysis scripts: `analyze-tokens`, `calculate-multiplier`, `test-multiplier`
- Remove redundant dual OpenAI tokenization for better efficiency

### 1.3.0

- Add `getFileStats()` function to retrieve file statistics without content (#32)
- Returns file path, size, and total token counts (GPT and Claude) sorted by size
- Update Prettier script to exclude .snap files from formatting
- Add comprehensive snapshot tests for new API function

### 1.2.4

- Bug fixes and stability improvements (#30)

### 1.2.3

- Enable TypeScript declaration file generation for better IDE support
- Add library support for using ai-digest as external npm package (#18)
- Export functions: `generateDigest`, `generateDigestContent`, `generateDigestFiles`, `writeDigestToFile`
- Comprehensive API documentation for programmatic usage

### 1.2.2

- Bump lockfile
Expand Down
63 changes: 58 additions & 5 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,12 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
## Commands
- Build: `npm run build` - Compiles TypeScript code to dist/
- Start: `npm run start` - Run the application with ts-node for development
- Test: `npm run test` - Run all Jest tests
- Single test: `npx jest src/index.test.ts -t "test name"` - Run specific test by name
- Test: `npm run test` - Run all Jest tests (configured to run serially to prevent file conflicts)
- Single test: `npx jest src/cli.test.ts -t "test name"` - Run specific test by name
- Update snapshots: `npm run test -- -u` - Update Jest snapshots after changes
- Format: `npm run prettier` - Format code with Prettier (excludes .snap files)
- Format: `npm run format` - Format code with ESLint auto-fix
- Format (legacy): `npm run prettier` - Format code with Prettier (excludes .snap files)
- Lint: `npm run lint` - Check code with ESLint without fixing
- Publish: `npm run prepublishOnly` - Automatically runs build before publishing

## Token Analysis Scripts
Expand Down Expand Up @@ -43,9 +45,14 @@ The architecture supports two usage patterns:
5. **Output Generation**: Combines processed files into a single markdown document

### Key Components
- `src/index.ts` - Main entry point containing CLI logic, library exports, and core processing functions
- `src/index.ts` - Main entry point containing CLI logic and library exports
- `src/digest.ts` - Core processing functions for file discovery, content processing, and output generation
- `src/utils.ts` - Utility functions for file processing, token counting, ignore patterns, and file type detection
- `src/index.test.ts` - Comprehensive test suite covering both CLI and library functionality
- `src/types.ts` - TypeScript type definitions for the project
- `src/cli.test.ts` - CLI functionality tests (command-line interface behavior)
- `src/library.test.ts` - Core library function tests (generateDigest, generateDigestContent, etc.)
- `src/file-stats.test.ts` - File statistics tests (getFileStats function)
- `src/minify.test.ts` - Minify functionality tests (.aidigestminify patterns and callbacks)
- `scripts/` - Analysis and utility scripts for token counting research

### Critical Functions
Expand All @@ -61,6 +68,12 @@ The architecture supports two usage patterns:
The tool consistently uses processed content size (markdown wrapper + content) for all file size calculations and displays. This ensures consistency between CLI output and library functions.

### Testing Architecture
- **Test Organization**: Tests are split into 4 files based on functionality:
- `cli.test.ts` - CLI interface and command-line behavior
- `library.test.ts` - Core library functions and API
- `file-stats.test.ts` - File statistics and size calculation
- `minify.test.ts` - Minify patterns and callback functionality
- **Serial Execution**: Jest is configured with `maxWorkers: 1` to prevent file conflicts during concurrent test execution
- **CLI Tests**: Use `execAsync` with `ts-node` to test actual CLI behavior
- **Library Tests**: Direct function imports for unit testing
- **Temporary Directories**: Each test creates isolated temp directories for file operations
Expand All @@ -86,6 +99,32 @@ The multiplier (0.9048) was derived from analysis showing:
- OpenAI (GPT-4o): 310,641 tokens
- Ratio: 0.9048 (OpenAI/Claude)

## Feature: .aidigestminify (v1.5.0)
Added support for `.aidigestminify` file which works similarly to `.aidigestignore` but instead of excluding files completely, it includes them with a placeholder message. This is useful for large generated files, compiled assets, or files that don't need their full content in the AI context.

**How it works:**
- Create a `.aidigestminify` file with patterns similar to `.gitignore`
- Files matching these patterns will be included with placeholder content
- The placeholder shows the file type and indicates the content was excluded
- Useful for: minified files, compiled code, large data files, database files

**Example patterns:**
```
*.min.js
*.min.css
dist/*
build/*
*.db
*.sqlite
```

**Implementation details:**
- Added `minifyFile` parameter to all core functions
- Modified `processFiles()` to check minify patterns before processing content
- Added `minifiedCount` to statistics tracking
- Watch mode monitors `.aidigestminify` file for changes
- CLI option `--minify-file` to specify custom minify file name

## Library Usage
The tool exports functions for programmatic use:
- `generateDigest(options)` - Returns content string when `outputFile: null`, writes file otherwise
Expand All @@ -94,6 +133,8 @@ The tool exports functions for programmatic use:
- `getFileStats(options)` - Returns file statistics sorted by size with total token counts, no content
- `writeDigestToFile(content, outputFile, stats, showOutputFiles, fileSizes)` - File writing utility

All library functions now support the `minifyFile` option for `.aidigestminify` patterns.

## Code Style & Patterns
- Use 2 spaces for indentation
- Prefer double quotes for strings
Expand All @@ -106,3 +147,15 @@ The tool exports functions for programmatic use:
- Add timeout values to Jest tests for file operations (10000-15000ms)
- Use snapshot testing for complex data structures that should remain stable
- Binary files should be tested with actual binary data, not mocked content

## Code Quality & Pre-commit Hooks
- ESLint is configured for TypeScript with basic formatting rules
- Pre-commit hook automatically runs ESLint fixes on staged files
- Use `npm run format` for manual code formatting with ESLint
- Use `npm run lint` to check for linting issues without auto-fixing

## Test Configuration
- Jest is configured to run tests serially (`maxWorkers: 1`) to prevent file conflicts during temporary file operations
- Tests are organized into 4 files by functionality: CLI, Library API, File Stats, and Minify features
- Each test uses unique temporary directories to avoid conflicts
- Shared output files (like `codebase.md`) are cleaned up in `afterAll` hooks
Loading