|
1 | 1 | # Libby Downloader - Development Guide |
2 | 2 |
|
3 | | -This document provides context for AI assistants (Claude) working on this codebase. |
| 3 | +TypeScript CLI tool for downloading audiobooks from Libby with realistic user simulation to minimize detection risk. |
4 | 4 |
|
5 | | -## Project Overview |
6 | | - |
7 | | -A TypeScript CLI tool for downloading audiobooks from Libby with realistic user simulation to minimize detection risk. Built as a safer, more responsible alternative to the original TamperMonkey script. |
8 | | - |
9 | | -## Architecture |
10 | | - |
11 | | -### Core Modules |
12 | | - |
13 | | -1. **auth/** - Authentication and session management |
14 | | - - `libby-auth.ts` - Manual login flow, cookie persistence |
15 | | - |
16 | | -2. **browser/** - Puppeteer automation with stealth |
17 | | - - `manager.ts` - Browser lifecycle, session management |
18 | | - - `stealth.ts` - User behavior simulation (mouse, scrolling, delays) |
19 | | - |
20 | | -3. **downloader/** - Libby API interaction and chapter downloading |
21 | | - - `libby-api.ts` - Data extraction via JSON.parse hooks, BIF object access |
22 | | - - `chapter-downloader.ts` - Sequential downloads with rate limiting |
23 | | - |
24 | | -4. **processor/** - Audio processing |
25 | | - - `ffmpeg-processor.ts` - Chapter merging, metadata, chapter markers |
26 | | - |
27 | | -5. **metadata/** - ID3 tag embedding |
28 | | - - `embedder.ts` - Cover art, metadata embedding with node-id3 |
29 | | - |
30 | | -6. **utils/** - Shared utilities |
31 | | - - `delay.ts` - Random delays, sleep functions |
32 | | - - `fs.ts` - File operations, sanitization |
33 | | - - `logger.ts` - Structured logging |
34 | | - - `rate-limiter.ts` - Three-mode rate limiting (safe/balanced/aggressive) |
35 | | - |
36 | | -7. **types/** - TypeScript definitions |
37 | | - - Shared interfaces for books, chapters, configs |
38 | | - |
39 | | -8. **cli.ts** - Main CLI entry point using Commander.js |
40 | | - |
41 | | -## Key Technical Decisions |
42 | | - |
43 | | -### Why Puppeteer? |
44 | | - |
45 | | -- Real browser context prevents detection |
46 | | -- Access to Libby's internal objects (BIF, odreadCmptParams) |
47 | | -- Cookie persistence for session reuse |
48 | | -- Stealth plugins for anti-detection |
49 | | - |
50 | | -### Why Sequential Downloads? |
51 | | - |
52 | | -- Parallel downloads are easily detected (original script's issue) |
53 | | -- Humans read/listen sequentially, not all-at-once |
54 | | -- Variable timing between chapters mimics real behavior |
55 | | - |
56 | | -### Rate Limiting Strategy |
57 | | - |
58 | | -- **Safe mode**: 8-20s delays, breaks every 3 chapters (1 book/hour max) |
59 | | -- **Balanced mode**: 4-12s delays, breaks every 5 chapters (2 books/hour) |
60 | | -- **Aggressive mode**: 2-6s delays, minimal breaks (5 books/hour) - HIGH RISK |
61 | | - |
62 | | -### Why Not Use Libby's API Directly? |
63 | | - |
64 | | -- No documented public API |
65 | | -- Authentication is complex |
66 | | -- TamperMonkey approach (hooking internal data) is proven |
67 | | -- Browser context provides authenticated session automatically |
68 | | - |
69 | | -## Code Patterns |
70 | | - |
71 | | -### Error Handling |
72 | | - |
73 | | -- Use try/catch with logger.error() |
74 | | -- Clean up resources in finally blocks |
75 | | -- Provide helpful error messages to CLI users |
76 | | - |
77 | | -### Async Operations |
78 | | - |
79 | | -- All I/O operations are async |
80 | | -- Use await for sequential operations |
81 | | -- Rate limiter enforces delays between operations |
82 | | - |
83 | | -### Type Safety |
84 | | - |
85 | | -- Strict TypeScript mode enabled |
86 | | -- Explicit interfaces for all data structures |
87 | | -- No `any` types except for page.evaluate() contexts |
88 | | - |
89 | | -## Development Workflow |
90 | | - |
91 | | -### Running Locally |
| 5 | +## Common Commands |
92 | 6 |
|
93 | 7 | ```bash |
| 8 | +# Development |
94 | 9 | npm run dev -- login # Test login flow |
95 | | -npm run dev -- list # Test book listing |
96 | | -npm run dev -- download <id> # Test download (use safe mode!) |
| 10 | +npm run dev -- list # List borrowed books |
| 11 | +npm run dev -- download <id> # Download book (ALWAYS use --mode safe for testing!) |
| 12 | + |
| 13 | +# Testing & Validation |
| 14 | +npm test # Run Jest tests |
| 15 | +npm run test:coverage # Run tests with coverage report |
| 16 | +npm run check-all # Full validation: typecheck + lint + format + test |
| 17 | + |
| 18 | +# Building |
| 19 | +npm run build # Compile TypeScript to dist/ |
| 20 | +npm run typecheck # Type check without emitting files |
| 21 | + |
| 22 | +# Code Quality |
| 23 | +npm run lint # Check code with ESLint |
| 24 | +npm run lint:fix # Auto-fix linting issues |
| 25 | +npm run format # Format code with Prettier |
| 26 | +npm run format:check # Check if code is formatted |
97 | 27 | ``` |
98 | 28 |
|
99 | | -### Before Committing |
100 | | - |
101 | | -```bash |
102 | | -npm run check-all # Type check + lint + format + test |
103 | | -``` |
104 | | - |
105 | | -### Pre-commit Hook |
106 | | - |
107 | | -Automatically formats and lints staged files. |
108 | | - |
109 | | -### Pre-push Hook |
110 | | - |
111 | | -Runs full validation suite before push. |
112 | | - |
113 | | -## Testing Strategy |
114 | | - |
115 | | -### Unit Tests |
116 | | - |
117 | | -- Focus on pure functions (utils/delay.ts, utils/fs.ts) |
118 | | -- Mock external dependencies (Puppeteer, FFmpeg) |
119 | | - |
120 | | -### Integration Tests |
121 | | - |
122 | | -- Test rate limiter behavior |
123 | | -- Test metadata embedding with sample files |
| 29 | +## Core Files |
124 | 30 |
|
125 | | -### Manual Testing |
| 31 | +**Entry Points:** |
| 32 | +- `src/cli.ts` - Main CLI interface (Commander.js) |
| 33 | +- `src/index.ts` - Library exports |
126 | 34 |
|
127 | | -- Always test downloads in safe mode first |
128 | | -- Use test library card if possible |
129 | | -- Monitor for detection/bans |
130 | | - |
131 | | -## Important Constraints |
| 35 | +**Key Modules:** |
| 36 | +- `src/auth/libby-auth.ts` - Manual login, cookie persistence |
| 37 | +- `src/browser/manager.ts` - Puppeteer browser lifecycle |
| 38 | +- `src/downloader/libby-api.ts` - **CRITICAL**: Extracts data via JSON.parse hooks to capture BIF object and odreadCmptParams |
| 39 | +- `src/downloader/chapter-downloader.ts` - Sequential downloads with rate limiting |
| 40 | +- `src/utils/rate-limiter.ts` - Three modes: safe (8-20s), balanced (4-12s), aggressive (2-6s) |
132 | 41 |
|
133 | | -### Security & Ethics |
| 42 | +**Configuration:** |
| 43 | +- `config/stealth.json` - Rate limiting presets |
| 44 | +- `eslint.config.js` - ESLint v9 flat config |
| 45 | +- `jest.config.js` - Test configuration |
| 46 | +- `.husky/` - Git hooks (pre-commit: lint-staged, pre-push: check-all) |
134 | 47 |
|
135 | | -- Tool is for educational purposes only |
136 | | -- Users accept all responsibility for ToS violations |
137 | | -- Emphasize detection risks in all documentation |
138 | | -- No bypassing of DRM (only downloading already-accessible content) |
| 48 | +## CRITICAL Rules |
139 | 49 |
|
140 | | -### Performance |
| 50 | +**NEVER commit without passing validation:** |
| 51 | +- Pre-commit hook auto-formats and lints staged files |
| 52 | +- Pre-push hook runs full `check-all` suite |
| 53 | +- If hooks fail, fix issues before committing |
141 | 54 |
|
142 | | -- Sequential downloads are intentionally slow |
143 | | -- Rate limiting is critical for detection avoidance |
144 | | -- FFmpeg operations can be memory-intensive |
| 55 | +**NEVER use parallel downloads:** |
| 56 | +- Sequential downloads only (detection risk) |
| 57 | +- Rate limiter enforces delays between chapters |
| 58 | +- Users can choose safe/balanced/aggressive modes |
145 | 59 |
|
146 | | -### Dependencies |
| 60 | +**NEVER skip user simulation:** |
| 61 | +- Mouse movements, scrolling, random delays are intentional |
| 62 | +- Sequential pattern mimics human behavior |
| 63 | +- Parallel requests = instant detection |
147 | 64 |
|
148 | | -- Requires FFmpeg installed on system (not bundled) |
149 | | -- Large Puppeteer install (~300MB with Chromium) |
150 | | -- Node-ID3 for metadata (simpler than FFmpeg metadata) |
| 65 | +**IMPORTANT: TypeScript `any` usage:** |
| 66 | +- `any` types are ALLOWED in `page.evaluate()` contexts (browser environment) |
| 67 | +- `any` types are ALLOWED in logger variadic parameters |
| 68 | +- These ESLint warnings are expected and should NOT be "fixed" |
151 | 69 |
|
152 | | -## Common Issues |
| 70 | +## Architecture |
153 | 71 |
|
154 | | -### "Not logged in" errors |
| 72 | +**How it works:** |
| 73 | +1. Puppeteer launches real Chrome with stealth plugins |
| 74 | +2. User logs in manually (browser window opens) |
| 75 | +3. Cookies saved to `~/.libby-downloader/cookies.json` |
| 76 | +4. On book page, hook `JSON.parse` to capture `odreadCmptParams` (crypto keys) |
| 77 | +5. Extract `BIF` object (book metadata, chapter URLs) |
| 78 | +6. Download chapters sequentially with delays |
| 79 | +7. FFmpeg merges chapters, adds metadata/chapter markers |
155 | 80 |
|
156 | | -- Cookie expiration - re-run login |
157 | | -- Check ~/.libby-downloader/cookies.json exists |
| 81 | +**Rate Limiting:** |
| 82 | +- Safe: 8-20s delays, breaks every 3 chapters, max 1 book/hour |
| 83 | +- Balanced (default): 4-12s delays, breaks every 5 chapters, max 2 books/hour |
| 84 | +- Aggressive: 2-6s delays, no breaks, max 5 books/hour (HIGH DETECTION RISK) |
158 | 85 |
|
159 | | -### TypeScript errors in page.evaluate() |
| 86 | +## Code Style |
160 | 87 |
|
161 | | -- page.evaluate() runs in browser context (DOM types) |
162 | | -- Use `(window as any)` for custom properties |
163 | | -- Cast functions to `any` when needed for flexibility |
| 88 | +**IMPORTANT formatting rules:** |
| 89 | +- Prettier: single quotes, 100 char width, 2-space indent |
| 90 | +- ESLint: TypeScript recommended rules + Prettier integration |
| 91 | +- All staged files auto-formatted on commit |
164 | 92 |
|
165 | | -### FFmpeg not found |
| 93 | +**Naming:** |
| 94 | +- PascalCase: classes |
| 95 | +- camelCase: functions, variables |
| 96 | +- SCREAMING_SNAKE_CASE: constants |
166 | 97 |
|
167 | | -- User must install separately (Homebrew, apt, etc.) |
168 | | -- Check with `ffmpeg -version` before processing |
| 98 | +**Error handling:** |
| 99 | +- Use try/catch with `logger.error()` |
| 100 | +- Clean up resources in finally blocks |
| 101 | +- Unused error variables: use `catch { }` without parameter |
169 | 102 |
|
170 | | -### Rate limit warnings |
| 103 | +## Testing |
171 | 104 |
|
172 | | -- User tried to download too fast |
173 | | -- Enforce cooldown periods |
174 | | -- Suggest safe mode |
| 105 | +**Unit tests location:** `src/utils/__tests__/` |
175 | 106 |
|
176 | | -## Code Style |
| 107 | +**Current coverage:** Focus on pure utility functions |
| 108 | +- `delay.test.ts` - Timing utilities |
| 109 | +- `fs.test.ts` - File operations |
177 | 110 |
|
178 | | -### Formatting |
| 111 | +**IMPORTANT: Don't test against real Libby:** |
| 112 | +- Use mocks for Puppeteer, FFmpeg |
| 113 | +- Test with safe mode only if using real endpoints |
| 114 | +- Monitor for detection/bans |
179 | 115 |
|
180 | | -- Prettier with single quotes, 100 char width |
181 | | -- 2-space indentation |
182 | | -- Trailing commas (ES5 style) |
| 116 | +## Repository Etiquette |
183 | 117 |
|
184 | | -### Naming |
| 118 | +**Branch strategy:** |
| 119 | +- Main branch: `main` |
| 120 | +- Feature branches: `feature/descriptive-name` |
| 121 | +- Bug fixes: `fix/issue-description` |
185 | 122 |
|
186 | | -- PascalCase for classes |
187 | | -- camelCase for functions/variables |
188 | | -- SCREAMING_SNAKE_CASE for constants |
189 | | -- Descriptive names over brevity |
| 123 | +**Commit messages:** |
| 124 | +- Descriptive, present tense |
| 125 | +- Multi-line for complex changes |
| 126 | +- No emoji (per user preferences) |
190 | 127 |
|
191 | | -### Comments |
| 128 | +**Pull requests:** |
| 129 | +- All checks must pass (GitHub Actions CI) |
| 130 | +- Tests run on Node 18.x, 20.x, 22.x |
| 131 | +- Coverage thresholds: 50% branches, 60% statements |
192 | 132 |
|
193 | | -- Explain "why", not "what" |
194 | | -- TSDoc for public APIs |
195 | | -- Inline comments for tricky logic only |
| 133 | +## Common Issues |
196 | 134 |
|
197 | | -## Future Improvements |
| 135 | +**"Not logged in" errors:** |
| 136 | +- Cookie expired - run `npm run dev -- login` |
| 137 | +- Check `~/.libby-downloader/cookies.json` exists |
198 | 138 |
|
199 | | -### Potential Features |
| 139 | +**TypeScript errors in page.evaluate():** |
| 140 | +- This runs in browser context - `(window as any)` is correct |
| 141 | +- Don't try to "fix" these with stricter types |
200 | 142 |
|
201 | | -- Resume interrupted downloads |
202 | | -- Multiple library card support |
203 | | -- Download queue management |
204 | | -- Better chapter title detection |
205 | | -- M4B output format option |
| 143 | +**ESLint warnings about `any`:** |
| 144 | +- Expected in page.evaluate() and logger functions |
| 145 | +- These warnings are acceptable, don't suppress them |
206 | 146 |
|
207 | | -### Detection Avoidance Enhancements |
| 147 | +**FFmpeg not found:** |
| 148 | +- User must install: `brew install ffmpeg` (macOS) or `apt install ffmpeg` (Linux) |
| 149 | +- Required for chapter merging |
208 | 150 |
|
209 | | -- ML-based timing patterns |
210 | | -- Adaptive rate limiting based on success |
211 | | -- Session replay from real user behavior |
212 | | -- Proxy rotation support |
| 151 | +## Dependencies |
213 | 152 |
|
214 | | -## Maintenance Notes |
| 153 | +**IMPORTANT external requirements:** |
| 154 | +- Node.js 18+ (tested on 18.x, 20.x, 22.x) |
| 155 | +- FFmpeg (not bundled, must be installed separately) |
| 156 | +- Chromium (bundled with Puppeteer, ~300MB) |
215 | 157 |
|
216 | | -### When Libby Changes |
| 158 | +**Key packages:** |
| 159 | +- `puppeteer-extra` + `puppeteer-extra-plugin-stealth` for detection avoidance |
| 160 | +- `fluent-ffmpeg` for audio processing |
| 161 | +- `node-id3` for metadata embedding |
| 162 | +- `commander` for CLI |
| 163 | +- `chalk` + `ora` for terminal UI |
217 | 164 |
|
218 | | -- Monitor `BIF` object structure (core data source) |
219 | | -- Check `odreadCmptParams` hook still works |
220 | | -- Update selectors for shelf/book pages |
221 | | -- Test login flow for changes |
| 165 | +## When Libby Changes |
222 | 166 |
|
223 | | -### Dependency Updates |
| 167 | +**CRITICAL monitoring points:** |
| 168 | +- `BIF` object structure (contains book/chapter metadata) |
| 169 | +- `odreadCmptParams` availability (crypto keys for chapter URLs) |
| 170 | +- Shelf page selectors: `[data-test-id="shelf-loan"]` |
| 171 | +- Login flow changes |
224 | 172 |
|
225 | | -- Puppeteer: Check for breaking changes in page API |
226 | | -- Node-ID3: Verify metadata compatibility |
227 | | -- FFmpeg: Test chapter merging still works |
| 173 | +If downloads suddenly fail, check browser console for BIF object changes. |
228 | 174 |
|
229 | 175 | ## Resources |
230 | 176 |
|
231 | | -- Puppeteer Docs: https://pptr.dev |
232 | | -- FFmpeg Docs: https://ffmpeg.org/documentation.html |
233 | | -- Libby Web App: https://libbyapp.com |
234 | | -- Node.js Docs: https://nodejs.org/docs/ |
235 | | - |
236 | | -## Contributing |
237 | | - |
238 | | -When adding new features: |
239 | | - |
240 | | -1. Create feature branch from main |
241 | | -2. Write tests first (TDD when possible) |
242 | | -3. Implement feature |
243 | | -4. Update documentation |
244 | | -5. Run `npm run check-all` |
245 | | -6. Create pull request with clear description |
246 | | - |
247 | | -## Support and Issues |
248 | | - |
249 | | -- GitHub Issues: For bug reports and feature requests |
250 | | -- Include: OS, Node version, error logs, steps to reproduce |
251 | | -- Privacy: Never share library card details or personal info |
| 177 | +- Puppeteer: https://pptr.dev |
| 178 | +- FFmpeg: https://ffmpeg.org/documentation.html |
| 179 | +- Libby: https://libbyapp.com |
| 180 | +- Node.js: https://nodejs.org/docs/ |
0 commit comments