feat(ollama): add ollama-eval.mjs interactive evaluator#681
Open
mattkirkey wants to merge 3 commits into
Open
Conversation
Mirrors the gemini-eval.mjs pattern for local/free/private use.
- Probes Ollama reachability at startup (GET /api/tags, 5 s timeout)
so the user gets a clear error before prompt assembly begins.
- Loopback guard: rejects non-localhost OLLAMA_BASE_URL by default;
opt-out via OLLAMA_ALLOW_REMOTE=1 (strict equality, not truthy).
- Reads modes/_shared.md + modes/oferta.md + cv.md as context.
- Calls /v1/chat/completions (Ollama OpenAI-compat) with temperature 0.4,
num_ctx 32768 to handle typical 10K-15K token prompts.
- Saves evaluation to reports/{###}-{company}-{date}.md; prints tracker
entry reminder to stdout.
- JSDoc on readFile and nextReportNumber helpers.
- Add `ollama:eval` npm script.
Requires OLLAMA_BASE_URL / OLLAMA_MODEL (documented in .env.example,
added in the companion batch PR).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
|
Welcome to career-ops, @mattkirkey! Thanks for your first PR. A few things to know:
We'll review your PR soon. Join our Discord if you have questions. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughAdds a new CLI tool (ollama-eval.mjs) that embeds local context, validates and probes an Ollama endpoint, sends a chat completion request to evaluate a job description, parses a machine-readable score block, optionally saves a numbered markdown report, and adds an npm script to run it. ChangesOllama Job Evaluator CLI
Sequence DiagramsequenceDiagram
participant User as User / CLI
participant Script as ollama-eval.mjs
participant Ollama as Ollama Server
participant FS as File System
User->>Script: npm run ollama:eval [--file FILE] [--model MODEL] [--url URL] [--no-save]
Script->>FS: Read cv.md, modes/_shared.md, modes/oferta.md
Script->>Script: Build system prompt (include SCORE_SUMMARY instructions)
Script->>Ollama: GET /api/tags (reachability probe)
Ollama-->>Script: 200 OK / tags
Script->>Ollama: POST /v1/chat/completions (system + job description)
Ollama-->>Script: Full evaluation text including ---SCORE_SUMMARY---
Script->>Script: Parse SCORE_SUMMARY via regex -> COMPANY/ROLE/SCORE/ARCHETYPE/LEGITIMACY
Script->>FS: Create reports/ and write NNN-slug-date.md (strip machine block)
Script->>User: Print full evaluation and compact score/archetype/legitimacy line
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested labels
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
ESLint skipped: no ESLint configuration detected in root package.json. To enable, add Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@ollama-eval.mjs`:
- Around line 92-99: The current --file handling uses existsSync and then calls
readFileSync(filePath, 'utf-8') which can still throw on permission errors or if
filePath is a directory; wrap the read in a try/catch around readFileSync (and
optionally use statSync or fs.promises.stat to verify it's a regular file) and
on any error log a clear message including filePath and error.message, then call
process.exit(1) instead of letting the exception bubble; update the block that
assigns jdText to perform this guarded read and handle errors gracefully.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 510cbe1e-bd4e-4df0-b490-c0a64363cba9
📒 Files selected for processing (2)
ollama-eval.mjspackage.json
ollama-eval.mjs: - Wrap readFileSync in try/catch when reading --file input. The existsSync guard handles missing files, but a directory path, broken symlink, or permission error would throw an uncaught exception and print a raw stack trace. Now exits cleanly with a user-facing message. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@ollama-eval.mjs`:
- Around line 144-152: The nextReportNumber function will drop files once
numbers exceed 999; update it to accept any number of leading digits and to pad
to a larger width: change the filename filter from /^\d{3}-/ to /^\d+-/ (or use
a regex match to extract the leading digits) and replace the fixed slice(0, 3)
extraction with extracting the matched digit group before parseInt, then return
the next number padded with padStart(4, '0') (still using PATHS.reports and the
nextReportNumber function names to locate the change).
- Line 254: The parsed timeout value assigned to timeoutMs using parseInt can be
NaN for non-numeric OLLAMA_TIMEOUT_MS, which later leads to
AbortSignal.timeout(NaN) throwing; change the code around timeoutMs to validate
the parsed value (use Number.parseInt or Number and check
isFinite/Number.isInteger or !Number.isNaN), and if invalid either (a) fall back
to the default 300000 or (b) throw a clear configuration error that mentions
OLLAMA_TIMEOUT_MS must be a positive integer; update the code that calls
AbortSignal.timeout to use the validated numeric timeoutMs so
AbortSignal.timeout never receives NaN.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: a287cdd8-5fd7-473d-801a-14998f90e606
📒 Files selected for processing (1)
ollama-eval.mjs
ollama-eval.mjs:
- Fix report numbering above 999: replace fixed /^\d{3}-/ filter and
slice(0,3) with a regex capture group (/^(\d+)-/) that handles any
number of digits. padStart(3,'0') still zero-pads short numbers while
allowing 4+ digit names to pass through correctly.
- Validate OLLAMA_TIMEOUT_MS at startup: parseInt can return NaN on
non-numeric values (e.g. "30s"), which causes AbortSignal.timeout(NaN)
to throw a TypeError. Now exits with a clear error message instead.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
ollama-eval.mjs(~365 lines) — interactive single-offer evaluator that mirrors thegemini-eval.mjspattern, using a local Ollama model instead of a cloud API.GET /api/tags, 5 s timeout) so the user gets a clear error before prompt assembly begins.OLLAMA_BASE_URLby default; opt-out viaOLLAMA_ALLOW_REMOTE=1(strict equality, not truthy).modes/_shared.md+modes/oferta.md+cv.mdas context./v1/chat/completions(Ollama OpenAI-compat) withtemperature: 0.4,num_ctx: 32768.reports/{###}-{company}-{date}.md; prints tracker entry reminder to stdout.readFileandnextReportNumberhelpers.ollama:evalnpm script.Usage
Prerequisites: Ollama running locally with a model pulled:
ollama pull llama3.3 # or qwen2.5:32b, mistral-nemo, gemma3:27b ollama serveOLLAMA_BASE_URLandOLLAMA_MODELenv vars are documented in.env.example(added in companion PR #680).Test plan
node ollama-eval.mjs --helpprints usageOLLAMA_BASE_URL=http://remote:11434 node ollama-eval.mjs "test"triggers loopback guardreports/--no-saveflag skips report writeCompanion PR
PR #680 adds
ollama-batch.mjs+batch-runner.sh --backend ollama(batch mode).This PR covers the interactive/single-offer use case. Both PRs are independent and can merge in any order.
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Chores