feat: add standalone Kimi evaluator script#659
Conversation
📝 WalkthroughWalkthroughThis PR adds a complete Kimi/Moonshot API integration for resume evaluation. A new Node.js CLI script processes job descriptions against candidate profiles, calls Moonshot's API, extracts structured evaluation results, and persists reports with tracking metadata. ChangesKimi/Moonshot Evaluator
🎯 2 (Simple) | ⏱️ ~10 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
ESLint skipped: no ESLint configuration detected in root package.json. To enable, add Tip 💬 Introducing Slack Agent: The best way for teams to turn conversations into code.Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.
Built for teams:
One agent for your entire SDLC. Right inside Slack. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kimi-eval.mjs`:
- Around line 324-335: The TSV row is built from untrusted model outputs
(company, role) which may contain tabs/newlines or spreadsheet-formula prefixes
and can break parsing; before building tsvRow (the variables company and role
used with tsvRow and writeFileSync), sanitize each field by trimming and
replacing any tabs/newlines with a single space and neutralize leading formula
characters (=, +, -, @) by prefixing with a single quote (or otherwise escaping)
and also ensure filename is safe/normalised; then join the sanitized values into
tsvRow and call writeFileSync with tsvPath as before.
- Around line 218-229: The OpenAI call using openai.chat.completions.create (the
block that builds messages with systemPrompt, modelName and jdText) needs a
full-request timeout via AbortSignal rather than only the SDK timeout option;
create an AbortController before the call, set a timer to call
controller.abort() after the desired timeout, pass controller.signal into the
create() options (e.g., signal), and ensure you clear the timer after completion
or on error so you don’t leak timers.
In `@package.json`:
- Around line 18-20: The package.json is missing the openai dependency required
by the new script "kimi:eval" which imports openai in kimi-eval.mjs; add
"openai" to dependencies in package.json (not devDependencies) with an
appropriate semver (e.g., latest compatible version) and run npm install / yarn
install so the kimi:eval script can import openai at runtime.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 02305ac8-e78f-4dc0-bd9a-c27e04a40512
📒 Files selected for processing (4)
.env.exampleREADME.mdkimi-eval.mjspackage.json
| const response = await openai.chat.completions.create({ | ||
| model: modelName, | ||
| messages: [ | ||
| { role: "system", content: systemPrompt }, | ||
| { | ||
| role: "user", | ||
| content: `\n\nJOB DESCRIPTION TO EVALUATE:\n\n${jdText}`, | ||
| }, | ||
| ], | ||
| temperature: 0.4, // deterministic enough for structured evaluation | ||
| max_tokens: 8192, | ||
| }); |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
For the currently used OpenAI Node SDK version, what is the correct way to set request timeouts (client-level and per-request) for chat.completions.create?
💡 Result:
In the current OpenAI Node SDK, you can configure request timeouts using the timeout option, but it is important to understand its limitations regarding the request lifecycle [1][2]. Client-Level Configuration You can set a default timeout for all requests when initializing the OpenAI client [2]. This value is in milliseconds [3][2]. const client = new OpenAI({ timeout: 20 * 1000, // 20 seconds (default is 10 minutes) }); Per-Request Configuration You can override this timeout for individual chat.completions.create calls by passing a second argument to the method [2]. await client.chat.completions.create( { model: 'gpt-4o', messages: [...] }, { timeout: 5 * 1000 } // 5 seconds); Important Limitations The timeout option provided by the SDK only covers the time taken to receive the initial response headers (Time to First Byte) [1]. Once the headers are received, the internal timer is cleared, and the subsequent reading of the response body is not guarded by this timeout [1]. Consequently, if the response body stream hangs, the request may remain open indefinitely [1]. Recommended Workaround for Robust Timeouts Because of the limitation mentioned above, if you require a hard timeout that covers the entire duration of the request (including body streaming), the recommended approach is to use an AbortSignal [1]. You can create a signal with a hard timeout and pass it to the signal option in the request configuration [1]. // Example using a hard timeout for the entire request const controller = new AbortController; const timeoutId = setTimeout( => controller.abort, 30000); // 30s hard limit try { const completion = await client.chat.completions.create( { model: 'gpt-4o', messages: [...] }, { signal: controller.signal }); //... process response } catch (err) { if (err.name === 'AbortError') { console.error('Request timed out after 30s'); } } finally { clearTimeout(timeoutId); } This method ensures that the request is aborted regardless of whether the SDK's internal header-arrival timer has already cleared [1].
Citations:
- 1: SDK
timeoutoption does not cover response-body read phase (can hang forever) openai/openai-node#1825 - 2: https://www.github.com/openai/openai-node
- 3: https://jsr.io/@openai/openai/doc/~/ClientOptions
🏁 Script executed:
# First, find the package.json to check the OpenAI SDK version
find . -name "package.json" -type f | head -5 | xargs catRepository: santifer/career-ops
Length of output: 1267
🏁 Script executed:
# Search for the OpenAI client initialization in kimi-eval.mjs
rg "new OpenAI" kimi-eval.mjs -A 5 -B 5Repository: santifer/career-ops
Length of output: 342
🏁 Script executed:
# Get the complete context around lines 218-229 in kimi-eval.mjs
sed -n '200,240p' kimi-eval.mjsRepository: santifer/career-ops
Length of output: 1366
🏁 Script executed:
# Check for OpenAI import in kimi-eval.mjs
head -50 kimi-eval.mjs | grep -i "import\|require"Repository: santifer/career-ops
Length of output: 365
🏁 Script executed:
# Also check devDependencies and all dependencies sections
rg "openai" package.json -iRepository: santifer/career-ops
Length of output: 45
🏁 Script executed:
# Check if there's a node_modules or package-lock.json that might indicate the version
ls -la | grep -E "package-lock|yarn.lock|node_modules"Repository: santifer/career-ops
Length of output: 45
🏁 Script executed:
# Check if openai appears anywhere in package-lock.json or yarn.lock
find . -name "package-lock.json" -o -name "yarn.lock" | xargs grep -l "openai" 2>/dev/null | head -5Repository: santifer/career-ops
Length of output: 45
🏁 Script executed:
# Look at the full imports section of kimi-eval.mjs
head -20 kimi-eval.mjsRepository: santifer/career-ops
Length of output: 711
🏁 Script executed:
# Check if OpenAI is used with any signal/timeout handling elsewhere
rg "AbortController|AbortSignal|timeout" kimi-eval.mjsRepository: santifer/career-ops
Length of output: 45
🏁 Script executed:
# Check if there's a setup or init file that might install dependencies
rg "npm install|openai" . -l | grep -v node_modules | head -10Repository: santifer/career-ops
Length of output: 240
The proposed timeout parameter alone is insufficient; implement a complete request timeout using AbortSignal instead.
The external API call can hang indefinitely during response body streaming, even with a client-level timeout. While the timeout option in the OpenAI SDK covers the initial headers (TTFB), the response body stream remains unprotected, meaning hanging reads will not be interrupted [1].
For a robust solution that covers the entire request lifecycle, use AbortSignal:
Recommended fix
let evaluationText;
try {
+ const controller = new AbortController();
+ const timeoutId = setTimeout(() => controller.abort(), 120000);
+
const response = await openai.chat.completions.create({
model: modelName,
messages: [
{ role: "system", content: systemPrompt },
{
role: "user",
content: `\n\nJOB DESCRIPTION TO EVALUATE:\n\n${jdText}`,
},
],
temperature: 0.4,
max_tokens: 8192,
- });
+ }, { signal: controller.signal });
+
+ clearTimeout(timeoutId);
} catch (err) {
+ if (err.name === 'AbortError') {
+ console.error("❌ Kimi API timeout after 120s");
+ process.exit(1);
+ }
console.error("❌ Kimi API error:", err.message || err);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kimi-eval.mjs` around lines 218 - 229, The OpenAI call using
openai.chat.completions.create (the block that builds messages with
systemPrompt, modelName and jdText) needs a full-request timeout via AbortSignal
rather than only the SDK timeout option; create an AbortController before the
call, set a timer to call controller.abort() after the desired timeout, pass
controller.signal into the create() options (e.g., signal), and ensure you clear
the timer after completion or on error so you don’t leak timers.
| const tsvRow = [ | ||
| num, | ||
| today, | ||
| company, | ||
| role, | ||
| "Evaluated", | ||
| score, | ||
| "❌", | ||
| `reports/${filename}`, | ||
| "", | ||
| ].join("\t"); | ||
| writeFileSync(tsvPath, tsvRow, "utf-8"); |
There was a problem hiding this comment.
Sanitize TSV fields before writing tracker rows.
company and role come from model output and may include tabs/newlines (or spreadsheet formula prefixes), which can corrupt TSV parsing and downstream tooling.
Proposed fix
+ const sanitizeTsv = (value) => {
+ const s = String(value ?? "").replace(/[\t\r\n]+/g, " ").trim();
+ return /^[=+\-@]/.test(s) ? `'${s}` : s;
+ };
+
const tsvRow = [
num,
today,
- company,
- role,
+ sanitizeTsv(company),
+ sanitizeTsv(role),
"Evaluated",
- score,
+ sanitizeTsv(score),
"❌",
`reports/${filename}`,
"",
].join("\t");📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const tsvRow = [ | |
| num, | |
| today, | |
| company, | |
| role, | |
| "Evaluated", | |
| score, | |
| "❌", | |
| `reports/${filename}`, | |
| "", | |
| ].join("\t"); | |
| writeFileSync(tsvPath, tsvRow, "utf-8"); | |
| const sanitizeTsv = (value) => { | |
| const s = String(value ?? "").replace(/[\t\r\n]+/g, " ").trim(); | |
| return /^[=+\-@]/.test(s) ? `'${s}` : s; | |
| }; | |
| const tsvRow = [ | |
| num, | |
| today, | |
| sanitizeTsv(company), | |
| sanitizeTsv(role), | |
| "Evaluated", | |
| sanitizeTsv(score), | |
| "❌", | |
| `reports/${filename}`, | |
| "", | |
| ].join("\t"); | |
| writeFileSync(tsvPath, tsvRow, "utf-8"); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kimi-eval.mjs` around lines 324 - 335, The TSV row is built from untrusted
model outputs (company, role) which may contain tabs/newlines or
spreadsheet-formula prefixes and can break parsing; before building tsvRow (the
variables company and role used with tsvRow and writeFileSync), sanitize each
field by trimming and replacing any tabs/newlines with a single space and
neutralize leading formula characters (=, +, -, @) by prefixing with a single
quote (or otherwise escaping) and also ensure filename is safe/normalised; then
join the sanitized values into tsvRow and call writeFileSync with tsvPath as
before.
| "gemini:eval": "node gemini-eval.mjs", | ||
| "kimi:eval": "node kimi-eval.mjs" | ||
| }, |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "1) Verify script entry exists"
rg -n '"kimi:eval"\s*:\s*"node kimi-eval\.mjs"' package.json
echo "2) Verify OpenAI import exists in evaluator"
rg -n 'import OpenAI from "openai"' kimi-eval.mjs
echo "3) Verify dependency declaration"
python - <<'PY'
import json
p=json.load(open("package.json"))
deps=p.get("dependencies",{})
dev=p.get("devDependencies",{})
print("openai in dependencies:", "openai" in deps)
print("openai in devDependencies:", "openai" in dev)
PYRepository: santifer/career-ops
Length of output: 306
Add missing openai dependency to dependencies.
The new kimi:eval script imports openai (line 34 of kimi-eval.mjs) but the package is not declared in dependencies or devDependencies. This will cause the command to fail at runtime with a module not found error.
Proposed fix
"dependencies": {
"@google/generative-ai": "^0.24.1",
"dotenv": "^17.0.0",
"js-yaml": "^4.1.1",
+ "openai": "^4.0.0",
"playwright": "^1.58.1"
}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@package.json` around lines 18 - 20, The package.json is missing the openai
dependency required by the new script "kimi:eval" which imports openai in
kimi-eval.mjs; add "openai" to dependencies in package.json (not
devDependencies) with an appropriate semver (e.g., latest compatible version)
and run npm install / yarn install so the kimi:eval script can import openai at
runtime.
1 participant
Description
This PR introduces the standalone Kimi evaluator script (
kimi-eval.mjs) as requested by the maintainer.In alignment with the project's recent architectural shift to a single
AGENTS.mdfile, this PR drops the native Kimi CLI support and.kimi/directory structure from the previous attempt, focusing entirely on the standalone API script.Changes
kimi-eval.mjs: A free-tier alternative to the Claude-based pipeline using Moonshot's API.package.json: Added"kimi:eval": "node kimi-eval.mjs"..env.example: Added Kimi/Moonshot API keys with accurate documentation regarding Tier 0 rate limits (3 RPM) and the $1 initial recharge requirement for sustained workflows.README.md: Replaced old documentation with a new block explaining the standalone script, distinguishing between the flagship 1T MoEkimi-k2.6model and themoonshot-v1-8kfallback.Updates PR #508.
Summary by CodeRabbit
New Features
Documentation