Retry badge moderation when reasoning eats the whole budget - #70
Open
boomzero wants to merge 1 commit into
Open
Conversation
The moderation model reasons before answering and its reasoning tokens come out of the same budget as the answer, so a badge it finds hard could spend all 1024 tokens deliberating and return finish_reason "length" with no content at all. ReadModerationVerdict saw null, EditBadge failed closed, and the user was told the moderation service was unavailable when it had in fact answered at length -- just never in JSON. Observed on 陈开尔万岁, which the model reads as a profanity homophone and then argues with itself about: three of six runs at temperature 0 hit the cap and returned nothing. Raise the budget to 2048 and retry once, but only on truncation. A reply that is complete and still malformed is the model answering badly, and asking again would spend a second inference call to hear the same thing; that case and a thrown error both still fail closed on the first attempt. Ordinary badges finish in a few hundred tokens, so the headroom is only billed on the inputs that need it. The cap and the retry are exported so tools/check-badge.js shares them and its neuron figure stays honest about what a real edit costs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Reviewer's GuideImplements bounded retry logic for badge moderation when the model truncates its response, increases the moderation token budget, and aligns the CLI check-badge tool and tests with the new behavior and shared constants. Sequence diagram for EditBadge badge moderation retry on truncationsequenceDiagram
actor User
participant Process
participant AI
participant Output
User->>Process: EditBadge(Data)
loop BadgeModerationAttempts
Process->>AI: AI.run(BadgeModerationModel, BadgeModerationMaxTokens)
AI-->>Process: Reply
Process->>Process: ReadModerationVerdict(Reply)
alt Verdict usable
Process-->>User: Result(success, badge updated)
note right of Process: Break loop
else Verdict null
Process->>Process: ModerationReplyTruncated(Reply)
alt Reply truncated
Process->>Output: Warn("Badge moderation ran out of tokens...")
note right of Process: Next loop attempt
else Reply not truncated
Process-->>User: Result(false, "内容审核服务暂时不可用，请稍后重试")
note right of Process: Break loop
end
end
end
alt No usable verdict after all attempts
Process->>Output: Error("Badge moderation returned an unusable verdict")
Process-->>User: Result(false, "内容审核服务暂时不可用，请稍后重试")
end
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
2 issues found across 3 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="tools/check-badge.js">
<violation number="1" location="tools/check-badge.js:107">
P3: Two truncated replies are reported as “retried 2x”, although only one retry was made after the initial call. Count only truncations that actually lead to another attempt so exhausted retries match production behavior and tool output.</violation>
</file>
<file name="Source/Process.ts">
<violation number="1" location="Source/Process.ts:1593">
P3: A second truncated reply logs “retrying” although no attempts remain and no retry occurs. Guard this warning to attempts that can actually continue, so operational logs accurately distinguish a retry from final failure.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| if (verdict !== null || !ModerationReplyTruncated(body.result)) { | ||
| return { verdict, neurons, retries }; | ||
| } | ||
| retries++; |
Contributor
There was a problem hiding this comment.
P3: Two truncated replies are reported as “retried 2x”, although only one retry was made after the initial call. Count only truncations that actually lead to another attempt so exhausted retries match production behavior and tool output.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At tools/check-badge.js, line 107:
<comment>Two truncated replies are reported as “retried 2x”, although only one retry was made after the initial call. Count only truncations that actually lead to another attempt so exhausted retries match production behavior and tool output.</comment>
<file context>
@@ -78,19 +81,32 @@ async function moderate(content, bearer) {
+ if (verdict !== null || !ModerationReplyTruncated(body.result)) {
+ return { verdict, neurons, retries };
+ }
+ retries++;
}
- return {
</file context>
Suggested change
| retries++; | |
| if (attempt + 1 < BadgeModerationAttempts) retries++; |
Comment on lines
+1593
to
1594
| Output.Warn("Badge moderation ran out of tokens before answering, retrying\n" + | ||
| "Username: " + this.Username); |
Contributor
There was a problem hiding this comment.
P3: A second truncated reply logs “retrying” although no attempts remain and no retry occurs. Guard this warning to attempts that can actually continue, so operational logs accurately distinguish a retry from final failure.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At Source/Process.ts, line 1593:
<comment>A second truncated reply logs “retrying” although no attempts remain and no retry occurs. Guard this warning to attempts that can actually continue, so operational logs accurately distinguish a retry from final failure.</comment>
<file context>
@@ -1554,22 +1569,29 @@ export class Process {
+ if (Verdict !== null || !ModerationReplyTruncated(Reply)) {
+ break;
+ }
+ Output.Warn("Badge moderation ran out of tokens before answering, retrying\n" +
"Username: " + this.Username);
- return new Result(false, "内容审核服务暂时不可用，请稍后重试");
</file context>
Suggested change
| Output.Warn("Badge moderation ran out of tokens before answering, retrying\n" + | |
| "Username: " + this.Username); | |
| if (Attempt + 1 < BadgeModerationAttempts) { | |
| Output.Warn("Badge moderation ran out of tokens before answering, retrying\n" + | |
| "Username: " + this.Username); | |
| } |
Member
|
looks good |
Member
Author
|
. |
Member
Author
|
Then pls force merge |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
glm-4.7-flashreasons before answering, and its reasoning tokens come out of the samemax_completion_tokensbudget as the answer. A badge the model finds hard could spend the entire 1024-token allowance deliberating and come back withfinish_reason: "length"andcontent: null.ReadModerationVerdictcorrectly saw nothing usable,EditBadgefailed closed, and the user was told 内容审核服务暂时不可用 — when the model had in fact answered at length, just never in JSON.Found via
npm run check-badge -- 陈开尔万岁, which reportedUNUSABLE. The model reads 陈开尔 as a disguised profanity homophone and then argues with itself about whether 万岁 makes it rule 1 or rule 11. Six runs at temperature 0:Half of them never produced a verdict. Any badge that makes the model deliberate this long was unsettable.
The fix
BadgeModerationMaxTokens = 2048, replacing the inline1024.ModerationReplyTruncated()distinguishes "ran out of budget mid-thought" from "answered badly".EditBadgeretries once, only on truncation. A complete-but-malformed reply and a thrown error both still fail closed on the first attempt — retrying those would spend a second inference call to hear the same thing.tools/check-badge.jsshares them and its neuron figure stays honest about what a real edit costs, retries included. The tool now annotates(retried 1x after truncation).Cost
Unchanged for ordinary badges — 5 mixed checks came to 123.9 neurons, ~25 each, same as before. The extra headroom is only billed on inputs that actually reason that long.
Verification
Full suite: 95 pass, 0 fail. Six new tests cover retry-lands-a-reject, retry-lands-an-allow, two-truncations-fails-closed-at-exactly-2-calls, malformed-is-not-retried, throw-is-not-retried, and the raised budget assertion.
Against the live model, three runs of the input that started this:
No more
UNUSABLE; one run did truncate and the retry caught it, which is the mechanism working.Known limitation
This fixes no verdict, not wrong verdict. 陈开尔万岁 remains non-deterministic at temperature 0 — a later batch returned
REJECT rule 3, and across today's runs it has produced allowed, rule 1, and rule 3. That's a prompt problem (the model treats a personal name as a profanity homophone), deliberately left out of scope here.🤖 Generated with Claude Code
Summary by Sourcery
Increase badge moderation token budget and add a single retry on truncated model replies so hard badges no longer fail closed without a verdict.
Bug Fixes:
Enhancements:
Tests:
Summary by cubic
Fixes badge moderation failing closed when
@cf/zai-org/glm-4.7-flashspends the entire completion budget on reasoning and returns no JSON. Increases the token cap and retries once on truncation so users get an allow/reject instead of a service-unavailable error; normal badges are unaffected.BadgeModerationMaxTokensto 2048 and export it along withBadgeModerationAttempts=2.finish_reason: "length") and retry once; malformed replies and thrown errors still fail closed.tools/check-badge.js; tool now reports retries and neuron usage that matches real edits.Written for commit cbb54b3. Summary will update on new commits.