Disable muse-spark bots and mask derived Metaculus tokens - #342
Closed
CodexVeritas wants to merge 1 commit into
Closed
Disable muse-spark bots and mask derived Metaculus tokens#342CodexVeritas wants to merge 1 commit into
CodexVeritas wants to merge 1 commit into
Conversation
The three muse-spark bots have failed on every question of the Fall 2026 season (45526-45569), producing zero forecasts. OpenRouter's only provider for these models returns HTTP 200 with an empty completion: content and reasoning are both null, there is no usage block, and the finish reason is "stop". Confirmed with direct calls to all three models. There is no alternative provider to route to, so they are disabled until it is fixed. Tokens pulled out of the METACULUS_TOKENS json are derived values that GitHub has never seen registered as secrets, so they were not masked and got printed in plain text in the logs of every bot job. This repo is public, so those logs are world readable. Masking them explicitly fixes new runs; the existing tokens still need rotating. The assertion that fired for the muse bots said only that the answer was not a string, which was not enough to tell an upstream provider fault from a parsing bug. It now names the model, the finish reason and the usage. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CodexVeritas
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
muse_spark_1_1,1_2and1_3have produced zero forecasts across the Fall 2026 season questions (45526–45569), and were the cause of most AIB workflow failures since Sept 7.Their passing runs were runs with
Questions to forecast: 0, which is why this was not visible during the Sept 2–6 gap between seasons.Root cause
OpenRouter returns HTTP 200 with an empty completion for all three models:
{"choices":[{"finish_reason":"stop","native_finish_reason":"completed", "message":{"role":"assistant","content":null,"refusal":null,"reasoning":null}}]}contentandreasoningare both null and there is nousageblock. This reproduces on any prompt, with and withouttemperature,include_reasoningandreasoning_effort, so there is no text for the client to read./models/meta/muse-spark-1.1/endpointslists a single provider, so provider routing is not an option either.muse-glimmer-30bis served by a different provider and is unaffected.The three bots are disabled following the existing convention (
TournConfig.NONEplus removing the workflow job), which keeps enabled bots (83) matched 1:1 with workflow jobs (83).Workflow token handling
The launcher extracts a single token from the
METACULUS_TOKENSsecret withjq. A value derived from a secret is a new string that Actions does not treat as masked, so it could be written to logs by later steps. This registers it with::add-mask::before use.Error message
The failure surfaced as
AssertionError: Answer is not a string and is of type: NoneType, which did not distinguish an upstream provider fault from a parsing bug. It now reports the model, the finish reason and the usage.Testing
code_tests/unit_tests/test_ai_models/test_general_llm.pycovering the empty response, the reasoning-content fallback and the normal path.main; they pass when run file-alone, so they are order-dependent.🤖 Generated with Claude Code