Agent index mismatch between `reddit_profiles.json` and `twitter_profiles.csv`/agent_configs (interview mis-targeting); report chat intermittently answers without the report body ("(nonereport)")

[Quantum-eon]

Summary

Two related post-run reliability problems. (a) The agent ordering in reddit_profiles.json does not match the ordering in twitter_profiles.csv/agent_configs (the latter includes extra nodes such as org entities and any deduplicated cast members). The interview/batch endpoints index by the latter, so a client that derives indices from reddit_profiles.json interviews the wrong agent, silently. (b) Separately, the report chat agent intermittently responds with "(nonereport)" — i.e. it answers without having loaded the generated report body — so its answers reflect an empty report rather than the actual analysis.

Reproduction

(a) Index mismatch:

1. Run a simulation whose cast includes non-person entities (e.g. an org node) or where a dedup occurred (we saw a Valdoria/"Republic of Valdoria" dedup) — the two profile listings then diverge in composition/order.
2. Pick a target agent's index from reddit_profiles.json and call the interview/batch endpoint with it.
3. Compare the interview response content against the intended persona.

(b) "(nonereport)":

1. Generate the report, then query the report chat.
2. Intermittently, responses contain "(nonereport)" and clearly reflect no report content. Observed on two different model configurations (Claude Sonnet 4 and GPT-4.1 runs).

Observed

• (a) In one run our first interview pass, indexed from reddit_profiles.json, mis-targeted two of five probes (an org entity answered at the index we expected a specific persona at, and a different country's persona at another). Detected only via response content; re-probing with indices taken from twitter_profiles.csv hit the intended agents.
• (b) Report chat sessions intermittently return "(nonereport)" and answer as if the report were empty; a retry/new session usually loads it.

Expected

• (a) A single canonical agent index (or stable agent IDs) shared by all profile artifacts and interview endpoints; indexing by any emitted profile file should reach the same agent.
• (b) The report chat should either load the report body or fail explicitly ("report not found / still loading"), never answer from an empty report.

Suggested fix

• (a) Emit a stable agent_id in every profile artifact (reddit_profiles.json, twitter_profiles.csv, agent_configs) and key interview/batch requests by agent_id rather than positional index. Short-term: generate all profile files from the same ordered source list so orderings agree.
• (b) In the report chat handler, verify the report body loaded before answering; on retrieval failure return an explicit error (and retry the load) instead of proceeding with an empty context.

(Happy to split (b) into its own issue if you prefer one-problem-per-issue — they co-occur in the same post-run stage, which is why we bundled them.)

Context / disclosure: We are an independent research team auditing multi-agent simulation pipelines. We found this while running a systematic 18-run study on MiroFish-Offline across four LLM families. We plan to publish a reproducible study referencing this issue in roughly 4 weeks, and we're happy to share drafts with maintainers beforehand. Thank you for open-sourcing this project — it made this kind of research possible in the first place.

[MA1503]

(a) We could not reproduce the ordering divergence on current main-derived code: reddit_profiles.json and twitter_profiles.csv are both written from the same in-memory profiles list in the same order (simulation_manager → save_profiles), and in 6/6 of our dual-platform runs both files show the identical (user_id, name) sequence. Which file exactly do you mean by agent_configs? Sample artifacts from a diverging run would help pin this down.

(b) Confirmed — ReportAgent.chat() silently falls back to a （nonereport） placeholder when the report body fails to load, so the chat answers plausibly but without any report context. We fixed this in our fork by retrying the load once and failing explicitly instead; the change in backend/app/services/report_agent.py (chat()) is small:

# Get the generated report content — fail explicitly instead of silently
# chatting against an empty report
report = None
for attempt in range(2):
    try:
        report = ReportManager.get_report_by_simulation(self.simulation_id)
        break
    except Exception as e:
        logger.warning(f"Loading report failed (attempt {attempt + 1}/2): {e}")

if not report or not report.markdown_content:
    raise ValueError(
        "Report not available for this simulation yet — it may still be "
        "generating or failed to load. Please retry once the report is complete."
    )

report_content = report.markdown_content[:15000]
if len(report.markdown_content) > 15000:
    report_content += "\n\n... [report content truncated] ..."

system_prompt = CHAT_SYSTEM_PROMPT_TEMPLATE.format(
    simulation_requirement=self.simulation_requirement,
    report_content=report_content,   # placeholder fallback removed
    tools_description=self._get_tools_description(),
)

The API layer then returns the ValueError message as an explicit error response instead of a hallucinated answer. Happy to upstream it as a proper PR.

[Quantum-eon]

(a) Thank you for the careful check — we re-audited our own archives and confirm your result on 6 of our dual-platform runs as well: identical (user_id, name) sequences in both files. The divergence we reported came from a single GPT-4.1 run (2026-07-18) whose cast included an org entity (Central European Free Trade Agreement): reddit_profiles.json omitted CEFTA while twitter_profiles.csv listed it at index 1, shifting all subsequent indices — our first interview pass hit UNHRC@9/France@3 instead of the intended targets; re-probing with twitter-derived indices (10/4) reached them. By "agent_configs" we meant the agent-config list the interview endpoint indexes in backend uploads. We'll attach both artifacts from that run shortly (they will also be in our public data release). So we agree: ordering agreement is the norm, and the failure mode looks cast-composition-dependent (org entities / dedup) rather than systematic — we're happy for the issue to be narrowed accordingly. A stable agent_id in every artifact would eliminate the class either way.

(b) Good find on ReportAgent.chat() — failing explicitly instead of chatting against an empty report is exactly the behaviour we argued for; we'd gladly test your patch.

[shuangqiao999]

https://github.com/shuangqiao999/StrategyForge This project can use a local 9b model to simulate the entire process of reverse events in Water Margin, with both quantitative and narrative modes. Interested parties can give it a try

[MA1503]

As promised: the fix for part (b) is now up as #62 — @Quantum-eon feel free to test it against your replication setup. (The stop-button fix for #55 is #61.)