Cohort construction is non-deterministic: the same input builds a different persona population on every run

[outerheaven666]

摘要：同一份输入重复跑一遍，每次都会构建出不同的人群——本体、实体、人格全是新的。交互式使用没问题，但只要涉及多次运行之间的比较，结果就不可复现，而 API 层面看不出这一点。

我们实测：同一输入、逐字相同的访谈提示词，两次跑出 9.5% 和 60.0% 的弃权率。把下列三处温度设为 0 之后，四次运行中有三次产出逐字节相同的人格花名册。

建议：把这几处做成可配置，默认值保持现状（如 MIROFISH_COHORT_TEMPERATURE，不设＝现有行为，设 0＝可复现人群），对现有用户零影响。详情见下方英文全文。

---

Thanks for MiroFish — being able to run the whole stack against a local model is what made the measurement below possible.

Issue: re-running the same input builds a different crowd each time — new ontology, new entities, new personas. For interactive use that is fine and arguably desirable. For any use where runs are compared against each other it makes results non-reproducible, and nothing in the API surface hints at it.

What we measured

Running the same event description repeatedly through graph/ontology/generate → graph/build → simulation/prepare → simulation/start → simulation/interview, with a byte-identical interview prompt, gave 9.5% and 60.0% abstention on two runs of the same input.

After setting the temperatures below to 0, three of four subsequent runs produced byte-identical persona rosters (comparing reddit_profiles.json, ignoring the random numeric suffix on usernames); the fourth was the first run after a container start and differed by a single entity.

Full data, hashes and the reproduction commands: https://github.com/outerheaven666/llm-agent-simulation-variance/blob/v1.0-finding/README.md

The call sites

file	line	current
backend/app/services/ontology_generator.py	~181	temperature=0.3
backend/app/services/oasis_profile_generator.py	~520	temperature=0.7 - (attempt * 0.1)
backend/app/services/simulation_config_generator.py	~445	temperature=0.7 - (attempt * 0.1)

The third is reached via SimulationManager.prepare_simulation() and is easy to miss — we missed it on a first pass that only grepped the files we already knew were on the path.

We deliberately did not touch the interview stage. Per-agent response variation is the interesting part; only cohort construction needs pinning.

Suggested change

Make these configurable, with today's values as the defaults. Something like MIROFISH_COHORT_TEMPERATURE — unset gives current behaviour, 0 gives reproducible cohorts. Nothing changes for existing users.

We have a patch that hard-sets 0, but that is right for our use case and wrong as a default for yours, so it is not offered as-is. Happy to send a PR in whichever shape you prefer.

One implementation detail worth knowing first: both retry ladders read temperature = 0.7 - (attempt * 0.1), i.e. 0.7 → 0.6 → 0.5 — descending, per the comment 每次重试降低温度. Anyone making the base configurable while keeping the ladder will find that a base of 0 computes negative temperatures on retry. Since the ladder's purpose is more reliable JSON on retry, and 0 is already the most reliable setting, a flat value is probably what is wanted when the base is 0.

Note for the Offline fork

nikmcfly/MiroFish-Offline adds a fourth site not present here — backend/app/storage/ner_extractor.py L85, temperature=0.1 — on the graph/build path. Happy to open a matching issue there if useful.

Also worth a README line (not a bug)

Both compose files declare the backend as build: . and mount only backend/uploads, so the Python source is baked into the image at build time. Editing the host tree changes nothing until docker compose build runs. This cost us a while, and one line in the docs would save the next person the same.

---

Context: this came out of a study measuring run-to-run variance of agent-population simulations at 7B scale on consumer hardware. The write-up, with the raw per-persona interviews and the roster hashes referred to above, is here:

https://github.com/outerheaven666/llm-agent-simulation-variance/blob/v1.0-finding/README.md

The finding — that readings of the same input are not reproducible at that scale even after the cohort is pinned — depends on the determinisation above being possible, which is what makes this fix load-bearing rather than cosmetic.

To be explicit: the finding is about what a 7B local model can support on 8 GB of consumer hardware, not a claim about MiroFish's design. The determinisation above is what let us rule out cohort churn as the explanation.

[outerheaven666]

Follow-up: the determinisation holds only within one container lifetime

Since filing this I ran the patched build repeatedly and want to add a caveat, because anyone adopting the change could otherwise be caught by it.

With the four cohort temperatures at 0, consecutive builds inside the same running container do reproduce byte-identical persona rosters — that part works. But the first build after a container restart often produces a different roster, including a different persona count.

Roster sha256 (the random numeric suffix on usernames stripped) and persona count:

date	build	personas	roster sha
07-29	1st	20	9dde46d86b32
07-29	2nd	19	8c585d217d02
07-29	3rd	20	9dde46d86b32
07-29	4th	20	c75b39718335
08-02	1st after restart	19	8c585d217d02
08-02	2nd	20	90caa587f7cb
08-02	3rd	20	90caa587f7cb
08-02	4th	20	90caa587f7cb
08-03	1st after restart	19	8c585d217d02
08-03	2nd	20	90caa587f7cb
08-03	3rd	20	90caa587f7cb
08-03	4th	20	90caa587f7cb

The 19-persona roster is missing one entity (market_event) relative to the 20-persona one.

Stated with the limits it has. Across all 33 builds I have on record the pattern holds in three sequences and is contradicted by the 07-29 one above, where a 20-persona build came first and the 19-persona one came second. The roster 8c585d217d02 also recurs on five separate dates, so it may be one particular stable cohort rather than a first-build effect. I have not identified the cause. Neo4j state at startup is the obvious suspect, but I have not tested it and am not asserting it.

Why it matters for the proposed change. If MIROFISH_COHORT_TEMPERATURE=0 is offered as "reproducible cohorts", someone comparing runs across a restart may still get different crowds with no way to notice. Either of these would help:

• document the scope as "reproducible within a container lifetime", or
• if the cause turns out to be startup state, fix it so the guarantee is unconditional

Our own workaround is to discard the first build after any restart and assert the roster hash across subsequent runs. Cheap, but it is a workaround, not a fix.

[outerheaven666]

Correction to my earlier comment, and a pointer to a deeper cause

Two things to add to what I wrote above.

First, a correction. I reported that with the cohort temperatures at 0, "three of four subsequent runs produced byte-identical persona rosters". That was a hash of persona names only, with the random numeric suffix stripped. Comparing the full reddit_profiles.json profiles instead, across those same four builds:

pair	persona field differs
1 vs 2 / 3 / 4	19/19
2 vs 3	3/20
2 vs 4	3/20
3 vs 4	0/20

Only one pair had identical personas. The names were stable; the descriptions that actually define each agent's viewpoint were not. So my "byte-identical rosters" claim overstated what had been measured, and I would rather correct it here than leave it standing in the thread.

It also does not behave the way one would assume. Mapping each run's interview results to a probability vector, the pair with identical personas differs by 0.2481 in the final reading, while the closest pair (0.0393) had three personas differing. Identical names do not imply identical people, and identical people did not produce identical readings.

Second, setting those temperatures to 0 does not on its own give reproducible cohorts. With all four at 0, two consecutive ontology/generate + graph/build in the same container still produce different entity sets — 27 vs 26, 24 shared. (I have not measured extraction stability at the default temperature, so I cannot say whether the temperature change helps, hurts, or makes no difference here — only that it is not sufficient.) Details, reproduction, and a related finding — non-person entities such as market event and MiroFish itself becoming interviewable agents — are in #759.

I am not asking to reopen anything here; the request in this issue still stands on its own. The point of this note is that if the change ships described as "reproducible cohorts", that description will not hold in practice, and #759 has what is needed to decide how to word it.

Full write-up with the underlying data: https://github.com/outerheaven666/llm-agent-simulation-variance/tree/v1.1-finding