There is an incongruency between the in-game moderator's stated rules and the actual game mechanics, which may lead non-expert visitors to mistakenly believe there is a bug.
The moderator statement visible in the game UI explicitly states:
"The discussion rule is: Players speak in round-robin order for 2 round(s). The starting speaker rotates to the next player in the list for each subsequent day, but remains the same for all rounds within that day."
However, the underlying game harness actually utilizes a bidding system where players bid for a chance to speak more, which frequently disrupts the strict round-robin order.
See e.g.
|
def _day_bid(self, entries, obs, common_args): |
Impact:
The probability of this causing confusion is high. For example, in at least one of the highlighted games featured on the Kaggle blog (https://www.kaggle.com/blog/game-arena-werewolf), specifically episode 74198839 (https://www.kaggle.com/benchmarks/kaggle/werewolf?episodeId=74198839), the round-robin rule is visibly broken. On the 3rd day, during a crucial discussion, one of the LLMs never takes its turns.
Proposed Solution:
Potential solutions to resolve this incongruency include:
- Adding a brief explanation in the UI clarifying that the speaking order is not strictly round-robin due to underlying agent dynamics.
- Modifying the moderator text for future games to accurately reflect the bidding-based or dynamic speaking mechanics instead of promising a strict round-robin.
There is an incongruency between the in-game moderator's stated rules and the actual game mechanics, which may lead non-expert visitors to mistakenly believe there is a bug.
The moderator statement visible in the game UI explicitly states:
"The discussion rule is: Players speak in round-robin order for 2 round(s). The starting speaker rotates to the next player in the list for each subsequent day, but remains the same for all rounds within that day."
However, the underlying game harness actually utilizes a bidding system where players bid for a chance to speak more, which frequently disrupts the strict round-robin order.
See e.g.
kaggle-environments/kaggle_environments/envs/werewolf/harness/base.py
Line 667 in 2dcd83e
Impact:
The probability of this causing confusion is high. For example, in at least one of the highlighted games featured on the Kaggle blog (https://www.kaggle.com/blog/game-arena-werewolf), specifically episode 74198839 (https://www.kaggle.com/benchmarks/kaggle/werewolf?episodeId=74198839), the round-robin rule is visibly broken. On the 3rd day, during a crucial discussion, one of the LLMs never takes its turns.
Proposed Solution:
Potential solutions to resolve this incongruency include: