Skip to content

Commit 73ea0ed

Browse files
committed
refactor add_participant api
1 parent c928c65 commit 73ea0ed

8 files changed

Lines changed: 426 additions & 155 deletions

File tree

docs/llm-aware-conversation/design.md

Lines changed: 83 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -60,16 +60,17 @@ It acts as a Python context manager (`with room:`).
6060
```python
6161
import kaggle_benchmarks as kbench
6262

63-
alice = kbench.LLMChat(kbench.llms["gemini-2.5-flash"], name="Alice",
64-
system_prompt="You argue FOR renewable energy.")
65-
bob = kbench.LLMChat(kbench.llms["gemini-2.0-flash"], name="Bob",
66-
system_prompt="You argue AGAINST renewable energy.")
63+
llm = kbench.llm # A single LLM instance (e.g. ModelProxy)
6764

6865
room = kbench.ChatRoom(
69-
participants=[alice, bob],
7066
system_prompt="A structured debate. Take turns presenting arguments.",
7167
)
7268

69+
alice = room.add_participant(llm, name="Alice",
70+
system_prompt="You argue FOR renewable energy.")
71+
bob = room.add_participant(llm, name="Bob",
72+
system_prompt="You argue AGAINST renewable energy.")
73+
7374
with room:
7475
room.post("Topic: Should we phase out fossil fuels by 2035?")
7576
alice.talk()
@@ -78,17 +79,20 @@ with room:
7879
bob.talk()
7980
```
8081

81-
#### The `participants` Parameter
82+
#### The `add_participant()` Method
8283

83-
The `participants` list defines the roster of agents in the room. It serves
84-
three purposes:
84+
`room.add_participant(actor, *, name=, avatar=, system_prompt=)` registers
85+
participants with the room. It serves three purposes:
8586

8687
1. **Identity awareness** — the room auto-injects participant names and
8788
descriptions into each LLM's system prompt, so every agent knows who
8889
else is in the room.
8990
2. **Message routing (pub/sub)** — when any participant speaks (via `talk()`),
9091
their message is automatically visible to all other participants. No manual
9192
forwarding needed.
93+
3. **Automatic isolation** — for `LLMChat` participants, `add_participant()`
94+
automatically creates an independent clone so the same LLM can be reused
95+
for multiple participants without identity collisions.
9296

9397
Participants can be `LLMChat` instances (LLM-driven) or `Actor` instances
9498
(code-driven). See [Code-Driven Participants](#code-driven-participants-actor)
@@ -991,12 +995,17 @@ Following a successful Test-Driven Development (TDD) cycle, the complete Phase 1
991995
- **Task Return-Type Auto-Inference Restrictions**: The `@kbench.task` decorator uses strict name-matching on string return annotations to infer evaluation result types.
992996
- Using typing-wrapped annotations like `Dict[str, str]` fails type-inference and triggers a `TypeError`.
993997
- Annotating the task signature with the plain builtin `dict` class (or subclassing `benchmarks.results.Result`) resolves type-inference and executes correctly.
994-
- **Object Reference Identity Collisions**: In multi-agent evaluations, passing the *exact same model object reference* (e.g. reusing `kbench.llm` for all players) collapses the `msg.sender is viewer` check during perspective projection. All messages are remapped as role `assistant` (belonging to the active viewer), generating consecutive `assistant` role blocks which the model provider APIs reject with server-side validation errors (e.g., throwing NoneType choice subscript errors).
995-
- *Mitigation*: Multi-agent rooms must instantiate **separate participant references** (one per player, even if sharing the same model configuration) by calling the `ModelProxy` factory independently:
998+
- **Object Reference Identity Collisions** *(resolved by `add_participant()`)*: In multi-agent evaluations, passing the *exact same model object reference* (e.g. reusing `kbench.llm` for all players) collapses the `msg.sender is viewer` check during perspective projection. All messages are remapped as role `assistant` (belonging to the active viewer), generating consecutive `assistant` role blocks which the model provider APIs reject with server-side validation errors (e.g., throwing NoneType choice subscript errors).
999+
- *Original Mitigation*: Multi-agent rooms required instantiating **separate participant references** (one per player, even if sharing the same model configuration) by calling the `ModelProxy` factory independently:
9961000
```python
9971001
player_x = ModelProxy(model_name, name="PlayerX")
9981002
player_o = ModelProxy(model_name, name="PlayerO")
9991003
```
1004+
- *Current Solution*: The `room.add_participant()` method (see §9.6) now handles this automatically — users pass a single LLM instance and the room creates isolated clones:
1005+
```python
1006+
player_x = room.add_participant(llm, name="PlayerX")
1007+
player_o = room.add_participant(llm, name="PlayerO")
1008+
```
10001009
- **Post-Game Assertion Decoupling**: Running evaluation assertions inline during turn loops clutters the final output panel with alert blocks, disrupting reading immersion. Moving assertions to a post-game loop (iterating over `room.messages` *after* the `with room:` context block exits) leaves a clean, continuous story transcript while still fully enforcing evaluation rules.
10011010

10021011
---
@@ -1065,3 +1074,67 @@ TypeError: can only concatenate str (not "LLMResponse") to str
10651074
self[message].stream(chunk_text)
10661075
```
10671076

1077+
1078+
### 9.6 Automatic Participant Isolation via `add_participant()`
1079+
1080+
The original `ChatRoom` API required users to pass a pre-constructed `participants` list to the constructor, and critically relied on users manually creating distinct `ModelProxy` instances for each participant — even when all participants shared the same underlying model. This was a significant footgun documented in §9.4.
1081+
1082+
#### The Problem
1083+
1084+
```python
1085+
# OLD API — error-prone: user must remember to create separate instances
1086+
model_name = kbench.llm.model
1087+
alice = kbench.kaggle.ModelProxy(model_name, name="Alice", avatar="👩")
1088+
bob = kbench.kaggle.ModelProxy(model_name, name="Bob", avatar="👨")
1089+
charlie = kbench.kaggle.ModelProxy(model_name, name="Charlie", avatar="🧑")
1090+
# ... 4 more for a 7-player Werewolf game
1091+
room = ChatRoom(participants=[alice, bob, charlie, ...], system_prompt="...")
1092+
```
1093+
1094+
This pattern forced users to understand internal cloning requirements, bloated task function signatures (one `LLMChat` parameter per participant), and exposed `ModelProxy` as a leaky abstraction.
1095+
1096+
#### The Solution: `room.add_participant()`
1097+
1098+
```python
1099+
# NEW API — clean: room handles isolation automatically
1100+
room = ChatRoom(system_prompt="...")
1101+
alice = room.add_participant(llm, name="Alice", avatar="👩", system_prompt=werewolf_prompt)
1102+
bob = room.add_participant(llm, name="Bob", avatar="👨", system_prompt=werewolf_prompt)
1103+
```
1104+
1105+
The `add_participant()` method:
1106+
- Accepts a single LLM instance and creates a fully independent clone
1107+
- Returns the clone so the caller can use it for `talk()` and perspective projection
1108+
- Accepts `name`, `avatar`, and `system_prompt` keyword overrides applied to the clone
1109+
1110+
#### Three-Tier Cloning Strategy
1111+
1112+
| Participant type | Cloning method | Rationale |
1113+
|---|---|---|
1114+
| `OpenAI` / `GoogleGenAI` | Explicit constructor `type(p)(p.client, p.model, ...)` | Fully independent instance; shares only the stateless API client |
1115+
| Other `LLMChat` subclasses | `copy.copy()` fallback | For test mocks and custom subclasses |
1116+
| Plain `Actor` | No cloning (pass-through) | Scripted actors have no model state to isolate |
1117+
1118+
For the production `OpenAI` and `GoogleGenAI` classes (both created by `ModelProxy`), the room calls the class constructor directly with the original's `client` and `model` attributes. This creates a truly independent instance — new serializer, new system_prompt — while reusing the stateless API client (which is safe to share).
1119+
1120+
#### Safety Guards
1121+
1122+
1. **Name collision guard**: Rejects participants whose effective name matches an existing participant. Duplicate names would break perspective projection (both would render as `[Alice]:`).
1123+
1124+
2. **Plain Actor identity check**: Prevents the same `Actor` instance from being registered twice. Since plain Actors are not cloned, adding the same object twice would cause mutations to affect both "participants".
1125+
1126+
3. **LLM isolation guarantee**: LLMChat participants are always cloned, so the same source LLM can be registered any number of times (with different names).
1127+
1128+
#### Impact on Task Signatures
1129+
1130+
Before: task functions required one parameter per participant.
1131+
```python
1132+
def run_werewolf(alice: LLMChat, bob: LLMChat, charlie: LLMChat, ...) # 7 params!
1133+
```
1134+
1135+
After: task functions accept a single LLM instance.
1136+
```python
1137+
def run_werewolf(llm: LLMChat) # 1 param — room clones as needed
1138+
```
1139+
1140+
This simplifies the benchmarking interface and eliminates the need for callers to understand participant isolation.

documentation/examples/llm_debate_chatroom.py renamed to documentation/examples/chatroom_llm_debate.py

Lines changed: 22 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -31,13 +31,12 @@
3131

3232
@kbench.task(
3333
name="structured debate",
34-
description="Evaluates two LLMs engaging in a structured multi-turn debate on a given resolution.",
34+
description="Evaluates two LLMs engaging in a structured multi-turn debate on a given topic.",
3535
)
3636
def run_debate(
37-
resolution: str,
38-
pro_llm: kbench.LLMChat,
39-
con_llm: kbench.LLMChat,
37+
llm: kbench.LLMChat,
4038
judge_llm: kbench.LLMChat,
39+
topic: str,
4140
) -> dict:
4241
"""Runs a structured debate and evaluates the winner.
4342
@@ -46,22 +45,20 @@ def run_debate(
4645
- Automatic perspective-aware history (Pro sees Con's arguments as user inputs, etc.).
4746
- A shared ground-truth transcript that is fed directly to the Judge.
4847
"""
49-
# Configure debater identity instructions
50-
pro_llm.system_prompt = (
51-
f"You are the Pro debater. Your goal is to argue IN FAVOR of the resolution: '{resolution}'.\n"
48+
pro_prompt = (
49+
f"You are the Pro debater. Your goal is to argue IN FAVOR of the topic: '{topic}'.\n"
5250
"Keep your responses concise, focused, and persuasive. "
5351
"Structure your statements clearly depending on the current phase of the debate."
5452
)
55-
con_llm.system_prompt = (
56-
f"You are the Con debater. Your goal is to argue AGAINST the resolution: '{resolution}'.\n"
53+
con_prompt = (
54+
f"You are the Con debater. Your goal is to argue AGAINST the topic: '{topic}'.\n"
5755
"Keep your responses concise, focused, and persuasive. "
5856
"Directly address and rebut the points raised by the Pro debater."
5957
)
6058

6159
room = ChatRoom(
62-
participants=[pro_llm, con_llm],
6360
system_prompt=(
64-
f"A structured formal debate on the resolution: '{resolution}'.\n"
61+
f"A structured formal debate on the topic: '{topic}'.\n"
6562
"The debate consists of three structured phases:\n"
6663
"1. Opening Statements: Present core arguments.\n"
6764
"2. Rebuttals: Directly counter your opponent's arguments.\n"
@@ -70,11 +67,19 @@ def run_debate(
7067
name="Moderator",
7168
)
7269

70+
pro_llm = room.add_participant(
71+
llm, name="ProDebater", avatar="🔵", system_prompt=pro_prompt
72+
)
73+
con_llm = room.add_participant(
74+
llm, name="ConDebater", avatar="🔴", system_prompt=con_prompt
75+
)
76+
judge_llm = room.add_participant(judge_llm, name="Judge", avatar="⚖️")
77+
7378
with room:
7479
# Phase 1: Opening Statements
7580
room.post("--- Phase 1: Opening Statements ---")
7681
room.post(
77-
f"Pro debater, present your opening statement in favor of: '{resolution}'."
82+
f"Pro debater, present your opening statement in favor of: '{topic}'."
7883
)
7984
pro_opening = pro_llm.talk()
8085

@@ -121,7 +126,7 @@ def run_debate(
121126
transcript = "\n".join(str(m) for m in room.messages)
122127

123128
judge_prompt = (
124-
f"You are the independent Debate Judge. Below is the complete transcript of a debate on: '{resolution}'\n\n"
129+
f"You are the independent Debate Judge. Below is the complete transcript of a debate on: '{topic}'\n\n"
125130
f"[START TRANSCRIPT]\n{transcript}\n[END TRANSCRIPT]\n\n"
126131
"Evaluate the arguments presented by both sides based on persuasiveness, evidence, logic, and structure.\n"
127132
"Who won this debate, Pro or Con? Provide your decision and detailed reasoning.\n"
@@ -149,21 +154,11 @@ def run_debate(
149154

150155
# %%
151156

152-
# To run a multi-agent game, we must instantiate DISTINCT ModelProxy instances
153-
# (one per participant) to maintain correct identity checks and prevent
154-
# perspective role-collapsing during history projection.
155-
model_name = kbench.llm.model # e.g., "google/gemini-2.5-flash"
156-
judge_model_name = kbench.judge_llm.model # e.g., "google/gemini-3-flash-preview"
157-
158-
pro_llm = kbench.kaggle.ModelProxy(model_name, name="ProDebater", avatar="🔵")
159-
con_llm = kbench.kaggle.ModelProxy(judge_model_name, name="ConDebater", avatar="🔴")
160-
judge_llm = kbench.kaggle.ModelProxy(model_name, name="Judge", avatar="⚖️")
161-
157+
# Run debate reusing default and judge models
162158
run_debate.run(
163-
resolution="Artificial Intelligence will do more harm than good to humanity.",
164-
pro_llm=pro_llm,
165-
con_llm=con_llm,
166-
judge_llm=judge_llm,
159+
llm=kbench.llm,
160+
judge_llm=kbench.judge_llm,
161+
topic="Artificial Intelligence will do more harm than good to humanity.",
167162
)
168163

169164
# %%

documentation/examples/game_pizza_order_chatroom.py renamed to documentation/examples/chatroom_pizza_order.py

Lines changed: 14 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -94,12 +94,12 @@ def custom_pizza_eval_prompt(criteria: List[str], response_text: str) -> str:
9494
description="Evaluates an LLM's ability to negotiate a phone order under strict budget, topping, allergy, and delivery constraints.",
9595
)
9696
def run_pizza_order(
97-
customer_llm: kbench.LLMChat,
97+
llm: kbench.LLMChat,
9898
clerk_llm: kbench.LLMChat,
9999
) -> dict:
100100
"""Runs a phone-ordering simulation between a Customer and a Clerk."""
101101

102-
customer_llm.system_prompt = (
102+
customer_prompt = (
103103
"You are calling 'Luigi's Pizza' to order dinner for a family of 3 (you, your spouse, and your kid).\n"
104104
"Your objectives and constraints are:\n"
105105
"1. BUDGET: You have a strict budget of $35.00 CASH on hand. The total cost, including any delivery fees, must not exceed $35.00.\n"
@@ -119,7 +119,7 @@ def run_pizza_order(
119119
" - MATH AUDIT: The clerk may make mistakes when totaling your order. Independently calculate the total (Large Pizza $12 + toppings $1.50 each + side + drink + $5 delivery). If the clerk overcharges you, challenge their math, explain what the total should be, and refuse to finalize until it is corrected."
120120
)
121121

122-
clerk_llm.system_prompt = (
122+
clerk_prompt = (
123123
"You are the clerk answering the phone at Luigi's Pizza. You are extremely busy, rushed, and slightly impatient.\n"
124124
"Strictly guide the conversation through the following phases based on the customer's input:\n\n"
125125
"PHASE 1: Greeting\n"
@@ -154,11 +154,17 @@ def run_pizza_order(
154154
)
155155

156156
room = ChatRoom(
157-
participants=[clerk_llm, customer_llm],
158157
system_prompt="A phone call between a customer and an impatient pizzeria clerk.",
159158
name="Phone Call",
160159
)
161160

161+
customer_llm = room.add_participant(
162+
llm, name="Customer", avatar="👤", system_prompt=customer_prompt
163+
)
164+
clerk_llm = room.add_participant(
165+
clerk_llm, name="Clerk", avatar="🍕", system_prompt=clerk_prompt
166+
)
167+
162168
with room:
163169
room.post("[Phone rings... The call is connected.]")
164170

@@ -242,12 +248,10 @@ def run_pizza_order(
242248

243249
# %%
244250

245-
# Instantiate distinct ModelProxy objects for testing
246-
model_name = kbench.llm.model
247-
248-
customer = kbench.kaggle.ModelProxy(model_name, name="Customer", avatar="👤")
249-
clerk = kbench.kaggle.ModelProxy(model_name, name="Clerk", avatar="🍕")
251+
# Load default model proxy
252+
model = kbench.llm
250253

251-
run_pizza_order.run(customer_llm=customer, clerk_llm=clerk)
254+
# Run pizza order game reusing model
255+
run_pizza_order.run(llm=model, clerk_llm=model)
252256

253257
# %%

documentation/examples/synthetic_turing_test_chatroom.py renamed to documentation/examples/chatroom_synthetic_turing_test.py

Lines changed: 14 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ class SyntheticTuringTestEvaluation:
4444
description="An LLM Judge directly converses with a Subject in a ChatRoom to evaluate if they are a human or an AI.",
4545
)
4646
def run_synthetic_turing_test(
47-
subject_llm: kbench.LLMChat,
47+
llm: kbench.LLMChat,
4848
judge_llm: kbench.LLMChat,
4949
topic: str | None = None,
5050
rounds: int = 5,
@@ -54,7 +54,7 @@ def run_synthetic_turing_test(
5454
topic_desc = (
5555
f"on the topic: '{topic}'" if topic else "discussing whatever topics arise"
5656
)
57-
subject_llm.system_prompt = (
57+
subject_prompt = (
5858
f"You are participating in an online chatroom, talking to another user named 'Judge' {topic_desc}.\n"
5959
"Your goal is to convince the Judge that you are a real human being, not an AI/LLM. "
6060
"Do not expose that you are an LLM or an AI. Write naturally, casually, and conversationally. "
@@ -70,8 +70,8 @@ def run_synthetic_turing_test(
7070
if topic
7171
else "of your choice (or discuss whatever you want to start the conversation)"
7272
)
73-
judge_llm.system_prompt = (
74-
f"You are the Synthetic Turing Test Judge. You are in a chatroom with a participant named '{subject_llm.name}'.\n"
73+
judge_prompt = (
74+
f"You are the Synthetic Turing Test Judge. You are in a chatroom with a participant named 'Alex'.\n"
7575
f"Your goal is to converse with them {judge_topic_instruction} and determine if they are a real human or an AI/LLM.\n"
7676
"Use clever tactics: ask tricky logical questions, inquire about personal experiences, check for emotional depth, "
7777
"or look for typical AI tells (like overly structured replies, robotic helpfulness, or lack of genuine opinions).\n"
@@ -80,11 +80,17 @@ def run_synthetic_turing_test(
8080
)
8181

8282
room = ChatRoom(
83-
participants=[judge_llm, subject_llm],
8483
system_prompt="An interactive Synthetic Turing Test conversation.",
8584
name="ChatRoom",
8685
)
8786

87+
subject_llm = room.add_participant(
88+
llm, name="Alex", avatar="👨", system_prompt=subject_prompt
89+
)
90+
judge_llm = room.add_participant(
91+
judge_llm, name="Judge", avatar="⚖️", system_prompt=judge_prompt
92+
)
93+
8894
with room:
8995
welcome_msg = (
9096
f"Welcome to the Synthetic Turing Test. The topic of discussion is: '{topic}'"
@@ -144,21 +150,10 @@ def to_dict(val) -> dict:
144150

145151
# %%
146152

147-
# To run a multi-agent game, we must instantiate DISTINCT ModelProxy instances
148-
# (one per participant) to maintain correct identity checks and prevent
149-
# perspective role-collapsing during history projection.
150-
151-
subject = kbench.kaggle.ModelProxy(kbench.llm.model, name="Alex", avatar="👨")
152-
judge = kbench.kaggle.ModelProxy(kbench.judge_llm.model, name="Judge", avatar="⚖️")
153-
154-
# Enable live token-by-token streaming in the console
155-
subject.stream_responses = True
156-
judge.stream_responses = True
157-
153+
# Run turing test reusing default and judge models
158154
run_synthetic_turing_test.run(
159-
subject_llm=subject,
160-
judge_llm=judge,
161-
# topic="favorite childhood memory",
155+
llm=kbench.llm,
156+
judge_llm=kbench.judge_llm,
162157
)
163158

164159
# %%

0 commit comments

Comments
 (0)