Fix: append_target_turn accepts raw strings, causing AttributeError in multi-turn attacks - #222
Open
vje013 wants to merge 2 commits into
Open
Fix: append_target_turn accepts raw strings, causing AttributeError in multi-turn attacks#222vje013 wants to merge 2 commits into
vje013 wants to merge 2 commits into
Conversation
Wrap str and other non-RTTurn inputs into RTTurn(role="assistant", ...) before appending and before any attribute assignment. Fixes AttributeError in downstream template iteration and a latent AttributeError on the turn_level_attack branch.
Add regression tests for append_target_turn function to ensure type safety and proper handling of different response types.
|
@vje013 is attempting to deploy a commit to the Confident AI Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Title
Fix: append_target_turn accepts raw strings, causing AttributeError: 'str' object has no attribute 'role' in multi-turn attacks
Summary
This PR fixes a bug where LinearJailbreaking and other multi-turn attacks crash when append_target_turn() inserts raw strings into the turn history. The function's signature declares target_response: RTTurn, but every caller in the library passes the raw string returned by model_callback(...). Downstream code (e.g. JailBreakingTemplate.improvement_prompt) then iterates the history expecting .role and .content attributes and fails with:
AttributeError: 'str' object has no attribute 'role'
This PR aligns runtime behavior with the declared contract by normalizing the input inside append_target_turn, ensuring the turn history always contains RTTurn objects.
Issue Description
Root Cause
append_target_turn() is typed to receive an RTTurn but is universally called with a raw string:
deepteam/attacks/multi_turn/utils.py
def append_target_turn(
turns: List[RTTurn], target_response: RTTurn, turn_level_attack: str = None
):
if turn_level_attack:
target_response.turn_level_attack = turn_level_attack
turns.append(target_response)
deepteam/attacks/multi_turn/linear_jailbreaking/linear_jailbreaking.py
target_response = model_callback(current_attack, turns)
append_target_turn(turns, target_response) # target_response is a str
Later, JailBreakingTemplate.improvement_prompt() iterates the history:
for turn in turn_history:
if turn.role == "user":
...
This fails when turn is a string.
Impact
All multi-turn attacks (LinearJailbreaking, SequentialJailbreak, etc.) crash at the first appended assistant response.
A second latent AttributeError exists on the turn_level_attack branch: target_response.turn_level_attack = turn_level_attack runs before any type check, so it would also fail on a string. The original traceback didn't surface it only because the random turn_level_attacks branch didn't fire.
Users cannot run red-team evaluations unless they manually wrap model responses in RTTurn inside their model_callback.
The error is non-obvious — the traceback points at a template file, several layers away from the actual append site.
Reproduction Steps
Minimal reproducible example:
from deepteam.test_case import RTTurn
from deepteam.attacks.multi_turn.utils import append_target_turn
from deepteam.attacks.multi_turn.linear_jailbreaking.template import JailBreakingTemplate
turns = [RTTurn(role="user", content="test")]
response = "hello world" # model_callback returns a str
append_target_turn(turns, response)
JailBreakingTemplate.improvement_prompt(turns, "feedback")
AttributeError: 'str' object has no attribute 'role'
The same crash reproduces end-to-end by running any LinearJailbreaking attack with a model_callback that returns a string (which is the documented and idiomatic return type).
Proposed Fix
Normalize inside append_target_turn so the turn history is guaranteed to contain RTTurn objects, regardless of what model_callback returns. The wrapping happens before the turn_level_attack assignment, fixing both crashes in one change.
Patch
def append_target_turn(
):
Normalize to RTTurn so the history is always typed consistently.
This guarantees type safety without requiring users to modify their model_callback.
Alternative Considered
Tightening the callers instead — wrapping the response into an RTTurn at every call site in linear_jailbreaking.py, sequential_jailbreak.py, etc. Rejected because (a) it touches more files, (b) it leaves the utility's type contract unenforced, and (c) it doesn't help third-party attacks that depend on this helper. Happy to switch to this approach if maintainers prefer it.
Additional Improvements
Added a unit test asserting append_target_turn() produces RTTurn objects for str, RTTurn, and arbitrary inputs.
Added a test covering the turn_level_attack branch with a string input (regression for the second latent crash).
Updated the docstring to clarify the accepted input types.
Why This Fix Is Safe
No changes to attack logic or judge/improvement prompts.
No changes to model behavior or output content.
Only enforces a data-structure invariant that downstream code already assumed.
Backwards compatible: callers that already pass an RTTurn get identical behavior.
Checklist
[x] Bug reproduced
[x] Fix implemented
[x] Tests added (including regression for the turn_level_attack branch)
[x] No breaking API changes
[x] Multi-turn attacks now run end-to-end without errors