Fix/gsm8k tuple response attribute error

[SYED-M-HUSSAIN]

Fix: Handle tuple responses in GSM8K model output to avoid AttributeError

📍 Summary

This PR resolves (https://github.com/confident-ai/deepeval/issues/1612) by adding robust handling for non-standard model responses (e.g., tuples) that previously caused an AttributeError in the GSM8K benchmark.

🐞 Problem

The original implementation assumed that model outputs would always return an object with an .answer attribute (e.g., NumberSchema). However, in some cases, the model returns a tuple, causing the following error:

AttributeError: 'tuple' object has no attribute 'answer'

This occurred specifically at:

prediction = str(res.answer)

and earlier:

prediction, score = self.predict(model, golden).values()

✅ Solution

This PR includes a targeted fix that preserves backward compatibility:

• Replaced unsafe .values() unpacking with direct dictionary key access for safer and clearer code.

• Introduced a helper method _extract_prediction_from_response() to robustly extract the prediction from various formats:

  • NumberSchema objects
  • Tuples
  • Raw strings, dictionaries, or objects with .text or .content attributes

• Improved exception handling by catching both AttributeError and TypeError in case of unexpected structures.

🧼 Code Changes

In gsm8k.py:

• Replaced:

  prediction, score = self.predict(model, golden).values()

  With:

  result = self.predict(model, golden)
  prediction = result["prediction"]
  score = result["score"]

• Modified:

  prediction = str(res.answer)

  To:

  prediction = self._extract_prediction_from_response(res)

• Added a new method _extract_prediction_from_response() for robust prediction extraction.

🛡️ Impact

• ✅ Fixes the reported crash in GSM8K benchmark (GSM8K: AttributeError: 'tuple' object has no attribute 'answer' #1612)
• ✅ Makes prediction extraction robust to unexpected model return formats
• ✅ Preserves existing functionality and backward compatibility

---

🧾 Related

• Issue: GSM8K: AttributeError: 'tuple' object has no attribute 'answer' #1612
• Related discussion: AttributeError: 'tuple' object has no attribute 'answer' #1535

---

[vercel]

@SYED-M-HUSSAIN is attempting to deploy a commit to the Confident AI Team on Vercel.

A member of the Team first needs to authorize it.

[penguine-ip]

hey @SYED-M-HUSSAIN have you tested the code out? THe if else looks really complicated, not sure if everything is necessary?

[SYED-M-HUSSAIN]

Hey @penguine-ip , yes I’ve tested the code, it’s working as expected. I agree the if-else logic looks a bit dense, but it’s designed to handle various response formats from different model types. That said, I’m open to simplifying it if we can keep the flexibility intact. Let me know your thoughts!

[penguine-ip]

@SYED-M-HUSSAIN yes please lets simplify it a bit! we can always fix it further if someone raises an issue again :)

[SYED-M-HUSSAIN]

Thanks @penguine-ip, I’ve refactored the method to simplify the if-else chain

[penguine-ip]

@SYED-M-HUSSAIN thanks!