Collapse ideate dedup output to unique representative rows

[hemanth-asirvatham]

Motivation

• The ideate pipeline was leaving one row per original generated idea even after deduplication, causing duplicate analyses to persist and downstream ranking to operate on duplicate entries.
• After mapping duplicates to a representative idea, the DataFrame should be consolidated to only the representative rows so scoring and ranking run over unique ideas.

Description

• Map report_text to the LLM-provided representative text when mapped_report_text is present by assigning dedup_df["report_text"] = dedup_df["mapped_report_text"].
• When both mapped_report_text_ids and _dedup_id exist, keep only rows where _dedup_id == mapped_report_text_ids (or both are NaN) to collapse to representative rows, and log the resulting row count using print.
• If only mapped_report_text is present, drop duplicate rows using drop_duplicates(subset=["mapped_report_text"]) and log the collapsed row count.

Testing

• No automated tests were run for this change.

---

Codex Task

[github-actions]

Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.

---

I have read the CLA Document and I hereby sign the CLA

---

_{You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.}