fix(agentic_rag): honor similarity_threshold in deduplicate_chunks

[andrewwhitecdw]

Bug: deduplicate_chunks accepted a similarity_threshold argument but only performed exact-match deduplication, so the threshold was silently ignored.

Fix: Compare chunks using Jaccard similarity over normalized token sets and only keep a chunk when it is below the threshold relative to every chunk already kept.

Test: Added regression tests covering exact duplicates, near-duplicates at the default threshold, threshold sensitivity, and empty input.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ santoshkumarradha
❌ andrewwhitecdw
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[AbirAbbas]

@andrewwhitecdw thanks for the fix — the threshold behavior in deduplicate_chunks is a real issue and this looks like a sensible correction. We can't accept any contribution until the CLA is signed though, and the check has been pending since the PR was opened: https://cla-assistant.io/Agent-Field/agentfield?pullRequest=829

If it's still unsigned in a few days we'll close this to keep the review queue tidy — it can be reopened any time once you've signed.

[AbirAbbas]

Thanks for catching this — you were right that similarity_threshold was dead weight in deduplicate_chunks; it was doing a plain exact-match dedup and silently ignoring the parameter entirely.

We can't merge patches from contributors who haven't signed the CLA, so rather than leave the bug sitting here we reimplemented the fix independently in #832. Full credit to you for the report — it's a real bug and it would have kept biting the multi-pass retrieval loop in main.py.

If you'd like to sign the CLA, we'd be glad to take contributions from you directly next time.

Closing this in favor of #832.