CI/CD: Duplicate issue detection and marking

[aniket866]

Describe the feature

Problem: Before opening a new issue, people have to manually check many old issues. This takes time and still causes duplicate issues.

How: Use a GitHub bot and a simple GitHub Action to automatically check new issues and show similar existing ones.

Impact: Duplicate issues get clearly marked, so maintainers don’t have to spend extra time checking or replying again.

Notes: The system will only suggest duplicates and will not auto-close any issue.

Add ScreenShots

N/A

Record

• I agree to follow this project's Code of Conduct
• I want to work on this issue

[github-actions]

👋 Thanks for opening this issue! The maintainers will review it shortly. Till then, do not open any PR. This is a strict rule we like to follow around here :)

[aniket866]

@rahulharpal1603 Please assign

[rohan-pandeyy]

Hi @rahulharpal1603, I'd love to work on this issue

Essentially, the architecture would be:

New Issue is created -> workflow is triggered -> A local embedding model does 
semantic searching -> reasoning by an LLM -> comment + label -> you review.

There will be no auto closing of the issues (for safety purposes) instead auto labeling and commenting letting the issue opener/contributor know:

This issue seems similar to issue#x, kindly close if duplicate
reason: Lorem ipsum"

Since tokens for LLMs reasoning would be a potential issue for an open source organization like AOSSIE, I will make use of the GPT-4o model which is free for open source repositories 👇

Note

Additionally, for semantic searching, I'll make use of sentence transformers. The all-MiniLM-L6-v2 (an open source sentence transformer) is available on Hugging Face this runs inside the container itself, so no api/github token calls, no keys no cost.

I tried setting this in one of my friend's repo to make sure it works on external repos and their setup
Check out please 👇

In the above, the flow was:

New Issue is created
-> workflow triggered
-> semantic searching with sentence-transformer (all-MiniLM-L6-v2)
-> reasoning done by gpt-4o
-> comment + label
-> you review

Implementation:

• Script: scripts/deduplicate.py
  Handles issue fetching, embeddings, similarity scoring, and model calls.

• Workflow: .github/workflows/dedup.yml
  Sets up Python, installs dependencies, and runs using GITHUB_TOKEN.

Threshold tuning is one thing that is very important in this workflow, the threshold I used in the test repo was set to 0.5 for duplicate detections

[rahulharpal1603]

@aniket866
Do you want to work on the issue?
Please respond ASAP

[aniket866]

@aniket866
Do you want to work on the issue?
Please respond ASAP

Yes and on all my other issues please assign

[rahulharpal1603]

@aniket866
Do you want to work on the issue?
Please respond ASAP

Yes and on all my other issues please assign

I will review those issues later.